Android Code Coverage on Firebase Test Lab (Part 3: CI/CD Integration)

Photo by Jonathan Hanna on Unsplash

This continuing series of articles will explain how to generate a code coverage report for Android instrumentation tests running on Firebase Test Lab.

Requirements for Part 3

In Part 1 of this series, we saw how to:

• Produce XML & HTML reports for code coverage of on-device tests
• Run on API 28, API 29, and API 30 devices in Firebase Test Lab
• Support Android applications targeting API 28, API 29, and API 30

In Part 2, we extended this with:

• Supporting Android Test Orchestrator
• Combining results with off-device unit tests
• Supporting multi-module applications

In Part 3, we will look at how to satisfy our final requirements:

• Integrate with flank
• Use scripts to run in a CI/CD pipeline

Integrate with flank

Flank is an amazing test runner that allows us to remove a lot of the boilerplate we used to execute tests in Firebase Test Lab. Recall that to execute tests using Android Test Orchestrator before we would need to run

gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables \
clearPackageData=true,
coverage=true,\
coverageFilePath=/sdcard/Download/ \
--directories-to-pull /sdcard/Download

For this we’ll use the Fulladle plugin (the multi-module Gradle plugin for Flank, not to be confused with the single-module plugin named Fladle). We just need to apply the Fullable plugin by adding the plugin to our project-level build.gradle:

plugins {
    id "com.osacky.fulladle" version "0.17.3"
    ...
}

Then defining a fladle block in the project-level build.gradle:

Finally we’ll need to write our service account credentials to a JSON file, here we’ve used flank-gradle-service-account.json in the root of the project as an example, though obviously you wouldn’t want to check this file into source control for security reasons. You can find instructions on creating this file here.

Now we’re ready to use Flank to run our tests. Rather than the complex gcloud commands we used before, we can just run

./gradlew runFlank

And all our tests will run in Firebase Test Lab, just like before.

Use scripts to run in a CI/CD pipeline

Often, the end goal of any code coverage system is to integrate with your CI/CD pipeline so code coverage numbers can be automatically generated for each build. This section will examine how we can use bash scripts to automatically execute the steps we’ve so far conducted manually, though it could obviously be tweaked to another shell.

We will also assume that your script already creates the flank-gradle-service-account.json before execution (and cleans it up afterwards), whether that’s writing the file from a base64-encoded variable or pulling it out of a more secure secrets manager.

As we discussed before, we can run the tests on Firebase Test Lab using runFlank, but if we use the tee command then we can pipe it both to stdout and to a file, allowing us to both observe output as well as save that output to a file for use later:

./gradlew runFlank | tee results.txt

For any given runFlank execution, all our coverage files are still dumped in a single GCS bucket. We can find the gs:// path for that bucket by running.

gcsbucket=$(cat results.txt | grep matrix_id | awk -F/ '{print "gs://" $6 "/" $7}')

which will yield something like

gs://<project-id/<timestamp>

The bucket will have a separate subfolder for each matrix of tests (typically each test apk that was uploaded) which will then contain individual coverage files for each test (because we’re using the Android Test Orchestrator)

gs://<project-id>/<timestamp>/matrix_0/Pixel2-29-en-portrait/artifacts/sdcard/Download/com.something.app.TestFileAlpha#test1.ec
...
gs://<project-id>/<timestamp>/matrix_0/Pixel2-29-en-portrait/artifacts/sdcard/Download/com.something.app.TestFileOmega#test99.ec
gs://<project-id>/<timestamp>/matrix_1/Pixel2-29-en-portrait/artifacts/sdcard/Download/com.something.feature.TestFileBeta#test1.ec
...
gs://<project-id>/<timestamp>/matrix_1/Pixel2-29-en-portrait/artifacts/sdcard/Download/com.something.feature.TestFileGamma#test99.ec

Retrieving all these files by hand is a bit of a chore, but we can automate some of this. Once we have the gs:// path from above, we can easily retrieve the paths for the individual matrices by running

gsutil ls $gcsbucket | grep -v matrix_ids | grep matrix

which will yield something like

gs://<project-id>/<timestamp>/matrix_0/
gs://<project-id>/<timestamp>/matrix_1/
gs://<project-id>/<timestamp>/matrix_2/
gs://<project-id>/<timestamp>/matrix_3/

We can turn around and use gsutil on each one of these to print out the full list of gs:// paths to each coverage file:

gsutil ls $gcsbucket | grep -v matrix_ids | grep matrix | while read -r line; do gsutil ls ${line}Pixel2-29-en-portrait/artifacts/sdcard/Download; done

And then finally use gsutil on each resulting line to download the corresponding code coverage file to app/build/outputs/code_coverageso that we can use our previous techniques to generate the coverage reports that we want.

gsutil ls $gcsbucket | grep -v matrix_ids | grep matrix | while read -r line; do gsutil ls ${line}Pixel2-29-en-portrait/artifacts/sdcard/Download; done | while read -r line; do gsutil cp $line app/build/outputs/code_coverage; done

Altogether, this looks something like

./gradlew runFlank 
  | tee results.txt
gcsbucket=
  $(cat results.txt 
    | grep matrix_id 
    | awk -F/ '{print "gs://" $6 "/" $7}')
gsutil ls $gcsbucket 
  | grep -v matrix_ids 
  | grep matrix 
  | while read -r line; do gsutil ls ${line}Pixel2-29-en-portrait/artifacts/sdcard/Download; done 
  | while read -r line; do gsutil cp $line app/build/outputs/code_coverage; done
./gradlew jacocoUnifiedReport

TL;DR

We’ve now achieved the requirements for this article and for this series. Looking back, these were the steps we needed to take:

1. In the project-level build.gradle, include the fulladle plugin and add a fladle block.
2. For our script to automate running the tests on Firebase Test Lab and generate a report, we’ll first execute ./gradlew runFlank and capture the results.
3. Next we’ll use grep and awk to figure out the gs:// path for the GCS bucket.
4. Next we’ll use gsutil ls to list the contents of that bucket and then use grep and while read to filter down to just the gs:// paths for each individual matrix’s sdcard/Download folder, and finally use gsutil cp to download all the created coverage file to app/build/outputs/code_coverage.
5. Finally we’ll run ./gradlew jacocoUnifiedReport to generate an HTML report for all the tests.

You can see a solution that combines all of these changes by looking at https://github.com/Aidan128/FirebaseTestLabCoverageExample and checking out the git tag part_three.