Android Code Coverage on Firebase Test Lab (Part 2: Advanced Topics)

Photo by Kai Dahms on Unsplash

This continuing series of articles will explain how to generate a code coverage report for Android instrumentation tests running on Firebase Test Lab.

Requirements for Part 2

In Part 1 of this series, we saw how to:

• Produce XML & HTML reports for code coverage of on-device tests
• Run on API 28, API 29, and API 30 devices in Firebase Test Lab
• Support Android applications targeting API 28, API 29, and API 30

In Part 2, we’ll extend this with:

• Supporting Android Test Orchestrator
• Combining results with off-device unit tests
• Supporting multi-module applications

Supporting Android Test Orchestrator

Android Test Orchestrator is a tool that allows us to run on-device tests in independent processes from one another, so that a crashing or misbehaving test cannot pollute the results of other tests.

To better illustrate this, we’ll first give our classes a few more methods, and then add another test case so we can see the Orchestrator at work.

To enable Orchestrator for our tests, we need to make a few changes to our module-level build.gradle

Finally, we’ll build the app (just like in the previous article)

./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest

And execute our tests with our gcloud command by taking the same one from the previous article and adding the --use-orchestrator option as well as the clearPackageData environment variable.

gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables \
clearPackageData=true,\
coverage=true,\
coverageFile=/sdcard/Download/coverage.ec \
--directories-to-pull /sdcard/Download

Then download the resulting coverage file

mkdir app/build/outputs/code_coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/sdcard/Download/coverage.ec app/build/outputs/code_coverage

And generate a report

./gradlew jacocoReport

Opening the report, we see

Wait… why are we only seeing the coverage results from one of our tests?

The problem is that each test in running in its own process, and each process is writing its coverage file to the same location specified in the coverageFile parameter passed to the gcloud command.

So we’ll replace coverageFile with coverageFilePath (note: coverageFilePath must end in / but directories-to-pull must not) so that each individual coverage file will be written to that directory. Our updated gcloud command is now

gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables \
clearPackageData=true,\
coverage=true,\
coverageFilePath=/sdcard/Download/ \
--directories-to-pull /sdcard/Download

Then when we look in the artifacts/sdcard/Download folder we see each coverage file stored separately. We’ll need to update our download command to download all .ec files:

mkdir app/build/outputs/code_coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/sdcard/Download/*.ec app/build/outputs/code_coverage

But our report generation task can stay the same, since it is already looking for all .ec files. Running gradlew jacocoReport, we now get the coverage that we’re expecting:

Generating .exec files for off-device unit tests

Adding off-device unit tests to the mix is comparatively simple. First we’ll add an off-device test to app/src/test/java/com/github128/coverage1/OffDeviceTests.kt:

We can easily run this from the command line:

./gradlew testDebugUnitTest

And a coverage file is generated in app/build/jacoco/testDebugUnitTest.exec.

A warning about testCoverageEnabled in unit tests

If you pass the -Pcoverage option (thereby setting testCoverageEnabled to true, as described in the previous article) then this will generate the .exec file in a different location (app/build/outputs/unit_test_code_coverage/debugUnitTest/testDebugUnitTest.exec ) but more importantly will not create the file properly for library modules, causing issues once we get to multi-module Android projects. (this issue is tracked at https://issuetracker.google.com/issues/210500600)

Combining .ec and .exec files

In any case, now we just need to tweak the getExecutionData().setFrom in our jacocoReport job in our module-level build.gradle to pull coverage files from this new location.

getExecutionData().setFrom(
  fileTree(dir: "${buildDir}/outputs/code_coverage", 
           includes: ['*.ec']),
  fileTree(dir: "${buildDir}/jacoco", 
           includes: ['*.exec'])
)

Now our coverage report will include the calls from our unit tests combined with the calls from our Firebase Test Lab tests. Our entire execution is now

./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables \
clearPackageData=true,\
coverage=true,\
coverageFilePath=/sdcard/Download/\
--directories-to-pull /sdcard/Download
mkdir app/build/outputs/code_coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/sdcard/Download/*.ec app/build/outputs/code_coverage
./gradle testDebugUnitTest
./gradlew jacocoReport

Which will create the desired report of

Multi-module applications: per-module report

There are two parts to supporting code coverage for a multi-module application: coverage reports for each individual module as well as a unified report combining the coverage of all modules.

First we’ll add a second module to our application. Like our application module, this new library module will include a pair of classes with some functions as well as both on-device and off-device tests.

One approach is to essentially replicate what we did before, adding a dependency on the jacoco plugin and a new task to the new module’s build.gradle. A better solution, however, is to refactor the plugin dependency and task into a separate gradle script and then import it into the build.gradle for both our primary module and our new module. The separate script would then be

Then we just need to modify our new library module’s build.gradle by adding

apply from: '../module-jacoco.gradle'

as well as our standard

buildTypes {
    debug {
        testCoverageEnabled (project.hasProperty('coverage'))
    }
}

Similarly, we can edit our original module’s build.gradle by removing the dependency on the jacoco plugin, deleting the jacocoReport job entirely, and adding in the dependency on module-jacoco.gradle.

At this point, we can run the jacocoReport task in each module (after either running the unit tests or downloading coverage reports from Firebase Test Lab). Remember that when running the Firebase Test Lab tests the APK passed to gcloud with the --app parameter should still be the APK generated from the application module. (i.e. app/build/outputs/apk/debug/app-debug.apk)

cd app
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>
cd ../library
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>

And the resulting code coverage reports for each module will accurately reflect the coverage of the unit tests combined with the coverage of the on-device tests that ran in Firebase Test Lab.

Multi-module applications: unified report

We could put all of this into the project-level build.gradle, but as with the individual modules’ gradle scripts, it’s a bit cleaner to put it into its own gradle script and then import it into the project-level build.gradle. We can follow the collect/flatten pattern to build up the desired list of files or directories. The project-level separate script will look like

And we’ll just need to add a

apply from: 'project-jacoco.gradle'

to our the build.gradle in the root of our project. Once this is in place, after going through the steps mentioned above to generate the .exec files for off-device tests and download the .ec files for Firebase Test Lab tests, we just need to go to the root of the project and run

./gradlew jacocoUnifiedReport

and we’ll then see a report for the entire project’s coverage in build/reports/jacoco/jacocoUnifiedReport/html/index.html.

TL;DR

We’ve now achieved the requirements for this article. Looking back, these were the steps we needed to take:

1. Make some small changes to the build.gradle for each module to enable Android Test Orchestrator, and then pass --use-orchestrator and clearPackageData=true to gcloud to instruct Firebase Test Lab to use it.
2. Because Android Test Orchestrator generates multiple .ec files, we’ll also need to change the coverageFile parameter to coverageFilePath and then change our download step to pull *.ec rather than just coverage.ec.
3. To generate the coverage files for off-device unit tests, all we need to do is make sure that the jacoco plugin is included (already done) and that we don’t pass -Pcoverage when running the actual unit tests. (because of https://issuetracker.google.com/issues/210500600)
4. To cleanly support generating per-module reports for multi-module applications, we refactored the jacocoReport task out of each module’s build.gradle and into a common module-jacoco.gradle file.
5. To support a project-wide unified coverage report, we added a new jacocoUnifiedReport task to a project-jacoco.gradle file that is included by the project-level build.gradle.

You can see a solution that combines all of these changes by looking at https://github.com/Aidan128/FirebaseTestLabCoverageExample and checking out the git tag part_two.

Next time

Now we have the ability to generate code coverage reports for unit tests and Firebase Test Lab tests, as well as to combine them across the entire project. This process is rather cumbersome, however, and requires many steps. In Part Three of this series, we’ll look at how this process can be scripted and integrated with Firebase Test Lab automation tools like Flank.

Special thanks

Saravana Thiyargaraj wrote a great article (and built a great example on github) that were very useful. Thank you.

Saravana Thiyagaraj