Code Coverage Guide
The Venice repository has two code coverage check on GitHub Action, submodule-level coverage verification and new-commit coverage verification, which aim to improve the testability and code quality overall. The GitHub Action fails the Pull Request if the coverage check doesn’t meet the requirements. This guide provides the details on how these reports are generated and how to debug them.
Venice has 4 modules,
services. Each module has its own submodules, except
Submodule-level coverage verification
This will check the overall code coverage at the submodule level. Jacoco generates the report based off the current submodule, performs the coverage verification and fails the build if the coverage is below the threshold. The threshold, targets at the branch coverage, is defined in the
build.gradle of each submodule independently.
# Template ./gradlew :<module name>:<submodule name>:jacocoTestCoverageVerification ./gradlew :clients:venice-push-job:jacocoTestCoverageVerification
The report will be located at
<module name>/<submodule name>/build/reports/jacoco/test/index.html.
Run commands against module doesn’t generate the report
Due to the current project setup, running the commands, e.g.
./gradlew :clients:jacocoTestCoverageVerification, doesn’t execute the unit tests thus no jacoco report will be generated. Please be sure to run the commands against the submodule.
New-commit coverage verification
This will check the overall code coverage at the commit-level. DiffCoverage, which is an extension of Jacoco, gathers the diff files by comparing the local branch and remote upstream main branch, and leverages the Jacoco report, to re-generate a new report only for these newly added lines/files. Similarly, it performs the coverage verification and fails the build if the coverage is below the threshold. The threshold is defined at 60% for branch coverage.
# Template ./gradlew :<module name>:<submodule name>:jacocoTestCoverageVerification diffCoverage --continue ./gradlew :clients:venice-push-job:jacocoTestCoverageVerification diffCoverage --continue
The report will be located at
Integration tests are added but DiffCoverage doesn’t identify the code coverage of it
Though integration tests are strongly encouraged, both Jacoco and DiffCoverage only work with unit tests so please consider writing unit tests.
Unit tests are added but the DiffCoverage report doesn’t reflect my changes
There are two possible reasons.
- Jacoco report isn’t up-to-date. DiffCoverage report relies on the Jacoco Report to reflect the correct coverage. If the Jacoco report isn’t up-to-date, for example, executing the wrong command
./gradlew :clients:venice-push-job:diffCoveragewhich misses the step of re-running unit tests and updating the Jacoco report, can cause this issue.
- Please be sure to run test and Jacoco report first.
- Units tests are placed in a different module. Jacoco can only analyze and generate reports based off the current submodule. So, if you write new source codes in
submodule B, and you only have written unit tests in
submodule Bwhich cover changes in
submodule A, this cannot be recognized by Jacoco thus the DiffCoverage doesn’t think there’s coverage too.
- Please move/re-organize some unit tests to the right submodule such that the coverage can be detected and reported.
The DiffCoverage report shows some files don’t belong to my local changes
That’s usually due to your local branch and upstream is not up-to-date so when it runs
git diff, newly merged codes in
linkedin/venice are mistakenly treated as your changes.
Please do the followings:
- Go to your Github fork(https://github.com/
/venice) and sync the `main` branch with upstream `linkedin:main`
git fetch upstreamlocally and pull the latest changes to your
mainbranch to your feature branch.
- Confirm the diff are only your changes by running
git diff upstream/main.