It’s important to know a little bit about Coveralls builds and build types in order to understand how to configure your CI service to work best with Coveralls:
Coveralls plugs into into your CI/CD toolchain.
Simply stated, when your CI service creates builds of your project, it sends coverage reports for those builds to Coveralls, which does stuff with them.
Because of this close relationship to your CI builds, Coveralls uses the same metaphor of “builds” to represent it’s primary units of output.
However, a Coveralls build is not identical to a CI build, of course. Instead, it’s a highly detailed coverage report for a CI build.
Coveralls will process and store a coverage report for each CI build of your project, for which you’ve sent it a report.
In this way, Coveralls tracks a history of coverage reports matching your history of CI builds.
It’s possible to send coverage reports for some CI builds and not others, but Coveralls works best when you send it coverage reports for all of your CI builds.*
We make some recommendations below regarding the ideal CI configuration for Coveralls coverage reports; but first, it helps to know about the different types of Coveralls builds in order to understand those recommendations.
* At least for all your push builds. See below for Recommended CI Configuration.
If you’re familiar with CI builds, you’ll be familiar with the two primary types of builds that are built in CI: push builds and pull request builds.
Coveralls mirrors these same two build types, and we also refer to them as push builds and pull request builds.
Since Coveralls builds are coverage reports, let’s discuss how coverage reports can differ for these two types of builds, and what to consider when configuring your CI service to send coverage reports to Coveralls.
In Coveralls, push builds are simply coverage reports for a push commit that your CI service has built in your CI environment.
As with push builds in CI, Coveralls push builds are the most straightforward type of build because they represent coverage for a single, historical commit on an actual branch of your repo.
push buildsTracking how coverage changes on your project over time is a key value proposition of Coveralls, which, in the case of push builds, Coveralls does by comparing each push build to the previous push build on the same branch.
In fact, Coveralls can compare coverage between any two push builds on your repo, which makes sending coverage reports for all of your push builds the most flexible way to configure Coveralls.
In Coveralls, pull_request builds are coverage reports for a pull_request commit that your CI service has built in your CI environment.
In contrast to push builds, pull_request builds are more complicated because they represent a temporary commit on an ephemeral branch of your repo, called the merge branch, making these builds prospective builds, rather than actual, historical ones, like push builds.
pull_request buildsGenerating accurate coverage reports for pull_request builds can be tricky, mainly due to the way most git hosts and CI services generate these prospective builds for the most typical use cases.
For instance, by default, GitHub generates pull_request commits using a two-dot diff comparison, which shows the difference between the latest commit on the PR branch, and the latest commit on the target branch.
This is useful for understanding the impact of a PR when it’s merged into the target branch, but it’s not as useful for understanding the coverage changes introduced by a PR.
A three-dot diff comparison, on the other hand, shows the difference between the latest common commit of both PR and target branches (the “merge base”) and the most recent version of the PR branch, which is what most users want to see when they’re viewing the coverage report for a PR.
In other words, a three-dot diff comparison focuses on just “what a pull request introduces,” and, therefore, just the coverage changes associated with that pull request.
We recommend configuring your CI service in a manner that allows Coveralls to generate three-dot diff comparisons for your pull_request builds, so we can always guarantee accurate coverage reports for them.
That’s because looking at a coverage report for a two-dot diff comparison can be misleading, since it shows the coverage changes introduced by the PR, plus all changes that have occurred on the target branch since the PR was opened.
Since those latter changes aren’t relevant to the PR’s changes, they can make it difficult to understand the coverage changes introduced by the PR.
For details on how to configure your CI service to generate accurate coverage reports for pull_request builds, see our Recommended CI Configurations.
While push builds and pull_request builds represent the core Coveralls build types, there’s one more important Coveralls build type to understand, which is Parallel Builds.
Parallel builds and push / pull_request builds are not exclusive of one another, which is to say that any push or pull_request build can also be a parallel build.
A parallel build is a build that represents the coverage for a single commit, but which is generated by multiple CI jobs running in parallel, each sending its own coverage report for a different part of the same project.
Parallel builds are useful for projects which can be broken up into smaller parts, whose coverage reports can ultimately be combined into a single coverage report for the entire project.
Parallel builds must be configured in your CI setup, in the workflow steps related to Coveralls.
The process is fully described here.
As it pertains to the other build types described here, you can think of configuring a parallel build as extending one of the core build types into multiple steps; and, if your project requires parallel builds, then all of its builds need to be parallel, whether they are push builds or pull_request builds.**
** This assumes you’re sending coverage reports for both types of core builds: push and pull_request. If you have separate workflows for push builds and pull_request builds, then both workflows must be extended into parallel builds. On the other hand, if you’re only sending coverage reports for push builds, then you’ll only need to extend your push workflow.
Given that it can be difficult to guarantee accurate coverage reports for pull_request builds—since they may be based on two-dot diff comparisons instead of three-dot diff comparisons—below are our recommendations on the ideal CI configurations to reliably receive accurate coverage reports at Coveralls.
Beyond stating our ideal and minimum recommendations, for clarity we’ve also listed recommendations by build type.
Following the K.I.S.S. principle, here’s our current recommendation, in the simplest possible terms:
push builds, on all branches, and send coverage reports for them to Coveralls.push builds, on all branches, and send coverage reports for them to Coveralls.
pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.push builds on your default branch, and send coverage reports for them to Coveralls. Build all pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.
pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.pull_request builds on pull request branches targeting your default branch, when you are also not building push builds on your pull request branches, will result in an error:
push builds, on all branches, and send coverage reports for them to Coveralls.push builds on your default branch.pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls. (Required: If only sending push builds on your default branch.)pull_request builds on all pull request branches.pull_request builds on all branches.
Any problems, questions or comments about this doc? Let us know.