Coveralls Builds

It’s important to know a little bit about Coveralls builds and build types in order to understand how to configure your CI service to work best with Coveralls:

• Introduction to Coveralls Builds
• Coveralls Build Types:
  • Push Builds
  • Pull Request Builds
    • Common Issue: “Drifted” PR Builds
  • Parallel Builds
• Recommended CI Configurations

Introduction

Coveralls plugs into into your CI/CD toolchain.

Once configured to use Coveralls:

• Your CI service sends coverage reports to Coveralls for each CI build of your project.
• Coveralls stores these reports as Coveralls builds and tracks changes in their coverage details over time.

Coveralls Builds ≈ CI Builds

Because of this close relationship to your CI builds, Coveralls uses the same metaphor of a “build” to represent its primary unit of coverage reporting.

However, a Coveralls build is not identical to a CI build. Instead, it’s a highly detailed coverage report for a CI build.

Over time, Coveralls stores a history of coverage reports matching the history of your CI builds.

Which Builds = Configurable

It’s possible to send coverage reports for some CI builds and not others, but Coveralls works best when you send it coverage reports for all your CI builds.^*

We make some recommendations below regarding the ideal CI configuration for reporting coverage to Coveralls; but first, it helps to know about the different types of Coveralls builds in order to understand those recommendations.

^* At least for all your push builds. See below for Recommended CI Configuration.

Coveralls Build Types

If you’re familiar with CI services, you’ll be familiar with the two types of builds they generate:

• push builds - Builds of git push commits
• pull request builds - Builds of git pull_request commits

Coveralls mirrors these same two build types and we also refer to them as push builds and pull request builds.

But since Coveralls builds are coverage reports, let’s discuss how coverage reports can differ for these two types of builds, and what to consider when configuring your CI service to send coverage reports for them.

Push Builds

In Coveralls, push builds are simply coverage reports for a push commit your CI service has built in your CI environment.

As with push builds in CI, Coveralls push builds are the most straightforward type of build because they represent coverage for a single, historical commit on a branch of your repo’s git tree.

Tracking coverage changes for push builds

One of the key value propositions of Coveralls is tracking how coverage changes on your project over time.

In the case of push builds, Coveralls does this by comparing each push build to the previous push build on the same branch:

Δ coverage = push build - previous push build (on same branch)

In fact, Coveralls can compare coverage between any two push builds on your repo, which makes sending coverage reports for all your push builds the most flexible way to configure Coveralls.

Pull Request Builds

In Coveralls, pull request builds are coverage reports for a pull_request commit your CI service has built in your CI environment.

In contrast to push builds, pull request builds are more complicated because they are based on a temporary commit on an ephemeral branch of your repo called the merge branch, making these builds prospective builds, rather than actual, historical ones, like push builds.

Tracking coverage changes for pull request builds

Generating accurate coverage reports for pull request builds can be tricky due to the way most CI services generate these prospective builds for most use cases.

For now, just know that Coveralls calculates the coverage changes for a pull request build by comparing the report for its current head commit (aka. the pull request build) to the report for its base commit, otherwise known as its base build:

Δ coverage = pull request build - base build (on target branch)

Details:

• A PR’s head commit is the latest commit on the PR branch.
• A PR’s base commit is the point where the PR first branched off from its target branch, also referred to as the latest common ancestor between the PR and target branches.

COMMON ISSUE: “DRIFTED” BUILDS & TWO-DOT VS. THREE-DOT DIFF COMPARISONS IN COVERAGE REPORTS

As mentioned above, accurately calculating coverage reports for pull request builds is challenging, and the reason is the reliance on two-dot diff comparisons in generating pull requests, whereas three-dot diffs are necessary to calculate precise PR coverage.

Key Concept:

• A two-dot diff comparison becomes equivalent to a three-dot diff when the PR’s base commit matches the HEAD commit of the target branch, ensuring accurate coverage reports:

  base commit (PR branch) = HEAD commit (target branch) → accurate coverage report
  

  
  

• Inaccuracy arises when the PR’s base commit diverges from the target branch’s HEAD commit:

  base commit (PR branch) ≠ HEAD commit (target branch) → inaccurate coverage report
  

This second scenario leads to what we call “drifted” builds:

A drifted build alert in Coveralls. When your PR build has drifted, this warning message will appear at the top of your Build Page.

Solutions for Drifted Builds:

1. Rebase the PR: Integrating the latest changes from the target branch into your PR’s commit history realigns the PR’s base with the target’s HEAD commit.
2. Modify CI Configuration: Adjust your CI to send Coveralls coverage reports for all push commits on PR branches. Coveralls can then accurately compare coverage between the PR’s HEAD and the merge base.

With this configuration, sending Coveralls reports for pull_request builds becomes optional. Coveralls will treat push builds on your PR branches as if they were pull_request builds, maintaining accuracy even if builds drift.

Further Insights: The following section delves into the nuances of two-dot and three-dot diffs, helping you understand and address drifted build alerts in Coveralls.

Two-Dot-Diff Comparisons (Default PR Build)

In git, a two-dot diff comparison shows the difference between the latest commit on the PR branch and the latest commit on the target branch. In other words, the current heads of both branches.

This is useful for understanding what the impact of a PR will be when it’s merged into the target branch, but it’s not as helpful for understanding the coverage changes introduced by a PR.

Three-Dot-Diff Comparisons (Accurate PR Coverage Report)

In git, a three-dot diff comparison shows the difference between the latest commit on the PR branch (also known as the PR head) and the latest common ancestor of both the PR and target branches (also known as the merge base).

In this way, a three-dot diff comparison focuses on just “what a pull request introduces,” and therefore, on just the coverage changes introduced by a pull request.

You’ll recall, this is the same kind of diff Coveralls wants to make to calculate the change in coverage for a pull request build:

Δ coverage = pull request build - base build (on target branch)

Configuring your CI service for accurate PR coverage reports

We recommend configuring your CI service in a manner that allows Coveralls to generate three-dot diff comparisons for all of your pull request builds, so we can always guarantee accurate coverage reports for them.

three-dot diff = changes introduced by a PR

That’s because looking at a coverage report for a two-dot diff comparison can be misleading, since it shows the coverage changes introduced by the PR, plus all changes that have occurred on the target branch since the PR was opened.

two-dot diff = changes introduced by a PR + changes on target branch since PR was opened

Since those latter changes aren’t relevant to the PR’s changes, they can make it difficult to understand the coverage changes introduced by the PR.

For details on how to configure your CI service to generate accurate coverage reports for pull_request builds, see our Recommended CI Configurations.

Parallel Builds

Parallel builds are a crucial build type in Coveralls, complementing the primary push and pull request builds. These builds are not mutually exclusive with push or pull request builds; any push or pull request build can also function as a parallel build.

Definition:

A parallel build aggregates multiple coverage reports into a single report for a specific commit. These individual reports are typically generated by various jobs or steps within a CI process, running concurrently. Each job covers a different segment of the project, and their reports are independently sent to the Coveralls API.

Advantages:

Parallel builds are particularly beneficial for large projects that can be segmented into smaller, independent parts. By analyzing coverage reports from these segments separately and then combining them, a comprehensive and unified coverage report for each build is produced. This approach enhances efficiency and can provide more detailed insights into the coverage of different parts of the project.

Setting Up Parallel Builds

To leverage parallel builds in Coveralls, you need to configure them within your CI setup, specifically in the workflow steps that interact with Coveralls.

Detailed Configuration Guide:

The complete setup process is outlined here.

Integration with Core Build Types:

Configuring a parallel build essentially involves expanding a core build type (push or pull request) into multiple steps. If your project requires parallel builds, this configuration applies to all builds, regardless of whether they are push or pull request builds.

Considerations Based on Build Types:

• If your project sends coverage reports for both push and pull request builds, then both types of workflows need to be adapted for parallel builds.
• If your workflows are distinct for push and pull request builds, each must be individually configured for parallel processing.
• In cases where only push build reports are sent, only the push workflow requires extension for parallel builds.

Recommended CI Configurations

Given that it can be difficult to guarantee accurate coverage reports for pull request builds—since they may be based on two-dot diff comparisons instead of three-dot diff comparisons—below are our recommended CI configurations to reliably receive accurate coverage reports at Coveralls.

Beyond stating our ideal and minimum recommendations, for clarity we’ve also listed these recommendations by build type.

K.I.S.S. Recommendation

Following the K.I.S.S. principle, here’s our current recommendation in the simplest possible terms:

• Build all push builds, on all branches, and send coverage reports for them to Coveralls.

Ideal CI Configuration

• Build all push builds, on all branches, and send coverage reports for them to Coveralls.
  • Optional: Build all pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.

Minimum CI Configuration

• Build all push builds on your default branch, and send coverage reports for them to Coveralls. Build all pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.
  • Required: Build all pull_request builds, on all pull request branches targeting your default branch, and send coverage reports for them to Coveralls.

Sub-Minimum CI Configuration

• Will result in errors:
  • Not building pull_request builds on pull request branches targeting your default branch, when you are also not building push builds on your pull request branches, will result in an error:
    • “Coveralls can’t generate coverage reports for your PRs right now.”

By Build Type

• Push Builds:
  • Recommended: Build all push builds on all branches and send coverage reports for them to Coveralls.
  • Required: Build all push builds on your default branch.
• Pull Request Builds:
  • If you are building all push builds on all branches (recommended):
    • Optional: Build all pull_request builds on PR branches and send coverage reports for them to Coveralls.
  • If you are only building push builds on your default branch (required):
    • Required: Build all pull_request builds on all PR branches targeting your default branch and send coverage reports for them to Coveralls.
• Parallel Builds:
  • Required: If your project requires parallel builds, you must configure parallel builds for all builds that send coverage reports to Coveralls.
    • Details: The process is fully described here.

Any problems, questions or comments about this doc? Let us know.