Bisect and build artefacts — finding the commit that broke the build

The most common bisect target in an infrastructure repository is not a unit test or a single function - it is a build: a container image, a binary, an Ansible collection tarball, a Terraform provider plugin, a Go module, a Python wheel. The regression manifests as a built artefact that does something wrong at runtime (crashes on startup, fails a smoke test, omits a flag), and the engineer wants to identify the commit whose source produced the broken artefact. The mechanism is the same as any other git bisect run session: a test script that builds the artefact, asserts on its behaviour, and exits 0 or 1. The discipline is in writing the script so it is fast enough to repeat and deterministic enough to be trusted.

The build-as-test pattern

The canonical pattern for bisecting build artefacts is a script that performs three steps at each midpoint. The first is build: invoke the build system to produce the artefact. The second is assert: run the artefact or a smoke test against it and check the result. The third is exit: return 0 if the assertion passed, 1 if it failed.

#!/bin/sh
# bisect-build-test.sh - exit 0 if the artefact is good, 1 if bad
set -e

make -j$(nproc) build/observability-collector
./build/observability-collector \
    --config ./configs/production.yaml \
    --self-test

The set -e at the top causes the script to abort on the first failure - if the build itself fails (a compilation error, a missing dependency), the script returns non-zero immediately. The --self-test flag is the assertion: the artefact runs against a known configuration and exits 0 if its invariants hold, non-zero if they do not. The combination produces a script whose exit code is the answer the bisect needs.

# The bisect session using the script
git bisect start
git bisect bad HEAD
git bisect good v3.4.0
git bisect run ./bisect-build-test.sh
# ...
# a3f1c2d is the first bad commit

The build-as-test pattern generalises to any artefact whose production is driven by a script. A Terraform plan can be the target (terraform plan -out=plan.bin && terraform show plan.bin followed by a grep for an unexpected diff). A container image can be the target (docker build -t bisect-target . && docker run --rm bisect-target ./smoke-test.sh). A Helm chart can be the target (helm template ./chart | yq '. | select(.kind == "Deployment")' | diff - expected.yaml).

Minimising the test script

The runtime of the bisect is build_time * step_count. For a session that takes 12 steps against a 5-minute build, the bisect takes an hour. The engineer cannot afford a long build at every midpoint. The discipline is to make the test script only as expensive as it needs to be to surface the regression.

flowchart LR
    A["Full build\n5-30 minutes"] --> B["Reduced build\nonly the affected target"]
    B --> C["Target-level test\nonly the failing assertion"]
    C --> D["Smoke test\nfast invocation of the artefact"]

The reductions to consider:

• Build only the affected target. If the regression is in a single Go package, go test ./pkg/metrics/... rather than make build-all. The bisect only needs the artefact that exhibits the regression.
• Skip dependencies that are stable. If a dependency has not changed in the candidate range, it does not need to be rebuilt. Use a build cache (go build -mod=readonly, ccache, Docker layer cache).
• Replace full integration tests with focused unit tests. A smoke test that exercises only the failing code path is faster than an integration test that exercises everything.

The discipline of minimisation is to write the test script backwards: start with the assertion, find the smallest build that produces the artefact the assertion needs, then find the smallest subset of the build that exercises the failing code path. The bisect is only as fast as the test, and the test is only as fast as its slowest step.

Bisecting binary artefacts

The build-as-test pattern requires the build to be reproducible at every midpoint. When the source has changed in ways that break reproducibility (a moved file, a renamed import, a deleted Makefile target), the build fails before the assertion runs, and the midpoint is marked bad - incorrectly. The alternative pattern, artefact-level bisect, decouples the build from the bisect by using a registry of pre-built artefacts.

# Artefact-level bisect using a registry
git bisect start
git bisect bad HEAD
git bisect good v3.4.0
git bisect run sh -c '
    SHA=$(git rev-parse HEAD)
    docker pull "registry.acme.io/observability:${SHA}"
    docker run --rm "registry.acme.io/observability:${SHA}" \
        --self-test
'

The artefact-level bisect assumes the registry contains a build for every commit in the candidate range. The build is performed once, by a separate CI pipeline that runs against every commit; the bisect only pulls and runs the pre-built artefact. The cost is a CI pipeline that runs N times during the candidate range; the benefit is a bisect that runs in seconds per step.

The artefact-level pattern is the right tool when:

• The build is slow (multi-hour compilations, large container images).
• The candidate range is small (a recent regression in a feature branch).
• The CI infrastructure can build against arbitrary commits (a CI pipeline with on-demand builds).

The artefact-level pattern is the wrong tool when the registry does not contain builds for the candidate range, when the artefact is environment-dependent (different build hosts produce different artefacts), or when the regression is in the build process itself (the build at the bad commit is broken; no artefact was produced).

Mapping to a CI pipeline

The bisect can run unattended inside a CI pipeline. The pattern is to push the regression-hunting commit, let the CI run the bisect, and post the conclusion as a PR comment or a build artefact. The pipeline implements the four phases of a bisect session as build steps.

# A CI job that runs a bisect unattended
stages:
  - bisect
bisect-job:
  stage: bisect
  script:
    - git bisect start
    - git bisect bad $CI_COMMIT_SHA
    - git bisect good $LAST_GREEN_SHA
    - git bisect run ./bisect-build-test.sh | tee bisect.log
    - FIRST_BAD=$(grep "is the first bad commit" bisect.log | awk '{print $1}')
    - echo "First bad commit: $FIRST_BAD"
    - git bisect reset
  artifacts:
    paths:
      - bisect.log

The CI mapping has three benefits. The first is that the bisect runs in a clean environment - no dirty working tree, no leftover state from prior runs. The second is that the bisect log is preserved as a build artefact for postmortem analysis. The third is that the conclusion is posted as a CI output, which can be consumed by the bug tracker or the deploy pipeline.

Production discipline

1. Minimise the test script before starting the session. A 30-minute build produces a 7-hour bisect against a 10000-commit history. Minimise first, then start.
2. Cache build artefacts between midpoints when possible. A ccache, a Go build cache, or a Docker layer cache can reduce the 5-minute build to a 30-second rebuild. The bisect runtime drops accordingly.
3. Prefer artefact-level bisect for slow builds. A pre-built artefact registry turns a 30-minute build into a 5-second pull. The trade-off is a separate CI pipeline that builds against every commit in the range.
4. Treat the build failure itself as a data point. If the build fails at a midpoint, the bisect marks that midpoint bad. The engineer should distinguish a build failure (the source did not compile) from a test failure (the source compiled but the artefact misbehaved) - they have different conclusions.

Cross-course references

• CI/CD Pipeline Patterns - Part IV (BuildOptimisation) covers the caching and minimisation patterns that make build-as-test bisect feasible. A bisect session is a sequence of N builds; the cost-reduction patterns apply to each.
• Container Workloads for Production Sysadmins - Part VI (ImageLayerCaching) covers the layer cache that makes artefact-level bisect fast against Docker images.
• Linux for Production Sysadmins - Part XXIV (BisectingKernels) describes the kernel community’s bisect practices against pre-built kernel packages, which is the canonical example of artefact-level bisect.

Quiz