Automated bisect with `git bisect run` — exit codes, test scripts, and the shell wrapper

Manual bisect requires the engineer to run a test at each midpoint, read the result, and mark the commit good or bad. For a 14-step bisect on a 10000-commit history that is fourteen tests, fourteen readings, and fourteen markings - all of them mechanical, all of them error-prone, all of them the same operation repeated. The mechanism that automates the repetition is git bisect run. Given a command, bisect checks out each midpoint, runs the command, reads its exit code, marks the commit good or bad, and moves to the next midpoint. The engineer’s role is reduced to writing the test and starting the session.

The exit-code protocol

git bisect run interprets the command’s exit code according to a fixed protocol. Exit code 0 marks the commit good - the test passed, the regression is not present. Exit codes 1 through 127 (except 125) mark the commit bad - the test failed, the regression is present. Exit code 125 is reserved for “this commit cannot be tested” - the bisect algorithm treats it as an implicit skip and continues without marking the commit. Any other exit code (negative numbers from signals, codes above 127) is treated as a fatal error and the bisect session aborts.

# The protocol, in shell terms
git bisect start
git bisect bad HEAD
git bisect good v3.4.0
git bisect run ./ci-test.sh
# 0        => good, continue
# 1-127    => bad, continue (except 125)
# 125      => skip, continue
# other    => abort
# a3f1c2d is the first bad commit
git bisect reset

The exit-code protocol is the entire interface between the test script and the bisect algorithm. The script does not print anything to stdout that the bisect reads; the algorithm reads only the exit code. A test that prints “PASS” but exits with 1 is marked bad. A test that prints “FAIL” but exits with 0 is marked good. The exit code is the contract.

A clean test script

The discipline of writing a bisect test script is to make the script a single command that exits 0 or non-zero based on the presence of the regression. A clean script has three properties:

#!/bin/sh
# ci-test.sh - exit 0 if the regression is absent, 1 if present

set -e

# 1. Build the thing that the regression is observed in
make build/observability-check

# 2. Run the check that distinguishes good from bad
./build/observability-check \
    --config ./configs/production.yaml \
    --threshold 100ms

# 3. Let the script's exit code reflect the check's exit code
#    `make` and `observability-check` already exit non-zero on failure

The three properties are: deterministic (the same input produces the same output, run after run), side-effect-free (the script does not write to a database, send a notification, push a tag, or modify the working tree beyond the build artefacts), and fast (the engineer will run this script ceiling(log2 N) times - if N is 5000 and each run takes 5 minutes, the bisect takes an hour).

A test that fails any of the three properties is a flaky test. A flaky test produces a bisect that mis-identifies the first bad commit, because the markings are sometimes wrong. The discipline of “fix the test, not the bisect” is the central operational rule of automated bisect.

Multi-line scripts via sh -c

A bisect run command is a single command line. A test that requires multiple steps - configure, build, run, assert - cannot be a single binary invocation. The mechanism for embedding multiple steps is sh -c '<script>', which runs the script in a subshell and propagates the final exit code.

git bisect run sh -c '
    ./configure --enable-strict
    make -j$(nproc) check-tls-handshake
    ./tests/run-regression.sh --case tls-handshake-timeout
'

The sh -c form is the right tool when the test is multi-step and the steps need a shell to compose. The shell variable expansion, the redirect, the conditional - all of them work inside the quoted script. The exit code that bisect reads is the exit code of the last command in the script, which is what the engineer wants.

The cost of sh -c is that errors in the script are harder to debug because bisect runs the script as a black box. The discipline is to test the sh -c script manually before launching it under bisect: run the same sh -c line by hand at HEAD and at the known-good commit, confirm both produce the expected exit code, then launch the bisect.

When automated bisect is the wrong tool

Automated bisect is the right tool when the test is fast, deterministic, side-effect-free, and exits cleanly. Automated bisect is the wrong tool when any of those properties is violated. The cases where it fails:

• Interactive tests. A test that requires human input cannot be automated; bisect will hang at the first midpoint.
• Network-dependent tests. A test that talks to a remote service is non-deterministic - the service might be slow, rate-limited, or down. The bisect’s markings are wrong on those midpoints.
• Stateful tests. A test that depends on prior state (a database connection, a running daemon, a populated cache) will fail on midpoints that did not set up that state.
• Slow tests. A test that takes an hour cannot be run ceiling(log2 N) times. The bisect’s runtime is test_runtime
  • step_count; if test_runtime is an hour and step_count is 14, the bisect takes 14 hours.

For each of these cases, the right tool is manual bisect (the engineer reads the output and marks each midpoint) or a different debugging approach (read the diff, profile the build, search the issue tracker).

Production discipline

1. Test the bisect command manually before launching. Run git bisect run <cmd> at the known-good and known-bad commits first. If the test exits the expected codes at both, launch the session. If it does not, fix the test.
2. Use a fresh working tree or a CI runner for the bisect. A bisect that runs against a dirty working tree picks up uncommitted changes that contaminate every midpoint’s test. Either commit the dirty changes first or use a clean checkout.
3. Bound the test runtime. A bisect against a 5-minute test takes ceiling(log2 N) * 5 minutes. If N is 10000, that is 70 minutes. If the runtime is longer than the engineer is willing to wait, find a faster test.
4. Treat a bisect that produces a surprising first-bad-commit as a test problem. A surprising conclusion (a routine dependency bump, a doc change) is a sign that the test is non-deterministic at some midpoints. Re-run with a more deterministic test or use git bisect log to inspect the markings.

Cross-course references

• CI/CD Pipeline Patterns - Part III (TestDesign) covers the design of tests that are deterministic and side-effect-free. The bisect test is a special case: it must be runnable headlessly at arbitrary commits, which is the strictest case of a CI test.
• Shell Scripting for Production Sysadmins - Part VII (ExitCodes) covers the POSIX exit-code conventions that the bisect run protocol depends on. A bisect test that returns 0 on success and non-zero on failure is the same convention as a CI job.
• Ansible for Production Sysadmins - Part XXII (MoleculeAndCI) covers the Molecule test pattern, which is exactly the kind of script a bisect run command targets: a multi-step shell that builds, tests, and exits.

Quiz