Bisect pitfalls and recovery — flaky tests, dependencies, side effects, and `bisect reset`

A bisect session that ends with a surprising first-bad-commit is a session that has hit one of four failure modes. The failure modes are not bugs in git bisect - they are properties of the inputs the engineer gave the bisect. The discipline is to recognise the symptoms, recover the session, and fix the input that caused the failure. This lesson enumerates the failure modes, the symptoms, the recovery procedures, and the design discipline that prevents them.

Failure mode 1: flaky tests

A flaky test is a test whose outcome depends on factors other than the source under test. The factors include wall-clock time (a midnight cron running during the test), network latency (a remote service slow at one midpoint, fast at another), shared state (a database that has accumulated data from a prior test), and randomness (a test that uses rand() without a seed). A bisect against a flaky test produces markings that contradict themselves - the same logical regression is marked good at one midpoint and bad at another - and the conclusion is whichever commit the test happened to fail loudest at.

# A bisect that hit a flaky test
git bisect run ./ci-test.sh
# running ./ci-test.sh
# 7e89b40 is the first bad commit
git show 7e89b40
# diff: bump dependency version from 3.2 to 3.3
# (no obvious link to the regression)

The symptom is a first-bad-commit whose diff does not plausibly explain the regression. The recovery is to inspect the bisect log, identify the midpoints where the test failed unexpectedly, re-run those midpoints manually, and confirm the flakiness. The fix is to make the test deterministic: seed the random number generator, mock the network calls, isolate the database, freeze the wall clock. A flaky test is a test that needs fixing, not a bisect that needs re-running.

Failure mode 2: missing build dependencies

A bisect against a build artefact can fail at midpoints where the source compiles against dependencies that no longer exist. The pattern is a commit that updates a dependency to a new major version, and later commits that depend on the new version’s API. Bisecting backwards, the engineer encounters a midpoint where the source is written against the new API but the registry has the old version installed; the build fails before the assertion runs.

# A bisect that hits a build failure
git bisect run sh -c 'make build && ./run-test.sh'
# running sh -c 'make build && ./run-test.sh'
# make: *** No rule to make target build'.  Stop.
# Some revisions cannot be tested, skipping: 7e89b40 bump provider

The symptom is a stream of Some revisions cannot be tested, skipping messages, with a final first-bad-commit identification that is not a true first bad commit but the midpoint where the bisect gave up searching further back. The recovery is to either fix the build at each midpoint (install both versions of the dependency, use a version manager like asdf or mise) or to mark the broken midpoints explicitly with git bisect skip and continue.

Failure mode 3: side-effecting test scripts

A test script that has side effects - writes to a database, pushes a tag, sends a notification, modifies a configuration - corrupts the environment at the midpoint and produces markings that reflect the corruption rather than the regression. The pattern is a test that writes to a shared cache: at the first midpoint the cache is empty, the test passes; at the second midpoint the cache has stale data, the test fails; at the third midpoint the cache has more stale data, the test fails differently. The markings are noise.

# A test with side effects
echo "$(date) $RANDOM" >> /tmp/bisect-state.log
# the file accumulates across midpoints and changes behaviour

The recovery is to clean the state at the start of the script: rm -f /tmp/bisect-state.log as the first line. Or to use a fresh sandbox per midpoint: a Docker container, a temporary directory, a CI runner that is destroyed between runs. The discipline is that the test must be a function from commit to exit code, not a function from commit and accumulated state to exit code.

Failure mode 4: dirty working trees

A bisect against a dirty working tree - uncommitted changes in the index or working directory - produces markings that reflect both the committed source and the uncommitted modifications. The pattern is an engineer who is in the middle of debugging and runs bisect without committing their scratch changes. Each midpoint is checked out on top of the uncommitted changes, and the test sees both.

# The symptom: a bisect that produces contradictory markings
git status
# modified: tests/regression-test.sh
git bisect run ./tests/regression-test.sh
# ... some midpoints pass, some fail, no clear pattern ...

The recovery is to commit the changes (git add tests/ && git commit) or to stash them (git stash) before starting the bisect. The discipline is that the bisect’s working tree must match the commit under test, with no overlay.

git bisect reset and recovery procedures

git bisect reset returns the repository to its pre-bisect state: HEAD moves back to the original branch tip, the .git/BISECT_* files are removed, and the working tree matches the pre-bisect state (modulo any uncommitted changes the engineer had, which are preserved). The reset is the canonical recovery procedure for any bisect session - successful or not.

# Reset after a failed session
git bisect reset
# Previous HEAD position was a3f1c2d... On branch: main

# Reset and replay from a saved log
git bisect start
git bisect replay ~/bisect-sessions/2024-01-15.log
# ... session runs again ...
git bisect reset

git bisect replay <logfile> re-runs a saved session against the current state. The replay is the recovery procedure when the engineer wants to re-bisect on a different branch or after the working tree has been refreshed. Editing the log before replaying is the recovery procedure for a wrong marking: change git bisect bad 9b2a814 to git bisect good 9b2a814, save the edited log, replay.

flowchart LR
    A["bisect concludes\nwrong commit"] --> B["inspect\ngit bisect log"]
    B --> C{"wrong marking\nor wrong test?"}
    C -- "wrong marking" --> D["edit log\nsave as fixed"]
    C -- "wrong test" --> E["fix test\nthen rerun"]
    D --> F["git bisect replay\nfixed log"]
    E --> G["git bisect reset\nthen start fresh"]
    F --> H["verify new\nconclusion"]
    G --> H

The recovery workflow is: identify the failure mode (read the log, look for skips, look for surprising conclusions), pick the recovery procedure (edit and replay for wrong markings, reset and rerun for wrong tests), and verify the new conclusion by reading the diff.

Production discipline

1. Verify the first-bad-commit conclusion before acting. Read the diff. Does it plausibly explain the regression? If not, the bisect hit a failure mode; reset and replay with corrected inputs.
2. Always reset the session. Even if the bisect succeeded, git bisect reset returns the repository to its pre-bisect state. A session that is not reset leaves .git/BISECT_* files in place and HEAD detached.
3. Save the log for every session. The log is the audit trail for the bisect. Save it alongside the bug ticket, the incident report, the PR description.
4. Design tests for resilience. A test that handles missing dependencies (returns 125), cleans state at the start, runs in an isolated sandbox, and does not depend on wall-clock time is a test that produces reliable bisect conclusions.

Cross-course references

• CI/CD Pipeline Patterns - Part XII (FlakyTests) covers the discipline of detecting and remediating flaky tests in CI. The same discipline applies to bisect test scripts; the bisect amplifies flakiness because it runs the test N times.
• Linux for Production Sysadmins - Part XXIV (BisectingKernels) describes the kernel community’s practice of saving the bisect log and attaching it to the bug report, even when the conclusion is wrong - the log is part of the debugging trail.
• Performance Engineering for Production Sysadmins - Part X (ReproducibleRuns) covers the discipline of making tests reproducible. A bisect test is the strictest case: it must be reproducible at arbitrary commits in the history, not just at HEAD.

Quiz