Skip to main content
RunBook Academy

Git, CI/CD & GitOpsXXIV · BisectBisect

What bisect does — binary search through commit history for the offending commit

Advanced⏱ ~20 mingit

What you'll learn

  • Explain what regression hunting means in a version-controlled repository
  • Identify the three things a bisect needs to start: a known-bad commit, a known-good commit, and a test that distinguishes them
  • Describe the bad/good marking protocol and the four phases of a bisect session
  • Recognise when manual bisect is appropriate and when it should be automated
  • Map the bisect mental model to other binary-search debugging patterns

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

A regression has been found in production. The build that ran yesterday worked; the build that ran this morning is broken. Between those two points sits a chain of commits - some merge commits, some feature commits, some dependency bumps - and the question “which commit introduced the regression?” is the question an engineer has to answer before any of the others (“who wrote it?”, “should we revert?”, “is the fix in this commit or downstream?”) make sense. The tool that answers the question is git bisect. This lesson establishes what bisect does, what it needs to start, and how its four-phase lifecycle works.

The regression problem

A regression is a behaviour change introduced by a commit. The change might be a new line of code, a removed default, a bumped dependency, a renamed variable, or a missed side effect in a build step. The change might be deliberate (a feature that broke a workflow) or accidental (a typo in a config that survived review). What the engineer needs is the commit, not the cause: knowing the commit narrows the diff to a few lines, names the author, points at the pull request, and usually surfaces the fix.

# Before bisect: a linear history where something broke
git log --oneline v3.4.0..HEAD
# a3f1c2d update IAM role trust policy
# 7e89b40 bump terraform-provider-aws to 5.31.0
# 1c0d3f5 add S3 lifecycle rule for cold storage
# 9b2a814 merge feature/observability into main
# 8c44d02 rotate KMS keys for production

The engineer’s mental model is “something in here is bad”. A linear scan of the diff is the wrong tool: there might be 200 commits between the last good build and the current broken build, each with a non-trivial diff, and reading 200 diffs is not an acceptable investigation. The right tool is a binary search that halves the candidate set on every step.

The bad/good marking protocol

Bisect needs three things to start. The first is a known-bad commit: a commit where the regression is observable (the current build, a tagged release, a specific SHA). The second is a known-good commit: a commit where the regression is not observable (the last green CI run, a tagged release that shipped clean, a SHA from a clean checkout). The third is a test that distinguishes the two - something whose output the engineer can read and decide “good” or “bad” from.

The protocol is two commands. git bisect bad marks the commit that exhibits the regression. git bisect good <commit> marks the commit that does not. Once both are recorded, Git computes the ancestor range between them and chooses a commit roughly halfway through, checks it out, and waits for the engineer to mark it good or bad.

flowchart LR
    A["git bisect bad\ncurrent HEAD"] --> Q["git bisect start"]
    Q --> R["range: bad..good (ancestor side)"]
    R --> S["git bisect good KNOWN_GOOD_COMMIT"]
    S --> T["Git picks midpoint\nchecks it out"]
    T --> U["engineer marks\ngit bisect good or bad"]
    U --> V{"more commits?"}
    V -- yes --> T
    V -- no --> W["first bad commit\nis identified"]

The engineer does not need to mark every commit individually - the engineer marks one commit per bisect step, and Git uses the mark to halve the remaining range. The role of the engineer is reduced from “read every diff” to “answer good-or-bad at each midpoint”.

The four phases of a bisect session

A complete bisect session has four phases. The first is start: git bisect start enters bisect mode and records the session state in .git/BISECT_* files. The second is seed: the engineer records the bad and good boundaries, either as bare git bisect bad (uses HEAD as the bad boundary) or as git bisect bad <commit> and git bisect good <commit>. The third is step: Git picks a midpoint, the engineer runs the test, marks good or bad, and the cycle repeats. The fourth is conclude: Git identifies the first bad commit, prints it, and returns to the original HEAD on git bisect reset.

# The four phases, in shell
git bisect start
# Bisecting: 0 revisions left to test after this (roughly)
git bisect bad HEAD
git bisect good v3.4.0
# Bisecting: 47 revisions left to test after this (roughly)
# ... engineer runs the test at each step ...
# 8c44d02 is the first bad commit
git bisect reset
# Previous HEAD position was 8c44d02... On branch: main

The fourth phase - the conclusion - is the moment Git prints “<sha> is the first bad commit”. The conclusion is deterministic given the markings. If the engineer marks good/bad consistently with the truth, Git will identify the unique commit in the marked range whose parent is good and whose own tree exhibits the regression. The conclusion is the deliverable.

Manual versus automated

There are two modes of bisect. The manual mode - the engineer marks each midpoint good or bad by running the test themselves - is appropriate when the test is interactive, when the engineer wants to read the test output for clues, or when the test cannot be scripted. The automated mode - git bisect run &lt;cmd&gt; - is appropriate when the test can be expressed as a command whose exit code is 0 for good and non-zero for bad. The automated mode is the topic of lesson 03 in this part.

Production discipline

  1. Always start with git bisect start, always end with git bisect reset. A bisect session that does not end with a reset leaves the repository in bisect mode and the HEAD detached. The discipline is mechanical: open the session, run the session, close the session, even if the session was a false start.
  2. Pin the known-good boundary to something verifiable. A “I think it was working last Tuesday” good boundary is not a good boundary. A green CI run, a tagged release, a passing E2E test - those are verifiable. Bisect cannot fix a fuzzy boundary; it can only amplify one.
  3. Mark the known-bad boundary at the current state. git bisect bad with no argument uses HEAD. This is the convention: the regression is in the present, the bisect looks backward.
  4. Treat the first bad commit as a hypothesis, not a verdict. Bisect’s conclusion is correct given the markings, but the markings are observations, and observations can be wrong (the test is flaky, the boundary was mis-recorded). Verify the first bad commit by reading its diff before assuming it is the cause.

Cross-course references

  • CI/CD Pipeline Patterns - Part VII (FailureAnalysis) covers the patterns for which bisect is the answer: nightly E2E goes red, perf regression lands in staging, integration tests fail on main. The bisect workflow is the same in each case.
  • Linux for Production Sysadmins - Part XXIV (BisectingKernels) describes the canonical use of git bisect against the Linux kernel, which is where the tool’s name comes from. The workflow applies identically to an infrastructure repository.
  • GitOps with Argo CD - Part IX (DriftDetection) covers the case where the diff between expected and actual is a commit whose effect was not noticed until it shipped. Bisect is the tool that identifies it.

Quiz

Knowledge check · 4 questions

  1. Q1. What three things does `git bisect` need to start a session?

  2. Q2. A bisect session that ends without `git bisect reset` leaves the repository in bisect mode with HEAD detached at the last midpoint.

  3. Q3. Name the four phases of a bisect session, in order.

  4. Q4. Identify what is missing from a bisect setup that will produce an inconclusive result, and explain the consequence.

    An engineer runs `git bisect start` then `git bisect bad` (marking HEAD as bad). They then pick a commit from two months ago as 'definitely good' and run `git bisect good &lt;two-months-ago-sha&gt;`. The bisect runs for many steps, every step marking bad, and eventually reports a first bad commit that is itself two months old. Inspection of that commit shows a routine dependency bump with no obvious link to the regression.

Passing score: 75%. Answers are checked in this browser.