Lab 6: Use `git bisect run` to identify the breaking infrastructure commit

Objective

By the end of this lab you will have built a history of twenty commits that each introduce a small Terraform change, planted a regression at one specific commit, and used git bisect run to find that commit in O(log n) steps. The script that decides good-or-bad will be a plain shell script — not a Terraform run, not a cloud provider call, not a complex test framework — and the point of that choice is that git bisect run works with anything that exits 0 or exits non-zero. That property is what makes bisect useful for infrastructure problems where the “test” is a terraform validate, a JSON schema check, or a YAML lint.

The point of this lab is not “Terraform bisecting” — it is “any bisect, applied to a problem where the answer is in the commit history”. The Terraform content is incidental.

Architecture

A linear history of twenty commits. Each commit either keeps the file valid or introduces a single line that breaks validation. One commit — at a deliberately chosen position — introduces the regression.

flowchart LR
    G0["C0 · initial valid"] --> G1["C1 · valid"]
    G1 --> G2["C2 · valid"]
    G2 --> G3["C3 · valid"]
    G3 --> G4["C4 · valid"]
    G4 --> G5["C5 · valid"]
    G5 --> G6["C6 · valid"]
    G6 --> G7["C7 · valid"]
    G7 --> G8["C8 · valid"]
    G8 --> G9["C9 · valid"]
    G9 --> G10["C10 · valid"]
    G10 --> G11["C11 · BUG"]
    G11 --> G12["C12 · broken"]
    G12 --> G13["C13 · broken"]
    G13 --> G14["C14 · broken"]
    G14 --> G15["C15 · broken"]
    G15 --> G16["C16 · broken"]
    G16 --> G17["C17 · broken"]
    G17 --> G18["C18 · broken"]
    G18 --> G19["C19 · broken"]

The bisect algorithm walks a binary search across the linear chain. With twenty commits, the search converges in at most ceil(log2(20)) = 5 test invocations to identify C11 as the first bad commit.

Requirements

Git 2.55.x on Linux or macOS.
bash or sh for the test script. The lab’s test.sh is POSIX-compatible.
A clean working directory. Nothing outside $HOME/bisect-lab is touched.
No network access. No real Terraform installation is required; the “test” is a text-level check.

Scenario

Production started reporting that the network module’s terraform validate exits non-zero with a syntax error on the CIDR attribute. You bisect the network module’s history and find the commit. The “test” you wrote for the bisect is a terraform validate invocation that returns 0 on a clean tree and 1 on a tree containing the regression.

The lab simulates this scenario with a synthetic regression — a line in main.tf that breaks the validator — and a synthetic test script that checks for the line. The point is the workflow, not the Terraform syntax.

Tasks

Task 1 — Build the empty repository

LAB="$HOME/bisect-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

# Set deterministic dates so OIDs are reproducible across runs if
# the commit content and timestamps are identical. The rebase-style
# @<unix> +<tz> format is not used here; we want every commit to
# have a distinct timestamp so that the OIDs are deterministic and
# the bisect log is easy to read.
GIT_AUTHOR_DATE='2026-08-01T00:00:00Z' \
GIT_COMMITTER_DATE='2026-08-01T00:00:00Z' \
git commit --allow-empty -m 'C0: initial valid'

The first commit is empty, dated 2026-08-01T00:00:00Z, and is the known-good starting point of the bisect.

Task 2 — Add the validation script and the file under test

# check-shell-blocks: allow-invalid
cd "$HOME/bisect-lab"

cat > test.sh <<'EOF'
#!/bin/sh
# test.sh — exits 0 (good) if main.tf is valid, 1 (bad) if it is not.
# Used by `git bisect run` to identify the commit that broke the file.
set -eu

if [ ! -f main.tf ]; then
  echo "FAIL: main.tf does not exist at this commit"
  exit 1
fi

# Synthetic validation rule: the file must contain exactly one line
# that begins with the marker "# valid". The regression in Task 4
# removes this marker.
if grep -q '^# valid$' main.tf; then
  exit 0
fi

echo "FAIL: missing '# valid' marker"
exit 1
EOF
chmod +x test.sh

# C1: introduce the file, with the marker. This commit is valid.
cat > main.tf <<'EOF'
# valid
module "network" {
  source = "./network"
  cidr   = "10.0.0.0/16"
}
EOF

GIT_AUTHOR_DATE='2026-08-01T00:01:00Z' \
GIT_COMMITTER_DATE='2026-08-01T00:01:00Z' \
git add main.tf test.sh
git commit -m 'C1: introduce main.tf with valid marker'

# Sanity-check that the script works on the current commit.
./test.sh && echo "test.sh passes on C1" || echo "test.sh FAILED on C1"

The test script is intentionally trivial: a single grep for a literal string. It exits 0 if the marker is present, 1 if not, and prints a diagnostic to standard error. POSIX set -eu makes any unexpected failure abort the script and produce a non-zero exit status, which is what git bisect run interprets as “this commit is bad”.

Task 3 — Build the first half of the history (C2 through C10, all valid)

cd "$HOME/bisect-lab"

# Helper: append a benign change that keeps the marker present.
append_valid() {
  local n="$1"
  local msg="$2"
  echo "# change-${n}: ${msg}" >> main.tf
  GIT_AUTHOR_DATE="2026-08-01T00:\${n}:00Z" \
  GIT_COMMITTER_DATE="2026-08-01T00:\${n}:00Z" \
  git add main.tf
  git commit -m "C${n}: ${msg}"
}

for n in 02 03 04 05 06 07 08 09 10; do
  append_valid "$n" "add a benign attribute"
done

# Run test.sh once at the head of the valid history to confirm it
# still passes.
./test.sh && echo "all valid at C10" || {
  echo "test.sh failed at C10 — fix the script before continuing"
  exit 1
}

git log --oneline

The script must continue to pass at the tip of the valid history. If it does not, one of the append_valid calls accidentally dropped the marker — re-run Task 2 to reset, then re-run Task 3.

Task 4 — Plant the regression at C11

cd "$HOME/bisect-lab"

# C11: introduce the regression — drop the marker.
sed -i '/^# valid$/d' main.tf

# Sanity-check: test.sh now fails at C11.
./test.sh && echo "PASS (unexpected)" || echo "FAIL (expected)"

GIT_AUTHOR_DATE='2026-08-01T00:11:00Z' \
GIT_COMMITTER_DATE='2026-08-01T00:11:00Z' \
git add main.tf
git commit -m 'C11: refactor main.tf (introduces regression)'

The regression is the removal of the # valid marker line. Every later commit is “broken” in the sense that the marker is absent, but the regression was introduced at C11 specifically. Bisect’s job is to identify C11 as the first bad commit.

Task 5 — Build the second half (C12 through C19, all broken)

cd "$HOME/bisect-lab"

append_broken() {
  local n="$1"
  local msg="$2"
  echo "# change-${n}: ${msg}" >> main.tf
  GIT_AUTHOR_DATE="2026-08-01T00:\${n}:00Z" \
  GIT_COMMITTER_DATE="2026-08-01T00:\${n}:00Z" \
  git add main.tf
  git commit -m "C${n}: ${msg}"
}

for n in 12 13 14 15 16 17 18 19; do
  append_broken "$n" "another attribute change on broken base"
done

git log --oneline
# expected: C0, C1, C2, ..., C19, with C11 being the first commit
# whose main.tf lacks the marker.

# Confirm the history is now 20 commits total.
git rev-list --count HEAD
# expected: 20

The history has twenty commits, with C11 as the dividing line. Every commit from C11 onwards is “bad” by the test script’s definition, but the regression was introduced at C11 and propagated through every subsequent commit.

Task 6 — Confirm the bisect endpoints

cd "$HOME/bisect-lab"

# Known-good: the very first commit, which is empty and cannot fail
# the test (no main.tf to fail on, the script returns 1 — but the
# first commit's *behaviour* is "no network module", which was the
# state before the regression, so it is good).
GOOD="$(git rev-list --max-parents=0 HEAD)"
echo "known good: $GOOD"

# Known-bad: the tip, which lacks the marker.
BAD="$(git rev-parse HEAD)"
echo "known bad:  $BAD"

# Sanity-check both:
git checkout "$GOOD" -- main.tf 2>/dev/null || true
git checkout "$GOOD" 2>&1 | head -3 || true
# Don't actually run the test here — the script returns 1 on
# missing main.tf, which is "bad" by bisect's contract, but the
# commit is "good" by the team policy (pre-regression). Re-test:
git checkout main 2>&1 | tail -3

# Better: redefine "good" to be C1, which has the marker, and
# "bad" to be C19, which does not. C0 is special (no main.tf) and
# is not the right endpoint.
GOOD="$(git rev-parse HEAD~18)"
BAD="$(git rev-parse HEAD)"

./test.sh 2>/dev/null && echo "tip is good (unexpected)" || echo "tip is bad (expected)"

git checkout "$GOOD" -- main.tf 2>/dev/null
git checkout "$GOOD" 2>&1 | tail -3
./test.sh 2>/dev/null && echo "C1 is good (expected)" || echo "C1 is bad (unexpected)"

The known-good endpoint is the first commit with main.tf present, which is C1. The known-bad endpoint is the tip. Bisect expects both endpoints to be valid for the test script’s contract: the good endpoint returns 0, the bad endpoint returns 1.

Task 7 — Run `git bisect run`

cd "$HOME/bisect-lab"

# Reset to main, then start the bisect.
git switch main

git bisect start
git bisect bad "$BAD"
git bisect good "$GOOD"

# Capture stats before the run.
BISECT_START_TIME="$(date +%s)"

# Run the bisect.
git bisect run ./test.sh

BISECT_END_TIME="$(date +%s)"
BISECT_DURATION=$((BISECT_END_TIME - BISECT_START_TIME))

echo "bisect took ${BISECT_DURATION}s"

# Save the bisect log.
git bisect log > bisect-log.txt

# Save the first-bad commit identified.
FIRST_BAD="$(git bisect terms --term-bad --commit)"
{
  echo "first bad commit: $FIRST_BAD"
  echo
  echo "--- git log -1"
  git log -1 --format='%H%n%an %ad%n%s' "$FIRST_BAD"
  echo
  echo "--- main.tf at first bad commit"
  git show "$FIRST_BAD":main.tf | head -20
} > first-bad.txt

# Save the stats.
{
  echo "commits in range: $(git rev-list --count "$GOOD".."$BAD")"
  echo "test.sh invocations: $(grep -c '^#' bisect-log.txt || true)"
  echo "wall time: ${BISECT_DURATION}s"
} > bisect-stats.txt

# Reset the bisect state — bisect leaves HEAD on the first-bad
# commit, which is not where the team usually wants to be.
git bisect reset

git switch main

The bisect log contains one line per step, each prefixed with the candidate OID. The first bad commit output is C11. The total number of invocations should be roughly ceil(log2(20)) = 5.

Task 8 — Read the bisect log

cd "$HOME/bisect-lab"

cat bisect-log.txt
# expected: roughly 5 lines, each starting with a SHA and ending
# with "test.sh: <message>".

Each line in the bisect log names the candidate commit and the result. Reading them in order tells the bisect’s path through the history: the first candidate is the midpoint, the next is a quarter-point, the next is an eighth-point, and so on until the binary search converges.

Task 9 — Document bisect’s limits and capture the deliverables

The regression in this lab is visible in the commit content: the marker is gone, and a grep finds it. Bisect is excellent at “visible” regressions. It is poor at three classes of problem that the lab does not cover but the team should know:

Regressions that were introduced and reverted within the searched range. If C11 broke the marker, C12 reverted it, and C13 re-broke it, bisect identifies C11 as the first bad commit even though C13 is the currently broken commit.
Regressions that depend on state outside the working tree. If the test script depends on a database, a cloud account, or a network resource that the bisect environment does not have, the test will fail for reasons unrelated to the regression.
Non-deterministic regressions. If the test script flakes, bisect produces a wrong first-bad commit because it trusts the script’s exit code.

Capture the deliverables and confirm the script still works on the tip:

cd "$HOME/bisect-lab"

ls -l test.sh bisect-log.txt first-bad.txt bisect-stats.txt

# Confirm the script is executable and re-runs cleanly on the tip.
./test.sh 2>/dev/null && echo "tip is good (unexpected)" || echo "tip is bad (expected)"

git log --oneline

The deliverables are the four files plus a working bisect that, if run again with the same input, produces the same first-bad commit.

Validation

git rev-list --count HEAD returns 20: C0 through C19.
./test.sh returns 0 at C1 and 1 at the tip.
git bisect run ./test.sh exits 0 with the message <sha> is the first bad commit.
The commit identified as first bad is C11, which has the message “C11: refactor main.tf (introduces regression)”.
bisect-log.txt has 5 ± 1 lines, each with a SHA prefix.
first-bad.txt matches the C11 commit’s OID, message, and main.tf content.
The deliverables test.sh, bisect-log.txt, first-bad.txt, and bisect-stats.txt exist and are non-empty.

Expected Outcome

A repository with twenty commits, a regression at C11, and a one-command bisect that identifies C11 in five steps.

$HOME/bisect-lab/
├── .git/
│   ├── BISECT_LOG                # the binary-search trail
│   ├── logs/
│   ├── objects/
│   └── refs/heads/main
├── bisect-log.txt                # captured `git bisect log`
├── bisect-stats.txt              # commits in range, invocations, time
├── first-bad.txt                 # the C11 commit + its main.tf content
├── main.tf                       # current contents, broken
└── test.sh                       # the test that bisect invokes

You can answer two questions with confidence: “what is the first commit that broke this?” (git bisect run finds it in O(log n)) and “what can bisect not find?” (regressions outside the working tree, regressions that were reverted, non-deterministic regressions).

Troubleshooting

git bisect run ./test.sh exits with code 125 immediately. The test script is not executable or has a syntax error. Run ./test.sh by hand and confirm it returns 0 (good) or 1 (bad) — not 127 or 126, which are the shell’s “not found” and “not executable” codes. Bisect treats 125 as “could not run” and aborts.

The bisect identifies a commit other than C11. Either the regression is in a different commit than Task 4 planted, or the test script’s contract does not match the commit history. Verify the marker is present at C1 and absent from C12 onwards:

for sha in $(git rev-list --reverse HEAD); do
  echo -n "$sha: "
  if git show "$sha:main.tf" 2>/dev/null | grep -q '^# valid$'; then
    echo "good"
  else
    echo "bad"
  fi
done | head -25

The first bad: line is the actual first bad commit. If it is not C11, fix the history and re-run the bisect.

bisect-log.txt has more than 6 lines. That is fine — bisect may take 5 to 6 steps depending on how it lands on the midpoint, and a couple of “skip” steps (test script returns 121–124) will inflate the count. What matters is that the first-bad commit identified is C11.

git bisect reset says “won’t bisect on this branch”. The bisect state is per-clone and resets automatically if you switch branches mid-bisect. Re-run git bisect start from the current branch and continue from where you were.

Cleanup

LAB="$HOME/bisect-lab"

# Keep the deliverables.
mv "$LAB"/test.sh "$LAB"/bisect-log.txt \
   "$LAB"/first-bad.txt "$LAB"/bisect-stats.txt \
   "$HOME"/ 2>/dev/null

rm -rf "$LAB"

find "$HOME" -maxdepth 1 -name 'bisect-lab' -print
# expected: (no output)

If you ran the lab in an existing repository by accident, the bisect state is reset by git bisect reset and the history remains. Revert the regression by reverting the C11 commit:

cd /path/to/that-repo
git revert $C11_SHA

What You Learned

git bisect run is binary search over commit history. With twenty commits, it converges in five steps; with a thousand commits, it converges in ten. The number of test invocations is logarithmic in the range size.
The test is anything that exits 0 or 1. It can be a grep, a terraform validate, a custom assertion, or a wrapper around a test framework. The discipline is that the script must be deterministic and fast, because bisect runs it once per candidate commit.
Bisect finds the first bad commit. It does not find currently-bad commits that were introduced and reverted within the range. It also does not find the root cause — only the commit whose introduction coincided with the regression.
The bisect log is the audit trail. It records every step and every test invocation, which is what makes a bisect reproducible and reviewable after the fact.
Bisect cannot fix non-determinism. A flaky test produces a wrong first-bad commit because bisect trusts the exit code blindly. The test script’s discipline is more important than the bisect’s algorithm.
git bisect reset ends the bisect. Bisect leaves HEAD on the first-bad commit; always reset before resuming normal work.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Build the empty repository

Task 2 — Add the validation script and the file under test

Task 3 — Build the first half of the history (C2 through C10, all valid)

Task 4 — Plant the regression at C11

Task 5 — Build the second half (C12 through C19, all broken)

Task 6 — Confirm the bisect endpoints

Task 7 — Run `git bisect run`

Task 8 — Read the bisect log

Task 9 — Document bisect’s limits and capture the deliverables

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Build the empty repository

Task 2 — Add the validation script and the file under test

Task 3 — Build the first half of the history (C2 through C10, all valid)

Task 4 — Plant the regression at C11

Task 5 — Build the second half (C12 through C19, all broken)

Task 6 — Confirm the bisect endpoints

Task 7 — Run git bisect run

Task 8 — Read the bisect log

Task 9 — Document bisect’s limits and capture the deliverables

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status

Task 7 — Run `git bisect run`