Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~30 min

Runbook: Troubleshoot a Failed CI Pipeline

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the failing run: gh run list --workflow <workflow> --branch <branch> --limit 5 --json databaseId,headSha,conclusion,status,event (or the equivalent for the platform)
  • · Confirm the failure is reproducible on the same SHA: re-run the job from the UI (gh run rerun <run-id> --failed) and observe the same step fail
  • · Identify the workflow file and the affected jobs: gh workflow view <workflow> and gh run view <run-id> --json jobs | jq ".jobs[] | {name,conclusion,steps[].name}"
  • · Capture the failing step's exit code, duration, and the exact log lines around the failure: gh run view <run-id> --log-failed (filtered to the failing step)
  • · Capture the environment matrix: runner OS, runner version, language/runtime version, region. These are in the run metadata and matter for flakiness diagnosis

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Confirm the failure is not a transient infrastructure error: check the platform status page (https://www.githubstatus.com, https://status.gitlab.com, https://status.circleci.com). If the platform reports an incident, the CI failure is not your bug — annotate the run, communicate to the team, and wait for platform recovery
  2. 2STEP 2 - Read the failure category from the job summary: gh run view <run-id> --json jobs,conclusion | jq ".jobs[] | select(.conclusion==\"failure\") | {name,conclusion,steps_failed:[.steps[]|select(.conclusion==\"failure\")|.name]}". Categories are: setup failure (runner init, checkout, dependency install), test failure (unit/integration/e2e), lint/format failure, build failure, deploy failure, secret detection failure, dependency resolution failure
  3. 3STEP 3 - Identify the failing step by name and exit code: gh run view <run-id> --log-failed 2>&1 | grep -B5 -A30 "exit code" | head -80. The exit code tells the story: 1 = assertion/test failed, 2 = misuse of shell, 127 = command not found, 137 = OOM killed, 143 = SIGTERM (timeout)
  4. 4STEP 4 - Inspect the failing step's logs in full: gh run view &lt;run-id&gt; --log > /tmp/run.log && grep -n "<failing step>" /tmp/run.log and then read the surrounding 200 lines. Treat the earliest relevant error as a lead, not proof: setup noise and expected negative-test output can precede the actual cause
  5. 5STEP 5 - Compare against the last green commit: git log --oneline &lt;last-green-sha&gt;..&lt;head-sha&gt; and look at the diff: git diff &lt;last-green-sha&gt;..&lt;head-sha&gt; --stat. If the failing test or step changed in the diff, the cause is the PR. If not, suspect flakiness or external dependency drift
  6. 6STEP 6 - Reproduce locally with the same toolchain as the runner: docker run --rm -v "$PWD:/work" -w /work &lt;runner-image&gt;:&lt;version&gt; bash -c "<failing command>". If the runner is GitHub-hosted ubuntu-latest, use mcr.microsoft.com/devcontainers/base:ubuntu-24.04 or the exact runs-on image. For self-hosted, replicate the runner's userland
  7. 7STEP 7 - Capture the environment from the job log (gh run view &lt;run-id&gt; --log | grep -E "Runner Image|Image:|Version:") and from workflow runs-on; gh run view --json does not expose an arbitrary runner object. Compare with the last green log. Drift in runner image, language runtime, or packages is a frequent cause of "passed yesterday, fails today"
  8. 8STEP 8 - Check for flakiness: re-run the same job 3-5 times (gh run rerun &lt;run-id&gt; --failed, repeated) and observe the outcome. If the failure is intermittent across runs, suspect a test-ordering issue, a network-dependent test, a timing assertion, or shared-state pollution between tests. If the failure is consistent across runs, it is a real bug, not a flake
  9. 9STEP 9 - Decide the action based on the diagnosis: real bug → open a PR to fix it; flakiness → quarantine the test (mark as @flaky or skip with a tracking issue, do not delete); transient infrastructure → re-run; environment drift → pin the runner image version and re-run; intentional change in the PR → re-run after the change author verifies the new expected output
  10. 10STEP 10 - Document in the run: gh run view &lt;run-id&gt; --json jobs | jq ".jobs[] | {name,conclusion}" | tee /tmp/run-diagnosis.txt and link the diagnosis in the PR conversation. Annotate the run with the category (bug, flake, infrastructure) so future trend analysis is accurate

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The failing run has a documented diagnosis in the PR conversation or the run annotation
  • ✓If real bug: the fix is in a commit on the same branch, the PR passes CI on the next push, and the fix is reviewed
  • ✓If flakiness: the test is marked flaky with a tracking issue linked, and three consecutive re-runs pass
  • ✓If infrastructure: the platform status incident is referenced in the run annotation, and a re-run after platform recovery passes
  • ✓If environment drift: the runner image version is pinned in the workflow file, and the next run uses the pinned version
  • ✓The CI trend for the workflow (gh run list --workflow &lt;workflow&gt; --limit 20 --json conclusion | jq "sort_by(.conclusion) | group_by(.conclusion) | map({(.[0].conclusion): length}) | add") shows a return to baseline success rate

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the failing run was on main and the failure has already been merged: revert the offending PR through a reviewed commit (see git-cicd-gitops-rb-05-revert-production-change). Do not rewrite the shared protected branch to move it back to the last green SHA
  • ↶If the failing run is on a feature branch: no rollback needed — the branch is isolated from main. Just fix the failure on the branch and re-push
  • ↶If the diagnosis was wrong (re-runs still fail): re-enter the diagnostic at step 4 with the assumption that the first error log line is not the cause. Capture the full log and the environment matrix in the run annotation for the next responder
  • ↶If the workflow file was the cause and the fix needs to ship on main: open a hotfix PR with the workflow fix, get a fast-track review, and merge before the next attempt
  • ↶If the runner pool is exhausted (no runners available for runs-on: &lt;label&gt;): see git-cicd-gitops-rb-11-restore-runner-capacity

6 · Escalation

When the runbook isn't enough, contact:

  • · Diagnosis reveals the failure is in a third-party action or image the team does not control: open an issue upstream, pin to a known-good version, and document the workaround
  • · Diagnosis reveals a secret leaked through the runner logs (gh run view --log | grep -E "(AKIA|ghp_|xox[abprs])"): stop the run, rotate the secret per git-cicd-gitops-rb-07-respond-to-secret-committed, audit for unauthorised use
  • · Diagnosis reveals the runner itself was compromised (unexpected processes, outbound connections to unknown IPs): take the runner offline per git-cicd-gitops-rb-13-respond-to-compromised-runner
  • · Diagnosis reveals the failure is caused by a dependency compromise (lockfile points at a known-bad version, package was tampered with): see git-cicd-gitops-rb-14-respond-to-compromised-dependency
  • · Multiple unrelated workflows fail at the same time with the same step category: platform-wide issue, escalate to the platform team before further investigation

A failing CI pipeline is a signal, not a verdict. The failure is a symptom; the cause may be the PR, the test, the runner, the dependencies, the platform, or the secret. The diagnostic is ten steps because the runbook must distinguish between “your code is wrong” and “your code ran against the wrong world”.

The steps are ordered by cost: cheap checks first (status page, log scan), expensive checks last (full reproduction, flakiness matrix). Skipping a step is how a flake gets marked as a bug, or a bug gets marked as a flake. Both are wrong, but in different directions.

1. Confirm the failure is reproducible on the same SHA

Read-only / Safe
$ RUN_ID="1234567890"
WORKFLOW="ci.yml"
gh run rerun "$RUN_ID" --failed
gh run watch "$RUN_ID" --exit-status || true
gh run view "$RUN_ID" --json jobs,conclusion --jq '{conclusion, jobs: [.jobs[] | {name,conclusion}]}'

If the re-run passes, the original failure was a flake. Mark the test as flaky and link the issue; do not investigate further. If the re-run fails with the same step, the failure is real; continue.

2. Read the failure category from the job summary

Read-only / Safe
$ RUN_ID="1234567890"
gh run view "$RUN_ID" --json jobs,conclusion | jq '.jobs[] | select(.conclusion=="failure") | {name,conclusion,steps_failed:[.steps[]|select(.conclusion=="failure")|.name]}'
echo '--- the category from the failing step name ---'
gh run view "$RUN_ID" --json jobs | jq -r '.jobs[].steps[] | select(.conclusion=="failure") | .name' | head -10

The step name tells the category. Setup node, Checkout, Install dependencies → setup failure. Run unit tests, pytest, go test → test failure. terraform plan, kubectl apply --dry-run → IaC failure. gitleaks, trufflehog → secret detection failure.

3. Identify the failing step’s exit code

Read-only / Safe
$ RUN_ID="1234567890"
gh run view "$RUN_ID" --log-failed 2>&1 | grep -B5 -A30 'Process completed with exit code' | head -80
echo '--- raw log around the first error ---'
gh run view "$RUN_ID" --log 2>&1 | grep -nE '(error|Error|ERROR|fatal|FATAL)' | head -20

Exit code → diagnosis:

  • 1 — assertion/test failure, the script itself ran fine
  • 2 — shell misuse (set -e triggered, [[ ]] failed)
  • 127 — command not found (PATH wrong, binary missing in runner image)
  • 137 — OOM-killed (runner ran out of memory)
  • 143 — SIGTERM, the runner killed the step (timeout)

4. Inspect the failing step’s logs

Read-only / Safe
$ RUN_ID="1234567890"
gh run view "$RUN_ID" --log > /tmp/run.log
echo '--- earliest error candidates ---'
grep -nE '(error|Error|ERROR|fatal|FATAL)' /tmp/run.log | head -1
echo '--- context around the earliest candidate ---'
FIRST_ERR_LINE=$(grep -nE '(error|Error|ERROR|fatal|FATAL)' /tmp/run.log | head -1 | cut -d: -f1)
START_LINE=$(( FIRST_ERR_LINE > 50 ? FIRST_ERR_LINE - 50 : 1 ))
sed -n "${START_LINE},$((FIRST_ERR_LINE + 150))p" /tmp/run.log

The earliest relevant error is a useful candidate, not a guaranteed root cause. Expected negative-test output and setup warnings may appear earlier. Confirm causality by reading the step boundary and enough surrounding context.

5. Compare against the last green commit

Read-only / Safe
$ BRANCH="REPLACE_WITH_BRANCH"
HEAD_SHA=$(gh run view "$RUN_ID" --json headSha --jq '.headSha')
LAST_GREEN_SHA=$(gh run list --workflow "$WORKFLOW" --branch "$BRANCH" --status success --limit 1 --json headSha --jq '.[0].headSha')
echo "head: $HEAD_SHA"
echo "last green: $LAST_GREEN_SHA"
git log --oneline "$LAST_GREEN_SHA..$HEAD_SHA"
echo '--- diff ---'
git diff "$LAST_GREEN_SHA..$HEAD_SHA" --stat | tail -40

If the failing step or test was changed in the diff, the cause is in the PR. If not, suspect flakiness or environment drift.

6. Reproduce locally with the same toolchain

Read-only / Safe
$ RUNNER_IMAGE="mcr.microsoft.com/devcontainers/base:ubuntu-24.04"
echo "reproducing with: $RUNNER_IMAGE"
docker run --rm -v "$PWD:/work" -w /work "$RUNNER_IMAGE" bash -c '
set -e
<install dependencies the way the workflow does>
<run the failing step>
' 2>&1 | tee /tmp/local-repro.log

The devcontainer image is an Ubuntu userland approximation, not a replica of GitHub’s hosted runner image or preinstalled tool cache. A matching failure is useful evidence. A pass does not clear the code until the workflow’s exact tool versions, services, permissions, and environment have been reproduced.

7. Capture the environment matrix

Read-only / Safe
$ RUN_ID="1234567890"
echo '--- failing run image metadata ---'
gh run view "$RUN_ID" --log | grep -E 'Runner Image|Image:|Version:' | head -30
echo '--- last green run image metadata ---'
LAST_GREEN_ID=$(gh run list --workflow "$WORKFLOW" --status success --limit 1 --json databaseId --jq '.[].databaseId')
gh run view "$LAST_GREEN_ID" --log | grep -E 'Runner Image|Image:|Version:' | head -30

Drift in the resolved hosted image, tool-cache versions, or installed package versions is a common cause of “passed yesterday, fails today” without a code change. The run log is the evidence source for those details.

8. Check for flakiness

Read-only / Safe
$ RUN_ID="1234567890"
for i in 1 2 3 4 5; do
echo "=== attempt $i ==="
gh run rerun "$RUN_ID" --failed
gh run watch "$RUN_ID" --exit-status || true
CONCLUSION=$(gh run view "$RUN_ID" --json conclusion --jq '.conclusion')
echo "attempt $i: $CONCLUSION"
done

If 4 of 5 attempts pass, the test is flaky. Mark it with @flaky (or the platform’s equivalent) and link a tracking issue; do not “fix” it by deleting it. If 0 of 5 attempts pass, the test is broken, not flaky.

9. Decide the action

Read-only / Safe
$ DIAGNOSIS="real-bug"  # or "flake", "infrastructure", "env-drift", "intentional"
BASE_BRANCH="main"
case "$DIAGNOSIS" in
real-bug)
  git switch -c "fix/ci-${RUN_ID}"
  echo 'apply and commit the diagnosed fix before opening the PR'
  gh pr create --base "$BASE_BRANCH" --title "fix: REPLACE_WITH_FAILURE" --body "Failing run: REPLACE_WITH_URL"
  ;;
flake)
  echo "mark test as flaky, link tracking issue"
  # Platform-specific: e.g. add @flaky marker to pytest
  ;;
infrastructure)
  echo "annotate run, re-run after platform recovery"
  ;;
env-drift)
  sed -i 's/runs-on: ubuntu-latest/runs-on: ubuntu-24.04/' .github/workflows/$WORKFLOW
  git commit -am "ci: pin runner image version"
  ;;
intentional)
  echo "verify the new expected output, update test fixtures"
  ;;
esac

10. Document in the run

Read-only / Safe
$ RUN_ID="1234567890"
gh run view "$RUN_ID" --json jobs | jq '.jobs[] | {name,conclusion}' > /tmp/run-diagnosis.txt
gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO --title "CI failure diagnosis: $(date -u +%Y-%m-%d)" --body "Run: REPLACE_WITH_URL. Diagnosis: REPLACE_WITH_CATEGORY. Action: REPLACE_WITH_ACTION. See run-diagnosis.txt" --label ci --label diagnosis

The run annotation is the audit trail. A future responder reading the run list sees “flake”, “infrastructure”, or “bug” without re-doing the diagnosis.

Verification

The failing run has a documented diagnosis. If real bug: the fix is in a commit on the same branch, the PR passes CI on the next push, and the fix is reviewed. If flakiness: the test is marked flaky with a tracking issue linked, and three consecutive re-runs pass. If infrastructure: the platform status incident is referenced, and a re-run after platform recovery passes. If environment drift: the runner image version is pinned in the workflow file. The CI trend for the workflow returns to baseline success rate (gh run list --workflow &lt;workflow&gt; --limit 20 --json conclusion).

Rollback

If the failing run was on main and the failure has already been merged, revert the offending PR. If the diagnosis was wrong (re-runs still fail), re-enter the diagnostic at step 4 with the assumption that the first error log line is not the cause. If the workflow file was the cause and the fix needs to ship on main, open a hotfix PR with the workflow fix and merge before the next attempt. If the runner pool is exhausted, see git-cicd-gitops-rb-11-restore-runner-capacity.

References

  1. GitHub Actions — Workflow syntax
  2. GitHub CLI — gh run
  3. GitLab CI — Job artifacts and logs
  4. CircleCI — Insights and troubleshooting
  5. Google SRE Book — Chapter 11: Being On-Call