As neededci-incident-triage

CI/CD Incident Triage Checklist

16 items ·14 critical ·2 warn ·0 info

How to use this checklist

This is the order to work through during a CI/CD incident. The ordering is deliberate: impact before cause, evidence before change, mitigation before fix. CI/CD incidents are particularly susceptible to “let me just retry” thinking, because pipelines look idempotent — but a retry destroys the evidence the team will need 20 minutes later.

A team that runs incidents against this checklist once a quarter on a non-prod scenario will use it from memory when the real one comes.

Sign-off

Every critical item must pass. A failing critical item blocks the incident from being closed; it is not a note for later. Record the date, the incident commander, and the disposition of every item that did not pass.

Critical14 items

The impact is stated in one sentence: which pipelines are failing, which environments are blocked, which deploys are paused, for whom, since when. Everything else is secondary to this.
Scope is established across the CI/CD stack: source (git hosting), pipeline (runner, control plane), build (image, dependency, cache), artifact (registry, signature), deployment (controller, sync), or environment (cluster, runtime). The scope eliminates most hypotheses.
Recent changes are listed: pipeline definition, runner image, third-party action version, controller manifest, secret, registry policy, certificate. Most CI/CD incidents are a change, and the change is usually recent.
```
git log --since="24 hours ago" -- .github/workflows/ .gitlab-ci.yml Jenkinsfile argocd/ flux/; kubectl rollout history deployment -n argocd; kubectl get events -n argocd --sort-by=.metadata.creationTimestamp
```
Evidence is captured before remediation: pipeline run logs (full, not truncated), controller logs (`kubectl logs -n argocd`), runner system logs, sync state, registry response, and the exact error string. A retry or queue drain destroys evidence.
```
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=500 --since=1h > /tmp/controller.log; gh run view &lt;run-id&gt; --log > /tmp/pipeline.log
```
No retry, no rebuild, no "let me just run it again" happens before the failure is captured. A second failure is not the same evidence as the first.
A determination is made: is the failure reproducible on a fresh pipeline run, or is it transient? Reproducible failures point at code, configuration, or environment; transient failures point at runners, network, or external dependencies.

For a "image won't deploy" or "image won't run" symptom, the artifact identity is verified: tag matches the expected one, digest matches the registry record, signature is valid, attestation is present.

crane digest ghcr.io/&lt;org&gt;/&lt;image&gt;:&lt;tag&gt;; cosign verify --certificate-identity-regexp "https://github.com/&lt;org&gt;/&lt;repo&gt;" --certificate-oidc-issuer "https://token.actions.githubusercontent.com" ghcr.io/&lt;org&gt;/&lt;image&gt;@sha256:&lt;digest&gt;

For a GitOps symptom, the controller state is checked: `argocd app get <app> --refresh` and `argocd app manifests <app>` show what the controller believes is the desired state and what it sees in the cluster. A diff between the two is the answer.
External dependencies (git hosting, registry, OIDC issuer, secret manager, KMS, container registry mirror, time server) are healthy and have not changed. A flaky third party often looks like a CI/CD bug.
A hypothesis is stated in one sentence that the evidence supports and that predicts something testable. The next action is chosen to confirm or refute the hypothesis, not to act on the first plausible story.
Mitigation precedes root-cause fix. A broken production deploy is mitigated (rollback, pause pipeline, freeze controller) before the team starts searching for why. Service and search are separate activities with different urgencies.
One thing is changed at a time. Parallel changes make the eventual post-mortem unresolvable. The change log is kept inline with the chat or ticket, not in someone's head.
Recovery is verified through the user-visible symptom (deploy succeeded, cluster reconciled, alert resolved) and not through the component that was changed. The pipeline being green does not mean the deploy ran.
A post-mortem is filed while detail is fresh (within 5 business days), with an owner, a date, contributing factors, root cause, and prevention actions. The post-mortem distinguishes what was known at the time from what was known afterwards.

Warning2 items

A decision is recorded: rollback to a known-good state, or fix forward with a tested patch. The decision is owned, dated, and tied to the incident ticket.
Stakeholders are updated on a fixed cadence (every 5 to 15 minutes during the active incident), including when nothing has changed. The cadence continues until the incident is closed.