Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~60 min

Runbook: Investigate a Production Deployment Failure (CI Green but Prod Broken)

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-09-troubleshoot-failed-ci-pipeline
  • git-cicd-gitops-rb-15-validate-production-artifact
  • kubectl access to the affected cluster
  • Access to the CI/CD pipeline logs and the artifact registry
  • Knowledge of the CI test environment vs the production environment (cluster type, dependencies, config)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm the CI pipeline actually ran and reported success. gh run view <run-id> --json conclusion,status for GitHub Actions; the corresponding command for the CI system in use. A "green" CI that did not run is a different problem
  • · Confirm the artifact deployed to production matches the CI artifact. Pull the digest from the registry (crane digest <registry>/<image>:<tag>) and the CI build output. Mismatch = the deploy did not deploy what CI built
  • · Capture the production symptom: kubectl get pods -n <ns>, kubectl describe pod <pod> -n <ns>, kubectl logs <pod> -n <ns>. The symptom (CrashLoopBackOff, ImagePullBackOff, error rate spike) identifies the failure class
  • · Capture the CI test environment: cluster type, Kubernetes version, dependencies installed, configuration. The CI test environment is rarely identical to production — the differences are the suspect. Document the differences explicitly
  • · Identify the time the deploy started. kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30; argocd app history <app> or flux get kustomization. The deploy start time anchors the investigation
  • · Identify the change window: was the CI artifact built before or after the last known-good production deploy? git log and the artifact's creation timestamp. The change window is what was different from the last working state

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Classify the failure. Five classes: (a) pod will not start (ImagePull, CrashLoopBackOff, pending); (b) pod starts but app errors (5xx, connection refused, health check fails); (c) pod starts, app is healthy, but downstream is broken (DB connection, cache miss, external API timeout); (d) pod runs but performance degrades (slow responses, memory growth, thread starvation); (e) pod runs and app is healthy but observability shows elevated errors (the metric is the failure, not the app)
  2. 2STEP 2 - For class (a) pod will not start: kubectl describe pod <pod> -n <ns>. The Events section names the failure: FailedScheduling (resource shortage), FailedMount (PVC/ConfigMap missing), ImagePullBackOff (image/credential/registry issue), CrashLoopBackOff (container crash on start). Each has a different fix
  3. 3STEP 3 - For class (a) ImagePullBackOff: the production image pull failed. Common causes: production uses a private registry and the production cluster's pull secret does not have access; the image was deleted from the registry after the CI build (TTL or retention); the CI built for one registry but production pulls from another. Fix the pull secret or repull the image
  4. 4STEP 4 - For class (a) CrashLoopBackOff: the container starts and immediately exits. kubectl logs <pod> -n <ns> --previous shows the exit reason. Common causes: missing environment variable (CI has it, production does not), wrong database URL, missing volume mount, wrong command/args. Compare the CI test pod spec to the production pod spec (kubectl get pod <pod> -n <ns> -o yaml)
  5. 5STEP 5 - For class (b) pod starts but app errors: the application is running but not functioning. kubectl logs <pod> -n <ns> shows the application errors. Common causes: a database migration was not run in production (CI runs migrations, production does not); a config map is missing in production; the application cannot reach a downstream it could reach in CI
  6. 6STEP 6 - For class (c) downstream connection failure: the application is running and responding, but cannot reach a dependency. kubectl exec -it <pod> -n <ns> -- curl -fsS <downstream-url> or kubectl exec -it <pod> -n <ns> -- nslookup <downstream>. The dependency may be available in CI but not in production (different VPC, different NetworkPolicy, different service mesh)
  7. 7STEP 7 - For class (d) performance degradation: the application is running but slower than expected. kubectl top pod <pod> -n <ns> for CPU/memory; kubectl exec -it <pod> -n <ns> -- ss -tnp for open connections. Common causes: production has more traffic than CI (the CI test never exercised the production scale); a memory leak in the new code; a thread pool configured for CI load, not production load
  8. 8STEP 8 - For class (e) observability-only failures: the app is healthy but the metric says it is broken. kubectl logs <pod> -n <ns> is empty of errors; kubectl exec -it <pod> -n <ns> -- curl -fsS http://localhost:8080/metrics | grep error shows the metrics. Common causes: a new error path was introduced that returns 2xx but logs an error; the metric definition changed in this release; the alert threshold is wrong
  9. 9STEP 9 - Compare CI and production environments side by side. The diff is the suspect. kubectl get pod -o yaml in CI (or the CI build output) vs production. Look for: different env vars, different resource limits, different volume mounts, different image tags (CI tested an older tag), different security contexts, different node selectors
  10. 10STEP 10 - Verify the artifact's signature and provenance. cosign verify and cosign verify-attestation. The CI-built artifact should match what was deployed; a mismatch means the deploy substituted an artifact that CI never validated. See git-cicd-gitops-rb-15-validate-production-artifact
  11. 11STEP 11 - Roll back to the last known-good if the investigation is taking too long. The investigation can continue in parallel with the rollback; the rollback restores production while the investigation finds the root cause. See git-cicd-gitops-rb-16-roll-back-deployment
  12. 12STEP 12 - Fix the gap between CI and production. Common fixes: add the missing env var to the production ConfigMap; add the production-only dependency to the CI test environment (a CI test that does not exercise the production setup is a gap); add a smoke test that runs against production after deploy (see git-cicd-gitops-rb-27-validate-gitops-after-cluster-recovery); add a canary stage between CI and full production

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The production cluster is healthy: kubectl get pods -n <ns> shows all pods Running; no CrashLoopBackOff, ImagePullBackOff, or Error
  • ✓The application's health/readiness endpoint returns ok: curl -fsS https://<service>/healthz | jq .status
  • ✓The error rate in the observability stack returns to baseline within 5 minutes
  • ✓The CI artifact deployed to production matches the CI artifact built: crane digest <registry>/<image>:<tag> matches the CI build output
  • ✓The artifact's signature and provenance are valid: cosign verify and cosign verify-attestation return success
  • ✓A smoke test runs against production after the deploy and passes
  • ✓The CI test environment now exercises the production dependency that caused the failure (config, env var, dependency)
  • ✓The next deploy of the same artifact type succeeds without manual intervention

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the rollback fails: escalate per the incident response process. The deploy is now a multi-regression (broken artifact AND broken rollback)
  • ↶If the fix to the CI/production gap cannot be deployed (the fix is in the application, but the deploy pipeline is also broken): deploy the fix manually as an emergency change and follow up with a proper PR. See git-cicd-gitops-rb-19-reconcile-emergency-manual-change
  • ↶If the rollback restored production but the CI environment is still wrong: the next CI run will reproduce the bug. Block the CI environment (gh workflow disable) until the gap is fixed. A CI that passes but does not catch production bugs is worse than a CI that fails — it gives false confidence
  • ↶If the investigation reveals the CI test environment is too different from production to be reliable: the fix is structural (move CI to a production-like environment, add a staging environment, add production canaries). Engage the platform team and the engineering leadership; this is a multi-week project
  • ↶If the artifact deployed to production was not the artifact CI built: see git-cicd-gitops-rb-15-validate-production-artifact. The signature/provenance check should have caught this. Add the check to the deploy pipeline before the next deploy
  • ↶If the artifact is correct but the cluster configuration drifted between CI and production (a manual change in production that CI did not pick up): see git-cicd-gitops-rb-19-reconcile-emergency-manual-change. The drift must be reconciled before the fix can be deployed
  • ↶If the smoke test against production passes but the application is still broken (the test does not exercise the broken code path): the test is insufficient. The fix is to expand the test to cover the production scenario. Document the gap in the incident postmortem

6 · Escalation

When the runbook isn't enough, contact:

  • · The production failure is a security incident (data leak, authentication bypass, privilege escalation): engage the security team immediately. The CI/production gap may have been the attack vector. Audit for similar gaps in other services
  • · The CI green/production broken pattern is recurring (it has happened multiple times): the CI test environment is fundamentally different from production. Engage the engineering leadership; this is a structural problem that requires investment in staging or production canaries
  • · The investigation reveals that CI is not actually testing the production code path (a mocked dependency, a stubbed downstream, a skipped test): the CI is a lie. Engage the platform team and the application owner; rebuild the CI to test the real code path
  • · The fix requires a database migration that cannot be rolled back: the deploy is irreversible. Engage the data owner and the application owner; the rollback plan is more important than the deploy. See the database runbook for one-way migrations
  • · The customer is affected by the production failure (downtime, data loss, financial impact): engage the incident response team. The investigation continues in parallel with customer communication. The status page update is critical
  • · The CI artifact was correct but the deploy substituted a different artifact (a different image tag, a tampered image): see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The substitution is a security incident; the CI result is irrelevant

A green CI that produces a broken production deploy is the most expensive failure mode in CI/CD — the team has false confidence that the artifact is good. The gap is between what CI tested and what production runs: different config, different dependencies, different scale, different paths exercised. The investigation identifies the gap and the fix closes it.

1. Confirm the CI ran and the artifact matches

Read-only / Safe
$ RUN_ID="1234567890"
REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api:v1.2.3"
echo "--- CI run conclusion ---"
gh run view "$RUN_ID" --json conclusion,status,headBranch,headSha
echo "--- CI artifact digest ---"
gh run view "$RUN_ID" --log | grep -iE 'digest|built|sha256' | head -5
echo "--- production artifact digest ---"
crane digest "$REGISTRY/$IMAGE"
echo "--- match? ---"
CI_DIGEST=$(gh run view "$RUN_ID" --log | grep -oE 'sha256:[a-f0-9]{64}' | head -1)
PROD_DIGEST=$(crane digest "$REGISTRY/$IMAGE")
[ "$CI_DIGEST" = "$PROD_DIGEST" ] && echo "ARTIFACT MATCH" || echo "ARTIFACT MISMATCH"

If the artifacts do not match, the deploy substituted a different artifact. This is a different problem; see git-cicd-gitops-rb-15-validate-production-artifact and git-cicd-gitops-rb-28-respond-to-supply-chain-compromise.

2. Capture the production symptom

Read-only / Safe
$ NS="prod"
SERVICE="checkout-api"
echo "--- pods ---"
kubectl get pods -n "$NS" -l app="$SERVICE"
echo "--- pod describe ---"
kubectl describe pod -n "$NS" -l app="$SERVICE" | grep -A20 "Events:" | head -40
echo "--- logs ---"
kubectl logs -n "$NS" -l app="$SERVICE" --previous --tail=100
echo "--- service endpoints ---"
kubectl get endpoints -n "$NS" "$SERVICE"

The symptom (CrashLoopBackOff, ImagePullBackOff, error logs, no endpoints) identifies the failure class.

3. Compare CI and production pod specifications

Read-only / Safe
$ NS="prod"
SERVICE="checkout-api"
echo "--- production pod spec ---"
kubectl get deploy "$SERVICE" -n "$NS" -o yaml | yq '.spec.template.spec'
echo "--- CI test pod spec (from the CI build output) ---"
gh run view 1234567890 --log | grep -A40 "pod spec" | head -50
echo "--- diff (CI vs production) ---"
kubectl get deploy "$SERVICE" -n "$NS" -o yaml | yq '.spec.template.spec' > /tmp/prod-pod.yaml
echo '<CI pod spec>' > /tmp/ci-pod.yaml
diff -u /tmp/ci-pod.yaml /tmp/prod-pod.yaml | head -40

The diff is the gap. Different env vars, different resource limits, different volume mounts, different node selectors. The diff is what needs to be closed.

4. Test downstream connectivity from production

Read-only / Safe
$ NS="prod"
POD=$(kubectl get pod -n "$NS" -l app=checkout-api -o jsonpath='{.items[0].metadata.name}')
echo "--- DNS ---"
kubectl exec -n "$NS" "$POD" -- nslookup database.internal
echo "--- TCP ---"
kubectl exec -n "$NS" "$POD" -- nc -zv database.internal 5432
echo "--- application-level ---"
kubectl exec -n "$NS" "$POD" -- curl -fsS http://database.internal:8080/healthz
echo "--- compare with CI ---"
gh run view 1234567890 --log | grep -iE 'database|downstream|connection' | head -20

Production may not be able to reach a downstream that CI could reach — different VPC, different NetworkPolicy, different service mesh. The connectivity test reveals the gap.

5. Roll back to the last known-good

Read-only / Safe
$ NS="prod"
SERVICE="checkout-api"
PREVIOUS_REVISION=4
kubectl rollout undo deploy/"$SERVICE" -n "$NS" --to-revision="$PREVIOUS_REVISION"
kubectl rollout status deploy/"$SERVICE" -n "$NS" --timeout=300s
kubectl get deploy "$SERVICE" -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'

The rollback restores production while the investigation continues. See git-cicd-gitops-rb-16-roll-back-deployment for the full procedure.

6. Verify the artifact”s signature and provenance

Read-only / Safe
$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api:v1.2.3"
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
DIGEST=$(crane digest "$REGISTRY/$IMAGE")
echo "--- signature ---"
cosign verify --key "$PUBKEY" "$REGISTRY/$IMAGE@$DIGEST"
echo "--- provenance ---"
cosign verify-attestation --key "$PUBKEY" --type slsaprovenance "$REGISTRY/$IMAGE@$DIGEST" | jq -r '.payload | @base64d | fromjson | .predicate | {buildType: .buildType, source: .invocation.config.source.uri, sha: .invocation.config.source.digest.sha1}'

If the artifact is not what CI built or the signature/provenance does not match, the gap is bigger than a configuration mistake — see the supply-chain runbook.

7. Fix the gap and add coverage

Read-only / Safe
$ echo "--- fix 1: add the missing env var to the production ConfigMap ---"
kubectl patch cm checkout-api-config -n prod -p '{"data":{"DATABASE_URL":"REPLACE_WITH_CORRECT_URL"}}'
echo "--- fix 2: add the production-only dependency to the CI test env ---"
echo 'services:' >> docker-compose.ci.yml
echo '  database:' >> docker-compose.ci.yml
echo '    image: postgres:16' >> docker-compose.ci.yml
echo "--- fix 3: add a smoke test that runs against production after deploy ---"
echo '- name: post-deploy-smoke' >> .github/workflows/deploy.yml
echo '  run: scripts/smoke-test.sh https://checkout-api.prod.internal' >> .github/workflows/deploy.yml
echo "--- fix 4: add a canary stage between CI and full production ---"
echo '- name: deploy-canary' >> .github/workflows/deploy.yml
echo '  if: github.event.inputs.environment == "prod"' >> .github/workflows/deploy.yml

The fix is structural, not punitive. Each fix closes a specific gap. Document each fix in the incident postmortem so the gaps do not recur.

Verification

The production cluster is healthy (all pods Running, no CrashLoopBackOff). The application”s health/readiness endpoint returns ok. The error rate returns to baseline within 5 minutes. The CI artifact deployed matches the CI artifact built. The artifact”s signature and provenance are valid. A smoke test runs against production after the deploy and passes. The CI test environment now exercises the production dependency that caused the failure. The next deploy of the same artifact type succeeds without manual intervention.

Rollback

If the rollback fails, escalate — the deploy is a multi-regression. If the fix cannot be deployed because the deploy pipeline is broken, deploy the fix manually as an emergency change. If the rollback restored production but CI is still wrong, block the CI until the gap is fixed. If the CI test environment is too different from production to be reliable, the fix is structural (staging or canary deploys). If the artifact deployed was not what CI built, see the supply-chain runbook. If the cluster configuration drifted, reconcile the manual change first. If the smoke test passes but the application is still broken, expand the test to cover the broken path.

References

  1. Kubernetes — Pod Lifecycle
  2. Kubernetes — Debug Running Pods
  3. kubectl — describe
  4. GitHub Actions — Required Status Checks
  5. Cosign — Verify
  6. 12-Factor App — Config