← All runbooks in Git, CI/CD & GitOps
critical risksecurity relevant~120 min
Runbook: Respond to a Supply-Chain Compromise
1 · Prerequisites
Confirm every item is in place before any state change.
- git-cicd-gitops-rb-07-respond-to-secret-committed
- git-cicd-gitops-rb-08-rotate-git-credentials
- git-cicd-gitops-rb-13-respond-to-compromised-runner
- git-cicd-gitops-rb-14-respond-to-compromised-dependency
- git-cicd-gitops-rb-25-rotate-deploy-identity
- git-cicd-gitops-rb-15-validate-production-artifact
- Access to the security team and the incident response process
- A documented supply-chain security policy (SLSA level, signing requirements, allowed sources)
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the compromise is real, not a false positive. Indicators: an unexpected dependency in the SBOM; a cosign signature that does not verify; a commit from an unexpected user; a credential used from an unexpected IP; a CI run that does not match the change ticket. Cross-reference with the security team's threat intelligence
- · Identify the compromise vector. Common vectors: a compromised dependency (typosquat, malicious update); a compromised CI runner (supply-chain attack on the build infrastructure); a compromised credential (a developer's token was phished); a compromised Git provider account; a compromised build cache; a compromised container base image. The vector determines the response
- · Identify the blast radius. Every artifact built since the compromise entered the pipeline is suspect. List the affected artifacts:
gh run list --workflow=build.yml --since=<compromise-time>for each CI run;crane ls <registry>for each image. Every artifact must be considered potentially malicious - · Identify the affected environments. Which clusters, which namespaces, which services are running the suspect artifacts?
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\\n' | sort -u | grep -F <suspect-image>. The blast radius determines the response priority - · Engage the security team immediately. A supply-chain compromise is a security incident, not a CI/CD incident. The security team leads the investigation; CI/CD provides the technical context. The handoff must be clear
- · Preserve evidence. Take a snapshot of the affected systems:
kubectl get all -A -o yaml > /tmp/affected-cluster.yaml,gh run view <run-id> --log > /tmp/run-log.txt,crane manifest <image> > /tmp/image-manifest.json. The evidence is the basis for the post-incident analysis and any legal action - · Stop further deploys.
gh workflow disable <wf>for the affected workflows;argocd app set <app> --sync-policy nonefor the affected Applications. The stop is the first priority
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1STEP 1 - Contain the compromise. Cut off the attack vector: disable the compromised workflow, freeze the affected GitOps Applications, revoke the compromised credential, remove the malicious dependency from the build pipeline. The containment is the first action and the highest priority
- 2STEP 2 - For a compromised dependency: identify the exact version and the replacement.
npm audit/pip-audit/cargo audit/govulncheckidentifies the vulnerability; the dependency lockfile (package-lock.json,Pipfile.lock,Cargo.lock) shows which version is installed. Update to a known-good version (or remove the dependency if no replacement is available) and re-build - 3STEP 3 - For a compromised CI runner: the runner must be considered fully compromised. See
git-cicd-gitops-rb-13-respond-to-compromised-runner. Destroy the runner, rotate every credential that was on it, audit every build the runner produced. The runner is the attack vector; the artifacts it produced are suspect - 4STEP 4 - For a compromised credential: see
git-cicd-gitops-rb-08-rotate-git-credentialsandgit-cicd-gitops-rb-25-rotate-deploy-identity. The credential must be revoked, rotated, and the audit log reviewed for misuse. Every system that trusted the credential must be re-keyed - 5STEP 5 - For a compromised Git provider account: every commit, branch, and tag from the account since the compromise is suspect.
git log --author=<compromised-user> --since=<time>enumerates the suspect commits. Audit each one. Rotate the user's credentials and remove the account from the organization if necessary - 6STEP 6 - For a compromised artifact (image, binary): every deployment of the artifact is suspect.
crane ls <registry>/<image> | sort -Vshows all tags; the suspect tags are those built since the compromise time. Roll back to a known-good tag and remove the suspect tags from the registry. Seegit-cicd-gitops-rb-22-recover-registry-outagefor the registry-side recovery - 7STEP 7 - Identify the production blast radius.
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | grep -F <suspect-image>. Every pod running the suspect image is a potential victim. For each, snapshot the pod (kubectl get pod <pod> -n <ns> -o yaml > /tmp/pod.yaml) and the logs (kubectl logs <pod> -n <ns> > /tmp/pod.log). The evidence is critical for the post-incident analysis - 8STEP 8 - For pods running the suspect artifact: contain by cordoning (if the compromise is still active) or quarantining (if the compromise has been contained).
kubectl cordon <node>to prevent new pods;kubectl delete pod <pod> -n <ns>to remove the compromised pod. The replacement pod should pull the known-good image - 9STEP 9 - Rebuild the artifacts from a known-good baseline. The CI pipeline must be exercised against a known-good source SHA, with a known-good dependency set, on a fresh runner, with rotated credentials. The new artifacts must be signed and verified:
cosign sign --key <key> <image>andcosign verify --key <key> <image> - 10STEP 10 - Redeploy the known-good artifacts. Update the GitOps manifests in Git to reference the known-good image tags. Push. The controller reconciles. Verify every Application is Synced + Healthy with the new image digest
- 11STEP 11 - Audit every system the compromise could have touched. The compromise may have spread beyond the original artifact: credentials cached on disk, SSH keys in
~/.ssh, environment variables in shell history, build caches, container layers in the registry. Audit each:find / -name '*.env' 2>/dev/null | head;grep -r <compromised-secret> .;crane ls <registry> --include <compromised-namespace> - 12STEP 12 - Notify stakeholders and regulators. The compromise must be communicated to the application owners, the security team, the engineering leadership, and (if applicable) the regulators (GDPR, HIPAA, PCI, SOX). The notification must include: what was compromised, when, the blast radius, the affected data (if any), the remediation, the timeline. The communication is both a legal and a reputational obligation
- 13STEP 13 - Conduct a post-incident review. The review identifies the root cause (how did the compromise enter the pipeline?), the gaps (why was it not caught earlier?), and the improvements (what would have detected it?). The review produces a list of action items with owners and dates. The action items close the gaps and prevent recurrence
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓The compromise vector is cut off: the workflow is disabled, the credential is revoked, the dependency is removed. Every action that the attacker could have taken is now blocked
- ✓Every affected artifact has been identified and quarantined. The registry does not contain the suspect tags (or the tags are marked deprecated and cannot be pulled)
- ✓The production cluster is running the known-good artifact:
kubectl get pods -A -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'shows the known-good image digest - ✓The new artifacts are signed and verified:
cosign verify --key <key> <image>returns "Verified OK" - ✓The audit log shows no further use of the compromised credential since the cutover. The credential is revoked at the source
- ✓The security team has confirmed the compromise is contained and the blast radius is fully understood. No further suspicious activity is observed
- ✓The post-incident review has produced action items with owners and dates. The action items are tracked in the issue tracker
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If the containment was too aggressive (disabled a workflow that was not actually compromised): re-enable the workflow after verifying it is clean. Audit the workflow's recent runs before re-enabling. The re-enable must not happen until the security team approves
- ↶If the rollback to a known-good artifact causes a regression (the known-good artifact is also broken or has a known vulnerability): the rollback is not possible. Engage the security team and the application owner. The compromise has revealed a deeper problem; the artifact's history must be audited
- ↶If the production cluster cannot be reconciled to the known-good artifact (the controller is broken, the registry is down, the Git provider is down): see the respective runbooks. The containment is partial until the cluster is reconciled
- ↶If the compromise has spread beyond the original artifact (lateral movement in the cluster, exfiltration of data): the blast radius is larger than initially understood. Engage the incident response team and the security team. The response is now a full security incident, not a CI/CD incident
- ↶If the audit reveals credentials that were not rotated (a backup credential, a personal token, a service account key that was forgotten): rotate them immediately. The compromise may continue through the forgotten credential
- ↶If the rebuild produces artifacts that still fail cosign verification: the signing infrastructure is compromised. Engage the security team; do not deploy unsigned artifacts. The rebuild must wait for the signing infrastructure to be secured
- ↶If the post-incident review is delayed or skipped: the gaps that allowed the compromise remain. The next compromise will be worse. Engage the engineering leadership; the review is non-negotiable
6 · Escalation
When the runbook isn't enough, contact:
- · The compromise is in progress (the attacker is still in the pipeline, still deploying, still exfiltrating): engage the incident response team immediately. The CI/CD team supports the IR team; the IR team leads the response. Containment is the priority; investigation follows
- · Customer data has been exfiltrated or modified: this is a data breach. Engage the legal team, the communications team, and (where applicable) the regulator. The 72-hour GDPR clock, the HIPAA breach notification, the PCI forensic investigation, the SOX disclosure — all apply
- · The compromise is a known supply-chain attack (SolarWinds, Log4Shell, xz-utils): engage the CISA / national CERT. The response is now a coordinated disclosure. The team may need to share indicators of compromise (IoCs) with the broader community
- · The compromised artifact is running in a regulated environment (PCI, HIPAA, FedRAMP): the regulator may require a formal incident report. Engage the compliance team. The report must include the timeline, the blast radius, the remediation, and the prevention measures
- · The compromise was made possible by a process gap (no SBOM scanning, no signature verification, no dependency pinning): the gap must be closed. Engage the platform team and the security team. The fix is structural (SLSA L3+ supply chain), not a one-off patch
- · The attacker has demonstrated persistence (still in the pipeline after the initial containment): the attacker has a backdoor. Engage the incident response team for a forensic analysis. The CI/CD system must be considered fully compromised and may need to be rebuilt from scratch
- · The compromise affects multiple teams or multiple organizations (a popular open-source dependency was compromised): coordinate with the upstream maintainer and the broader community. The fix requires coordination; a unilateral response may be insufficient
A supply-chain compromise is a security incident, not a CI/CD incident. The CI/CD system is the venue; the attack is the compromise of an artifact, a dependency, a credential, or a build infrastructure. The response is containment, blast-radius identification, credential rotation, rebuild from a known-good baseline, and post-incident review.
1. Contain the compromise
Read-only / Safe
$ echo "--- disable affected workflows ---"
gh workflow disable build.yml
gh workflow disable deploy.yml
echo "--- freeze affected GitOps Applications (disable automated sync) ---"
argocd app set checkout-api --sync-policy none
argocd app set frontend --sync-policy none
echo "--- revoke the compromised credential ---"
gh token revoke REPLACE_WITH_COMPROMISED_TOKEN
echo "--- preserve evidence ---"
kubectl get all -A -o yaml > /tmp/affected-cluster-$(date -u +%s).yaml
gh run view REPLACE_WITH_RUN_ID --log > /tmp/run-$(date -u +%s).log
crane manifest REPLACE_WITH_REGISTRY/REPLACE_WITH_IMAGE:REPLACE_WITH_TAG > /tmp/manifest-$(date -u +%s).jsonThe containment is the first action. Every minute of delay allows the compromise to spread. The evidence snapshot is the basis for the post-incident analysis.
2. Identify the compromise vector and blast radius
Read-only / Safe
$ COMPROMISE_TIME="2026-08-15T00:00:00Z"
echo "--- suspect artifacts since compromise time ---"
gh run list --workflow=build.yml --created=">=$COMPROMISE_TIME" --json databaseId,conclusion,headBranch | jq '.[] | {id: .databaseId, conclusion: .conclusion, branch: .headBranch}'
echo "--- suspect image tags ---"
crane ls REPLACE_WITH_REGISTRY/myorg/checkout-api | sort -V | tail -10
echo "--- which clusters/namespaces are running suspect images? ---"
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u | grep REPLACE_WITH_SUSPECT_PATTERN
echo "--- suspect dependencies ---"
npm audit --json | jq '.vulnerabilities'
echo "--- the suspect commit/dependency ---"
git log --since="$COMPROMISE_TIME" --all --oneline | headThe blast radius determines the response priority. The vector determines the remediation.
3. For a compromised dependency
Read-only / Safe
$ PKG="myorg/checkout-api"
echo "--- find the compromised version ---"
npm ls myorg/checkout-api 2>&1 | head
echo "--- check the lockfile ---"
grep -A2 "myorg/checkout-api" package-lock.json
echo "--- update to a known-good version ---"
npm install myorg/checkout-api@REPLACE_WITH_KNOWN_GOOD_VERSION
echo "--- or remove if no replacement ---"
npm uninstall myorg/checkout-api
echo "--- re-build ---"
gh workflow run build.yml
gh run watchThe lockfile pins the exact version. The compromised version is replaced with a known-good version. The build is re-run to produce clean artifacts.
4. For a compromised credential
Read-only / Safe
$ OLD_TOKEN="ghp_compromised..."
echo "--- revoke ---"
gh auth logout --hostname github.com
gh token revoke "$OLD_TOKEN"
echo "--- audit misuse ---"
aws cloudtrail lookup-events --lookup-attributes AttributeKey=User,AttributeValue=REPLACE_WITH_COMPROMISED_USER
gh api 'repos/myorg/myrepo/actions/runs?per_page=20' --jq '.workflow_runs[] | {id, head_branch, event, created_at, actor: .actor.login}'
echo "--- rotate to OIDC (preferred) ---"
gh workflow enable oidc.yml
echo "--- or issue a new token with minimal scope ---"
gh token create --scopes 'repo:read,workflow'The credential is revoked at the source. The audit log is reviewed for misuse. A new credential is issued with minimal scope; OIDC is preferred over long-lived tokens.
5. Quarantine suspect artifacts and rebuild
Read-only / Safe
$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api"
echo "--- identify suspect tags ---"
SUSPECT_TAGS=$(crane ls "$REGISTRY/$IMAGE" | sort -V | tail -5)
echo "$SUSPECT_TAGS"
echo "--- delete suspect tags from the registry ---"
for tag in $SUSPECT_TAGS; do
echo "deleting: $tag"
crane delete "$REGISTRY/$IMAGE:$tag" 2>&1 || echo "delete failed (may need registry admin)"
done
echo "--- rebuild from a known-good source SHA ---"
KNOWN_GOOD_SHA="abc123def456"
git checkout "$KNOWN_GOOD_SHA"
gh workflow run build.yml
gh run watch
echo "--- sign the new artifact ---"
cosign sign --key cosign.key "$REGISTRY/$IMAGE@$(crane digest $REGISTRY/$IMAGE:rebuilt)"
echo "--- verify ---"
cosign verify --key cosign.pub "$REGISTRY/$IMAGE@$(crane digest $REGISTRY/$IMAGE:rebuilt)"The suspect tags are removed from the registry. The rebuild is from a known-good source SHA on a fresh runner. The new artifact is signed and verified.
6. Redeploy the known-good artifact
Read-only / Safe
$ GITOPS_REPO="/path/to/gitops-prod"
KNOWN_GOOD_TAG="v1.2.3-known-good"
cd "$GITOPS_REPO"
sed -i "s|image:.*checkout-api:.*|image: ghcr.io/myorg/checkout-api:$KNOWN_GOOD_TAG|" apps/checkout-api/deployment.yaml
git diff
git add -A
git commit -m "security: redeploy known-good checkout-api (incident REPLACE_WITH_INC_ID)"
git push
echo "--- reconcile ---"
argocd app sync checkout-api --revision HEAD
argocd app wait checkout-api --health --timeout 300
echo "--- verify the digest ---"
kubectl get pods -n prod -l app=checkout-api -o jsonpath='{.items[*].status.containerStatuses[*].imageID}{"\n"}'The known-good tag is the new source of truth. The GitOps controller redeploys automatically. The digest verification proves the cluster is running the known-good artifact.
7. Notify and document
Read-only / Safe
$ INCIDENT_ID="SEC-2026-001"
gh issue create --repo myorg/myorg --title "supply-chain compromise: $INCIDENT_ID" \
--body "Vector: REPLACE_WITH_DESCRIPTION. Blast radius: REPLACE_WITH_LIST. Affected data: REPLACE_WITH_LIST. Remediation: redeployed known-good artifact. Post-incident review: REPLACE_WITH_DATE. Action items: REPLACE_WITH_LIST." \
--label security-incident --label supply-chain --label p1
echo "--- or open a security advisory ---"
gh api repos/myorg/myorg/security-advisories --method POST \
-f summary="supply-chain compromise" \
-f description="REPLACE_WITH_FULL_INCIDENT_DESCRIPTION" \
-f severity=criticalThe notification closes the loop with stakeholders and regulators. The post-incident review is scheduled; the action items are tracked.
Verification
The compromise vector is cut off. Every affected artifact has been identified and quarantined. The production cluster is running the known-good artifact. The new artifacts are signed and verified. The audit log shows no further use of the compromised credential. The security team has confirmed the compromise is contained. The post-incident review has produced action items with owners and dates.
Rollback
If the containment was too aggressive, re-enable the workflow after security approves. If the rollback to a known-good artifact causes a regression, engage security and the application owner. If the cluster cannot be reconciled, see the registry and GitOps runbooks. If the compromise has spread (lateral movement, exfiltration), engage the incident response team. If the audit reveals credentials that were not rotated, rotate them immediately. If the rebuild produces artifacts that fail cosign verification, engage security — do not deploy unsigned artifacts. If the post-incident review is delayed or skipped, engage engineering leadership — the review is non-negotiable.