critical risksecurity relevant~90 min

Runbook: Respond to a Compromised Dependency

1 · Prerequisites

Confirm every item is in place before any state change.

Package pinning and lockfiles — npm shrinkwrap, pip-tools, go.sum, Cargo.lock
SBOM distribution and attestation — SBOM as an in-toto attestation
The dependency-compromise attack — the upstream-package scenario
Authority to update dependencies and roll out a fix (or contact with the team that does)
Access to the dependency registry (npm, PyPI, Maven, crates.io, Docker Hub, ghcr.io, etc.)
Access to the SBOM and the build provenance of the current artefacts

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the compromise vector: published package tampered (npm/PyPI/Maven), container image tampered (Docker Hub/GHCR/quay.io), binary downloaded from a CDN was replaced. The source of the alert (advisory, scanner, upstream notification) determines which
· Identify the compromised version range: the advisory timestamp and the package version that introduced the compromise. For npm: npm view <pkg> time; for PyPI: pip index versions <pkg>; for Docker: docker image inspect <image> for the layer timestamps
· Identify every consumer of the compromised version: grep -rl "<package>@<compromised-version>" . for source manifests, and grep -rl "FROM <compromised-image>:<tag>" . for Dockerfiles. Also check CI workflows for transitive dependencies via the SBOM
· Identify a known-good version: a version before the compromise window, with a verifiable build provenance (signature, attestation, or reproducible build). For npm: npm view <pkg> signatures (if the registry supports sigstore); for Docker: cosign verify --key <pubkey> <image>@sha256:<digest>
· Engage security/IR if the compromise involved code execution in production (the malicious code ran, not just was present): the response is parallel with the IR engagement, not after

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Pin every consumer to a known-good version. For npm: update package.json to the pinned version, then npm install to update package-lock.json. For PyPI: pip install <pkg>==<good-version> and update requirements.txt. For Docker: replace <image>:<tag> with <image>@sha256:<digest> (a digest, never a tag, since tags are mutable). The change is in a hotfix branch with reviewer approval
2STEP 2 - Rebuild every artefact that included the compromised version. For Docker: docker buildx build --provenance=true --sbom=true -t <registry>/<image>:<new-tag> . && docker push <registry>/<image>:<new-tag>. For language packages: npm run build && npm publish (for libraries), or rebuild the application image per the Dockerfile
3STEP 3 - Verify the new artefact's signature and provenance. For Docker: cosign verify --key <pubkey> <image>@sha256:<digest> and cosign verify-attestation --key <pubkey> --type slsaprovenance <image>@sha256:<digest>. For npm: npm view <pkg>@<version> dist.signatures. For PyPI: pip install --require-hashes -r requirements.txt (the hashes file is the verification)
4STEP 4 - Roll out the new artefact to staging first. kubectl apply -k overlays/staging/ (or the equivalent), kubectl rollout status deploy/<svc> -n staging --timeout=10m, validate with smoke tests
5STEP 5 - Audit the production environment for any artefact built against the compromised version. For Docker: crictl images | grep <compromised-image>@<compromised-digest> on every production node, and kubectl get pods -A -o jsonpath="{.items[*].status.containerStatuses[*].imageID}" | tr " " "\\n" | sort -u. For language packages: deploy the SBOM-scanning tool (syft, trivy) and check the runtime inventory
6STEP 6 - Roll out the new artefact to production. kubectl apply -k overlays/prod/ (or the equivalent), kubectl rollout status deploy/<svc> -n prod --timeout=10m. Watch the production dashboard for the next 30 minutes
7STEP 7 - If the compromise involved code execution in production: take immediate action beyond the rollout. Roll back the production deploy to the last known-good version, audit for IOCs (indicators of compromise) from the malicious code, and engage IR. The dependency replacement is necessary but not sufficient
8STEP 8 - Update the dependency-pinning policy to prevent the next compromise. Require digests (not tags) for container images, require hashes (--require-hashes) for language packages, and require signature verification as a CI gate. The policy lives in the repo's CONTRIBUTING.md and the CI workflow
9STEP 9 - Add supply-chain detection: subscribe to the GitHub Advisory Database (gh advisory-db, https://github.com/advisories), the npm/PyPI security advisories, and the OSV database (https://osv.dev/). The detection fires when a new advisory affects a dependency in your lockfile. Use Dependabot or Renovate to surface the alert as a PR
10STEP 10 - Document the incident: compromised package/image name, version range, consumer list, hotfix SHA, roll-out SHA, IR findings, and the detection added. This goes into the change ticket and the IR ticket

4 · Verification

Confirm the procedure actually fixed the problem.

✓No consumer in the repo references the compromised version: grep -r "<package>@<compromised-version>" . returns empty. grep -r "FROM <compromised-image>:<compromised-tag>" . returns empty
✓The new artefact's signature verifies: cosign verify --key <pubkey> <image>@sha256:<digest> returns OK
✓The new artefact's SBOM does not include the compromised version: syft <image>@sha256:<digest> -o json | jq '.artifacts[] | select(.name=="<package>") | .version' does not match the compromised version
✓No production node runs the compromised artefact: for n in $(kubectl get nodes -o name); do ssh "$n" sudo crictl images | grep -F "<compromised-digest>"; done returns empty across all nodes
✓The CI gate (signature verification, hash verification) is present in the workflow file and passes on the next PR
✓The dependency-pinning policy is updated in CONTRIBUTING.md and the team is notified

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the new artefact fails verification (signature mismatch, unexpected provenance): do not deploy. Investigate the build pipeline; the rebuild may have introduced a different compromise
↶If the rollout to staging fails: the new artefact is incompatible with the runtime (config drift, missing dependency, library API change). Roll back to the last known-good artefact and re-diagnose
↶If the rollout to production causes a regression: roll back per git-cicd-gitops-rb-05-revert-production-change. The dependency replacement was the wrong fix; the actual bug is elsewhere
↶If the supply-chain detection (Dependabot/Renovate) fires repeatedly on legitimate updates: tune the dependency-pinning policy to allow security updates through faster. A noisy alert trains responders to ignore
↶If the compromised dependency is not replaceable (no known-good version, the project is abandoned): pin to the last version before the compromise window, fork the project, and maintain the fork until upstream recovers. Document the fork in the dependency-pinning policy
↶If the compromised dependency was the only consumer of an internal package: the internal package is also affected. Audit and update the internal package before deploying the upstream replacement

6 · Escalation

When the runbook isn't enough, contact:

· The compromise is widespread (affects hundreds of packages, or the registry itself is compromised): engage the registry's security team and the upstream maintainers. The response is coordinated, not local
· The compromise is in a base image (e.g. node:18, python:3.11): every artefact built on that base is affected. Scope the response to every image built on the compromised base, not just the one that triggered the alert
· The compromise involved code execution and data exfiltration from production: legal/PR involvement, GDPR/CCPA notification may be required. Escalate to legal in parallel with the technical response
· The compromise is from a nation-state actor (indicators of advanced tooling, persistent presence): engage external IR. The compromise is likely broader than the advisory suggests
· The compromised package is in the critical path of production and there is no known-good version: engage the application owner and the platform team to decide whether to ship a workaround, fork the package, or accept the risk temporarily

A dependency compromise is the most common supply-chain attack. A single published version of an upstream package is replaced with a malicious one, every downstream consumer that pulls the new version (or fails to pin) inherits the compromise. The response is: pin every consumer to a known-good version, rebuild every artefact, verify the new artefact”s signature and provenance, roll out, and add detection so the next compromise is caught at the advisory stage, not the runtime stage.

The pinning matters. A tag (npm install left-pad@1.0.0) is a mutable reference; the same tag can resolve to a different version tomorrow. A digest (docker pull left-pad@sha256:abc123) is an immutable reference. The whole supply-chain defence rests on the immutability of the digest.

1. Identify the compromised version

Read-only / Safe

$ PKG="left-pad"
echo '--- npm: timestamps of each version ---'
npm view "$PKG" time --json 2>/dev/null | jq 'to_entries | sort_by(.value) | .[] | {version: .key, published: .value}' | tail -30
echo '--- npm: any deprecation notice ---'
npm view "$PKG" deprecated
echo '--- Docker: layer timestamps of the suspect tag ---'
docker pull "REPLACE_WITH_COMPROMISED_IMAGE:REPLACE_WITH_TAG" 2>&1 | head -10
docker image inspect "REPLACE_WITH_COMPROMISED_IMAGE:REPLACE_WITH_TAG" --format '{{ index .Config "created" }} {{ index .Config "Env" }}'
echo '--- OSV: search for the package ---'
curl -fsS -X POST https://api.osv.dev/v1/query -d '{"package":{"name":"'"$PKG"'","ecosystem":"npm"}}' | jq '.vulns[] | {id, summary, affected: .affected}' | head -60

The OSV API is the authoritative source for known advisories. The npm view time output gives the publish timestamps; cross-reference against the advisory window to find the compromised version range.

2. Identify every consumer

Read-only / Safe

$ PKG="left-pad"
COMPROMISED_VERSION="1.0.1"
echo '--- direct consumers in source ---'
grep -rln "$PKG@$COMPROMISED_VERSION" --include='package.json' --include='package-lock.json' --include='requirements.txt' --include='Pipfile.lock' --include='go.mod' --include='Cargo.toml' .
echo '--- Docker images from the compromised base ---'
grep -rln "FROM .*REPLACE_WITH_COMPROMISED_IMAGE:REPLACE_WITH_COMPROMISED_TAG" --include='Dockerfile*' .
echo '--- transitive consumers via SBOM ---'
for image in $(kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '
' | sort -u); do
syft "$image" -o json 2>/dev/null | jq --arg pkg "$PKG" '.artifacts[] | select(.name==$pkg) | {image: "'"$image"'", version: .version}'
done | head -20

The direct consumer list is the authoritative starting point. The transitive consumer list (via SBOM) catches dependencies the team did not know were in use.

3. Identify a known-good version

Read-only / Safe

$ PKG="left-pad"
GOOD_VERSION="1.0.0"
echo '--- npm: signature of the good version ---'
npm view "$PKG@$GOOD_VERSION" dist
echo '--- verify the integrity hash matches the lockfile ---'
grep -A2 '"'"$PKG"'":' package-lock.json | head -10
echo '--- Docker: verify the good image''s signature ---'
cosign verify --key REPLACE_WITH_PUBKEY "REPLACE_WITH_IMAGE@sha256:REPLACE_WITH_GOOD_DIGEST" 2>&1 | head -20
echo '--- Docker: verify the SBOM does not include the compromised version ---'
cosign verify-attestation --key REPLACE_WITH_PUBKEY --type spdxjson "REPLACE_WITH_IMAGE@sha256:REPLACE_WITH_GOOD_DIGEST" 2>&1 | head -10
syft "REPLACE_WITH_IMAGE@sha256:REPLACE_WITH_GOOD_DIGEST" -o json | jq --arg pkg "$PKG" '.artifacts[] | select(.name==$pkg) | .version'

If cosign verify returns OK, the image”s signature is valid and the digest can be trusted. If it returns an error, the registry”s signing identity has been compromised; escalate to the registry operator.

4. Pin every consumer to the known-good version

Read-only / Safe

$ PKG="left-pad"
GOOD_VERSION="1.0.0"
GOOD_DIGEST="sha256:abc123..."
echo '--- for npm: update package.json and lockfile ---'
sed -i "s/"$PKG": "\^[0-9.]\+"/"$PKG": "$GOOD_VERSION"/" package.json
npm install --package-lock-only
git diff package.json package-lock.json
echo '--- for Docker: pin to digest ---'
sed -i "s|FROM REPLACE_WITH_IMAGE:REPLACE_WITH_TAG|FROM REPLACE_WITH_IMAGE@$GOOD_DIGEST|g" Dockerfile
git diff Dockerfile
echo '--- for Go: update go.mod ---'
go get "$PKG@$GOOD_VERSION"
go mod tidy
git diff go.mod go.sum

The lockfile update is critical. Without it, the next npm install or go build may pull the compromised version again.

5. Rebuild every artefact

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
NEW_TAG="v1.2.3-hotfix-$(date -u +%Y%m%d)"
docker buildx build \
--provenance=true \
--sbom=true \
--tag "$REGISTRY/$IMAGE:$NEW_TAG" \
--tag "$REGISTRY/$IMAGE:$NEW_TAG@sha256:$(docker inspect --format='{{index .Id}}' $REGISTRY/$IMAGE:$NEW_TAG)" \
--push .
docker buildx imagetools inspect "$REGISTRY/$IMAGE:$NEW_TAG" --raw | jq '. | {digest, signatures}'

--provenance=true --sbom=true requires BuildKit to emit SLSA provenance and an SBOM as attestations. The attestations are signed by the registry”s signing identity and verifiable with cosign verify-attestation.

6. Verify the new artefact

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
NEW_DIGEST="sha256:def456..."
echo '--- signature ---'
cosign verify --key REPLACE_WITH_PUBKEY "$REGISTRY/$IMAGE@$NEW_DIGEST"
echo '--- provenance ---'
cosign verify-attestation --key REPLACE_WITH_PUBKEY --type slsaprovenance "$REGISTRY/$IMAGE@$NEW_DIGEST"
echo '--- SBOM does not include the compromised version ---'
cosign verify-attestation --key REPLACE_WITH_PUBKEY --type spdxjson "$REGISTRY/$IMAGE@$NEW_DIGEST" | jq -r '.payload | @base64d | fromjson | .packages[] | select(.name=="left-pad") | .versionInfo' | grep -v "1.0.1" || echo 'compromise not present'

If the SBOM still contains the compromised version, the rebuild did not actually replace the dependency — the Dockerfile or lockfile update did not take effect. Re-verify the source files before redeploying.

7. Audit the production environment

Read-only / Safe

$ COMPROMISED_DIGEST="sha256:abc123..."
echo '--- check every node for the compromised digest ---'
for n in $(kubectl get nodes -o name); do
echo "=== $n ==="
ssh "$n" sudo crictl images | grep -F "$COMPROMISED_DIGEST" && echo 'COMPROMISED' || echo 'clean'
done
echo '--- check pods ---'
kubectl get pods -A -o jsonpath='{.items[*].status.containerStatuses[*].imageID}' | tr ' ' '
' | sort -u | grep -F "$COMPROMISED_DIGEST" && echo 'COMPROMISED' || echo 'clean'

If a node still runs the compromised digest, the production rollout has not reached that node yet, or the digest is cached locally. Drain the affected workload to a clean node and purge the cri-tools image cache.

8. Roll out the new artefact

Read-only / Safe

$ echo '--- staging first ---'
kubectl apply -k overlays/staging/
kubectl rollout status deploy/REPLACE_WITH_SVC -n staging --timeout=10m
curl -fsS https://REPLACE_WITH_SVC.staging.example.com/healthz

echo '--- then production ---'
kubectl apply -k overlays/prod/
kubectl rollout status deploy/REPLACE_WITH_SVC -n prod --timeout=10m
echo '--- production dashboard for the next 30 minutes ---'
echo 'https://grafana.example.com/d/REPLACE_WITH_SVC-prod/overview'

Staging first because a hotfix that breaks the application permanently is worse than the compromise. Production second, with a watch window.

9. Add supply-chain detection

Read-only / Safe

$ echo '--- enable Dependabot security updates ---'
cat >> .github/dependabot.yml <<'EOF'
version: 2
updates:
- package-ecosystem: "npm"
  directory: "/"
  schedule:
    interval: "daily"
  open-pull-requests-limit: 10
- package-ecosystem: "docker"
  directory: "/"
  schedule:
    interval: "daily"
  open-pull-requests-limit: 10
- package-ecosystem: "github-actions"
  directory: "/"
  schedule:
    interval: "daily"
  open-pull-requests-limit: 10
EOF
echo '--- enable Renovate as an alternative ---'
cat > .renovate.json5 <<'EOF'
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", ":automergeMinor"],
"vulnerabilityAlerts": { "enabled": true },
"osvVulnerabilityAlerts": { "enabled": true }
}
EOF
echo '--- gate CI on signature verification ---'
cat >> .github/workflows/ci.yml <<'EOF'
verify-signatures:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: |
        cosign verify --key REPLACE_WITH_PUBKEY "$IMAGE@$DIGEST"
EOF

Dependabot and Renovate both surface advisories as PRs. The CI gate prevents an unsigned or unverified artefact from being deployed.

10. Document the incident

Read-only / Safe

$ gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "dependency compromise $(date -u +%Y-%m-%d)" \
--body "Package: REPLACE_WITH_NAME. Compromised version: REPLACE_WITH_V. Known-good version: REPLACE_WITH_V. Hotfix SHA: REPLACE_WITH_SHA. Roll-out SHA: REPLACE_WITH_SHA. Detection added: <Dependabot, Renovate>. CI gate: <signature verify>. IR ticket: REPLACE_WITH_LINK." \
--label security --label supply-chain --label incident

Verification

No consumer in the repo references the compromised version. The new artefact”s signature verifies with cosign verify. The new artefact”s SBOM does not include the compromised version. No production node runs the compromised digest. The CI gate (signature verification) is present in the workflow file and passes on the next PR. The dependency-pinning policy is updated and the team is notified.

Rollback

If the new artefact fails verification, do not deploy; investigate the build pipeline. If the rollout to staging fails, roll back to the last known-good artefact. If the rollout to production causes a regression, roll back per git-cicd-gitops-rb-05-revert-production-change. If the supply-chain detection fires repeatedly on legitimate updates, tune the policy. If the compromised dependency has no known-good version, pin to the last version before the compromise window and fork the project. If the compromised dependency is the only consumer of an internal package, audit and update the internal package first.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Identify the compromised version

2. Identify every consumer

3. Identify a known-good version

4. Pin every consumer to the known-good version

5. Rebuild every artefact

6. Verify the new artefact

7. Audit the production environment

8. Roll out the new artefact

9. Add supply-chain detection

10. Document the incident

Verification

Rollback

References