Runbook: Vulnerable image response (CVE drop)

1 · Prerequisites

Confirm every item is in place before any state change.

• You have the CVE identifier, its CVSS score, the affected package and the version that fixes it
• You have an inventory of which images and services use the affected base or package
• You have a working build pipeline and can rebuild and push images
• You have a staging environment close enough to production for a smoke test to mean something
• You know the change window and who authorises an out-of-band release
• Set the variables reused below: BASE=docker.io/library/debian:bookworm-slim, IMG=myorg/api:2.4.0, NEW=myorg/api:2.4.1

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · docker scout cves --only-severity critical,high "$IMG" lists the vulnerabilities in the deployed image with the fixed version for each
• · docker scout cves --only-fixed "$IMG" narrows the list to what a rebuild would actually resolve. Anything absent from this list will not be fixed by rebuilding, no matter how many times you try
• · docker scout cves --format sbom "$IMG" produces the package inventory that answers whether the affected package is even present at an affected version
• · docker image inspect --format "{{index .RepoDigests 0}}" "$IMG" records the exact digest currently deployed
• · docker inspect -f "{{.Config.Image}}" on each running container confirms what is actually deployed, which is not always what the pipeline believes
• · docker buildx imagetools inspect "$BASE" prints the current upstream base digest without pulling, so you can tell whether a fixed base has shipped yet
• · grep -rn "^FROM " Dockerfile lists every base this build depends on, including intermediate build stages that are easy to miss
• · docker image ls --filter reference="myorg/*" --format "{{.Repository}}:{{.Tag}}" lists local images that may share the affected base
• · Read the CVE record itself for the affected version range and whether it is known to be exploited. The score alone is not the decision

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Establish exposure before urgency. From the SBOM, confirm the affected package is present at an affected version. A CVE against a package your image does not contain needs only a recorded finding, not a release
2. 2Establish reachability. If the vulnerable code path is never invoked, agree a documented, time-boxed exception with the security team rather than shipping an emergency change that carries its own risk
3. 3Set the timeline from exploitation as well as score. A known-exploited vulnerability is an emergency regardless of CVSS, and a critical score in unreachable code is not
4. 4Check a fixed base actually exists: docker buildx imagetools inspect "$BASE" shows the current upstream digest, and docker scout cves "$BASE" shows whether the CVE is still present in it. Never rebuild against a base that is not fixed
5. 5If no fixed base exists, the options are pinning the patched package in your own Dockerfile, moving to a different base, or applying a compensating control. Choose with the security team; do not silently defer
6. 6Pin the fixed base by digest rather than tag: FROM docker.io/library/debian:bookworm-slim@sha256:REPLACE_ME. A tag can move underneath you, and reproducibility is the entire point of this change
7. 7Rebuild without reusing stale layers: docker build --pull --no-cache -t "$NEW" . The --pull is what fetches the fixed base instead of the local copy, and without it the rebuild can silently produce the same vulnerable image
8. 8Confirm the rebuild fixed it before shipping: docker scout cves --only-severity critical,high --exit-code "$NEW" exits 0. If the CVE is still listed, the rebuild did not take and you must find out why rather than deploying
9. 9Push, then record the new digest: docker image inspect --format "{{index .RepoDigests 0}}" "$NEW". This is what you deploy and what you would roll back from
10. 10Deploy to staging and run the full smoke test, including the specific behaviour the CVE describes. A base bump changes system libraries, locales and TLS defaults, and breaks things unrelated to the CVE
11. 11Deploy one canary replica to production by digest and leave it long enough to observe error rate, latency and restart count against the untouched replicas
12. 12Promote to the remaining replicas only when the canary is indistinguishable from the baseline. If it is not, roll the canary back and treat the regression as a separate incident
13. 13Repeat for every other image sharing the affected base. One remediated service in a fleet of ten is not remediation, and the inventory from the pre-checks is the checklist
14. 14Quarantine the vulnerable image so it cannot be redeployed: remove or re-point the moving tag, and confirm the pipeline can no longer select the old digest
15. 15Communicate on the agreed cadence: a status update every 30 minutes, the security team informed at CVSS 7.0 and above, and compliance informed at CVSS 9.0 and above or for any known-exploited CVE
16. 16Close the loop on prevention: add the fixed base to the pinned set, schedule the base-image refresh, and add a scan gate to the pipeline so the next one is caught at build time rather than in production

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓docker scout cves --only-severity critical,high --exit-code on the deployed digest exits 0
• ✓The CVE identifier does not appear anywhere in docker scout cves output for the deployed image
• ✓docker inspect -f "{{.Config.Image}}" on every replica prints the new digest, with no replica left on the old one
• ✓docker image inspect --format "{{index .RepoDigests 0}}" "$NEW" matches the digest recorded before deploy, proving you shipped what you scanned
• ✓docker scout cves --format sbom on the new image shows the affected package at or above the fixed version
• ✓The number of remediated images equals the number in the pre-check inventory, counted rather than assumed
• ✓docker ps --filter health=unhealthy -q returns no output across the fleet after promotion
• ✓Error rate and latency after promotion sit within the pre-change baseline, not merely under the alert threshold
• ✓The new pipeline scan gate fails a deliberately reintroduced vulnerable build, proving the gate works rather than assuming it does

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The previous image is unchanged and still in the registry, so rollback is a redeploy by digest: pin the compose file to the old digest and run docker compose up -d --no-deps --wait
• ↶Rolling back reintroduces the vulnerability. Do it only with an explicit, recorded decision from the security team and the incident commander, and set a deadline to roll forward again
• ↶If only the canary is bad, roll back the canary alone and leave the rest of the fleet where it is. Never promote a known regression to buy time
• ↶A base bump changes system libraries, locales and TLS defaults. If the regression is subtle, roll back first and diagnose from captured logs rather than debugging in production
• ↶If you quarantined the old tag before verifying the new build, the digest still works even though the tag is gone: docker pull myorg/api@sha256:REPLACE_ME
• ↶A database migration shipped alongside the security rebuild is not undone by rolling the image back. Ship the two changes separately so this runbook never depends on a schema decision

6 · Escalation

When the runbook isn't enough, contact:

• · The CVE is known to be exploited in the wild: escalate to the security team and incident commander immediately and treat it as an active incident, not a patch cycle
• · CVSS 9.0 or above, or regulated data in scope: escalate to compliance and legal on the first update, because notification obligations may already apply
• · No fixed base exists and the vulnerable path is reachable: escalate to the security team for a compensating control and a written, time-boxed risk acceptance
• · The rebuild does not clear the finding: escalate to the image owner. A vulnerable package pulled in by an application dependency rather than by the base needs a different fix entirely
• · The vulnerable image runs on hosts you do not control, or was pulled by a third party: escalate to the platform team to determine full exposure before declaring remediation complete
• · You find evidence the vulnerability was already exploited on your hosts: stop this runbook and switch to the compromised-container runbook, which captures evidence before remediating
• · Hand over: the CVE identifier and score, the affected package and fixed version, the old and new digests, the full list of affected images and which are remediated, and any accepted exception with its deadline

References

1. NIST National Vulnerability Database
2. docker scout cves: --only-severity, --only-fixed, --exit-code, --format sbom
3. docker buildx imagetools inspect - read a base image digest without pulling
4. docker compose up: --no-deps and --wait
5. NIST SP 800-190: Application Container Security Guide