Runbook: Compromised container response

1 · Prerequisites

Confirm every item is in place before any state change.

• A security incident is declared and the security team and incident commander are engaged before you touch the host
• You have written authorisation to capture forensic artefacts from this container, including any data it holds
• You have an evidence destination that is not on the compromised host, and enough space on it for a full image and rootfs export
• You have out-of-band access to the host that does not depend on the compromised service
• You have an inventory of what this container could read: environment variables, mounted files, and any cloud instance-metadata role
• Set the variables reused below: CNAME=api, then CID=$(docker inspect -f "{{.Id}}" "$CNAME"), then EVID=/var/tmp/evidence-"$CNAME"-$(date -u +%Y%m%dT%H%M%SZ)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Do not run docker exec against this container and do not stop it yet. Both destroy volatile evidence, and exec also tells an active intruder that you are present. Every check below reads from the host or the daemon instead
• · docker ps --filter name="$CNAME" --format "{{.ID}} {{.Image}} {{.Status}} {{.Networks}}" identifies the container, its image and every network it can reach
• · docker inspect -f "{{.Config.Image}} {{.Image}}" "$CID" prints the image reference and the resolved image ID, so you can later tell whether the image itself was the vector
• · docker inspect -f "{{.HostConfig.Privileged}} {{json .HostConfig.CapAdd}} {{json .HostConfig.SecurityOpt}}" "$CID" shows whether this container could escape trivially
• · docker inspect -f "{{json .Mounts}}" "$CID" lists every host path and volume it can reach. A mount of /var/run/docker.sock or of / means the host is compromised, not just the container
• · docker inspect -f "{{json .NetworkSettings.Networks}}" "$CID" lists the networks and addresses, which bounds how far lateral movement could have gone
• · docker top "$CID" aux lists the processes inside, read from the host namespace without entering the container
• · docker diff "$CID" lists every file added, changed or deleted in the writable layer since the image. A dropped payload almost always shows up here
• · sudo ss -tnp state established prints host-wide established connections. Correlate the PIDs from docker top against unexpected remote addresses
• · docker logs --timestamps --tail 200 "$CID" shows the recent application log without entering the container
• · docker events --since 24h --filter container="$CNAME" prints start, exec and mount events. An unexpected exec_create means someone used the daemon itself

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Capture before you contain, and contain before you eradicate. Create the evidence directory first: mkdir -p "$EVID", and record its path in the incident ticket
2. 2Freeze the process state without killing it: docker pause "$CID". The cgroup freezer suspends every process while leaving memory, open sockets and the filesystem exactly as they are. docker ps then shows the status as Paused
3. 3Capture the daemon-side metadata: docker inspect "$CID" > "$EVID"/inspect.json. Once the container is removed this is the only record of its environment and mounts
4. 4Capture the full log: docker logs --timestamps "$CID" > "$EVID"/container.log 2>&1. Container logs are deleted with the container
5. 5Capture the writable layer as an image: docker commit --no-pause "$CID" forensics/"$CNAME":captured, then docker save forensics/"$CNAME":captured -o "$EVID"/image.tar. --no-pause is correct here only because the container is already paused
6. 6Capture the filesystem as a flat archive: docker export "$CID" -o "$EVID"/rootfs.tar. Note that docker export excludes volume contents, so back up any in-scope volumes separately
7. 7Capture the runtime state you read earlier: docker top "$CID" aux > "$EVID"/processes.txt and docker diff "$CID" > "$EVID"/filesystem-diff.txt
8. 8Hash every artefact so the chain of custody holds: sha256sum "$EVID"/* > "$EVID"/SHA256SUMS, then copy the whole directory off the host to the evidence store
9. 9Only now contain. Cut the network while leaving the process state intact: docker network disconnect -f NETWORK "$CID" for each network the pre-checks listed. The container keeps running but can reach nothing
10. 10If a network cannot be disconnected, drop the traffic at the host instead: IP=$(docker inspect -f "{{.NetworkSettings.IPAddress}}" "$CID"), then sudo iptables -I DOCKER-USER -s "$IP" -j DROP and sudo iptables -I DOCKER-USER -d "$IP" -j DROP. Use DOCKER-USER, which the daemon evaluates before its own rules and never rewrites
11. 11Confirm containment before going further: docker inspect -f "{{json .NetworkSettings.Networks}}" "$CID" prints {}, or sudo iptables -S DOCKER-USER lists your DROP rules ahead of everything else
12. 12Stop the container only once every artefact is captured and hashed: docker stop "$CID". Do not remove it - the writable layer is still evidence
13. 13Rotate every secret from the mount and environment inventory: application credentials, database passwords, API tokens, and any cloud role reachable through instance metadata. Assume all of them are disclosed, because you cannot prove otherwise
14. 14Revoke the registry credentials and any access token the container held, then confirm the old value is now rejected
15. 15Name the entry point from the evidence before you rebuild: the changed files in filesystem-diff.txt, the unexpected process in processes.txt, the first anomalous line in container.log. Redeploying without this only resets the clock
16. 16Rebuild from a known-good source, never from the running container or the forensic image. Build again from the reviewed Dockerfile at a known-good commit, with base images pinned by digest
17. 17Scan the rebuild before it goes anywhere: docker scout cves --only-severity critical,high --exit-code myorg/"$CNAME":rebuilt exits 0, or you have a written exception
18. 18Close the specific gap you found. At minimum run as a non-root user, add --read-only with an explicit tmpfs for writable paths, and use --cap-drop ALL plus only the capabilities the workload demonstrably needs. Never redeploy with --privileged and never mount the Docker socket
19. 19Deploy the rebuilt image as a new container, with a new name, on a new network, using the rotated secrets
20. 20Retain the compromised container and its evidence until the security team releases them in writing. Only then docker rm "$CID" and remove the forensic image

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓sha256sum -c "$EVID"/SHA256SUMS reports OK for every artefact, and a copy of the directory exists off the host
• ✓docker inspect -f "{{json .NetworkSettings.Networks}}" "$CID" prints {} - the compromised container has no network attachment
• ✓docker inspect -f "{{.State.Status}}" "$CID" prints exited
• ✓Every credential in the inventory is rotated. Test at least one old value against its service, confirm it is rejected, and record the result
• ✓docker inspect -f "{{.Config.User}}" on the replacement prints a non-root user, not an empty string
• ✓docker inspect -f "{{.HostConfig.Privileged}} {{.HostConfig.ReadonlyRootfs}}" on the replacement prints false true
• ✓docker inspect -f "{{json .Mounts}}" on the replacement contains no /var/run/docker.sock and no bind of /
• ✓docker scout cves --only-severity critical,high --exit-code on the replacement image exits 0
• ✓docker diff on the replacement after an hour of normal traffic shows only paths an application is expected to write
• ✓The indicator that triggered the alert - process name, destination address, file path - is absent from every other host. Confirm by searching the fleet, not by assuming

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶There is deliberately no rollback to the compromised state. Restoring the original container, image or secrets re-establishes the intruder. Every artefact here is evidence, never a restore source
• ↶The forensic capture is additive and changes nothing about the incident. Delete the evidence directory only when the security team releases it in writing
• ↶docker pause is reversible with docker unpause, but do not unpause a container already assessed as compromised except on explicit direction from the security team
• ↶A network disconnect is reversible with docker network connect, and a DOCKER-USER rule with sudo iptables -D DOCKER-USER, but both re-expose the intruder. Neither is a rollback you perform in a hurry to fix a mistake
• ↶Secret rotation cannot be undone and must not be. If a rotated credential breaks a downstream consumer, fix the consumer with the new value rather than restoring the old one
• ↶If the replacement deployment fails, roll forward to another verified good image. Never roll back to the compromised one to restore service

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate to the security team immediately, before any remediation. This runbook assumes they are already engaged: containment without their agreement can destroy evidence or alert an active intruder
• · The container mounted /var/run/docker.sock, ran with --privileged, or bind-mounted the host root: treat the host as compromised rather than the container. Escalate for host isolation and rebuild, because nothing done inside Docker is sufficient
• · Evidence of lateral movement, such as the same indicator on another host or connections to other internal addresses: escalate to the incident commander to widen scope to the whole environment
• · Any indication that personal, payment or otherwise regulated data was reachable: escalate to legal, privacy and compliance immediately, as statutory notification clocks may already be running
• · The vector was the base image or a dependency rather than the running workload: escalate to the supply-chain owner and switch to the vulnerable-image runbook for the fleet-wide rebuild
• · You cannot identify the entry point from the captured evidence: escalate to digital forensics before removing anything, rather than redeploying and hoping
• · Hand over: the evidence path and its SHA256SUMS, the inspect output, the mount and environment inventory, every containment action with its timestamp, the list of rotated secrets, and every indicator of compromise observed

References

1. NIST SP 800-190: Application Container Security Guide
2. docker container commit
3. docker container export
4. docker image save
5. docker network disconnect
6. docker scout cves
7. docker inspect
8. Docker and iptables — the DOCKER-USER chain