Containment — shrinking the blast radius without losing the evidence

Containment is the step between “we know something is wrong” and “we know what”. Its goal is narrow: stop the damage spreading, while giving up as little as possible of the ability to explain it later.

Those two goals pull against each other, and the resolution is a ladder. Every rung contains more and destroys more. Climb to the rung that achieves your goal and stop there.

The ladder

Most engineers under pressure jump to rung 5 or 6 because those are the commands they type every day. Rungs 1 to 4 are the ones that make a postmortem possible, and they are all reversible.

Rung 1: contain at the edge first

The cheapest containment does not touch the container at all. Take the instance out of the proxy or load balancer and reload:

# In the upstream block, mark the instance down rather than deleting it,
# so the change is one word to reverse.
#   server app-b.example.com:8080 down;

docker exec proxy nginx -t
docker kill -s HUP proxy

Users stop reaching the broken instance within one reload. The container keeps running with every socket, every thread and every byte of heap intact, and you can take as long as you need over it.

This is almost always the correct first move for a service-level problem, and it is the one people skip because it involves a different system from the one that is broken.

Rung 2: isolate the container

When the container itself is the danger — it is hammering a dependency, or it is compromised — cut it off the network without killing it:

$ docker network disconnect proj_default proj-api-1

The container stays running. Its processes, memory and open file descriptors are untouched. What it loses is the network interface for that network, so it can neither be reached nor reach out. Established connections stop working immediately.

For a container that is not cooperating — a wedged daemon can leave a stale endpoint — -f forces the disconnect.

docker inspect proj-api-1 --format \
'{{range $n, $c := .NetworkSettings.Networks}}{{$n}} {{end}}'

This is the right containment for a suspected compromise. Stopping the container destroys the process memory that a responder would want to examine, and it also tells an attacker with any monitoring that they have been noticed. Disconnecting cuts command and control while leaving the evidence sitting there.

Rung 3: freeze it

docker pause suspends every process in the container using the kernel’s freezer cgroup. Nothing runs; nothing is lost.

$ docker pause proj-worker-3 && docker ps --filter name=proj-worker-3 --format '{{.Names}} — {{.Status}}'

proj-worker-3
proj-worker-3 — Up 6 hours (Paused)

Use it for a process that is actively consuming something finite — CPU that is starving its neighbours, a queue it is draining incorrectly, rows it is deleting — and that you want to examine rather than lose. docker unpause resumes it exactly where it stopped.

Rung 4: clamp without restarting

docker update changes resource limits on a running container. No restart, no lost state.

# A container starving the host of CPU
docker update --cpus 0.5 proj-worker-3

# A leaking process, given a ceiling so the host survives the night
docker update --memory 512m --memory-swap 512m proj-worker-3

# A fork bomb, or a runaway thread pool
docker update --pids-limit 256 proj-worker-3

And the one that matters most during an incident:

docker update --restart=no proj-api-1
docker stop proj-api-1

Without that first line, restart: unless-stopped brings the container back while you are trying to capture it, RestartCount climbs, and every command you run races a new container ID. Setting the policy to no first makes the container stay stopped, keeps it present for docker logs, docker diff and docker commit, and is reversed by one more docker update when you are done.

Choosing the rung

1. Name the blast radius you are trying to stop: users reaching a broken instance, a dependency being overwhelmed, data being corrupted, the host running out of a resource, or an attacker with a live connection.
2. Pick the highest rung on the ladder that stops it. For "users are getting errors", that is rung 1 and nothing else is needed.
3. Capture before descending past rung 4, because rung 5 destroys process state and rung 6 destroys everything.
4. Say in the incident channel what you did, to which container, at what time, and that it is reversible. Half of all incident confusion comes from an unannounced containment action.
5. Write down how to reverse it, before you need to. docker update --restart=unless-stopped and docker network connect are easy to forget at hour three.

Sanity check