Recovery — restart, roll back, restore, or rebuild

The damage is contained. Traffic is off the broken instance, the evidence is in /var/tmp, and the channel knows what is happening. Now somebody has to decide how to get back to normal, and there are only four moves.

Choosing between them badly is how a thirty-minute incident becomes a six-hour one. Choosing quickly and reversibly is most of what separates practised teams from unpractised ones.

The four moves

Answer the data question first

Before choosing, answer one question: has data been lost or corrupted?

It comes first because restore is the only move that addresses it, it is the slowest, and every other move can quietly make it worse. A restart against a corrupted volume writes more corruption. A rollback to an older application version against a partially migrated schema can delete columns it does not know about.

# Is the volume still there and does it have the shape you expect?
docker volume inspect proj_pgdata
docker run --rm -v proj_pgdata:/d:ro alpine:3.20 sh -c 'ls -la /d | head -20; du -sh /d'

# Did the application report integrity errors before it went down?
docker logs proj-db-1 --since 2h 2>&1 | grep -iE 'corrupt|checksum|fatal|panic' | tail -20

If the answer is yes, or “we cannot tell”, stop and involve whoever owns the data. Recovery is now a restore, and the sequencing matters far more than the speed.

Roll back if the incident started at a deploy

The single most useful correlation in incident response: did this begin within a few minutes of a change?

docker events \
--since '2026-08-11T08:00:00Z' \
--until '2026-08-11T10:30:00Z' \
--filter 'event=create' --filter 'event=start' \
--format '{{.Time}} {{.Action}} {{.Actor.Attributes.image}}'

If a new image started at 09:48 and the error rate rose at 09:52, you do not need to understand the bug to act. Roll back. It is the fastest available test of the strongest available hypothesis, and if it works you have both resolved the incident and confirmed the cause.

Roll back by digest, not by tag:

$ docker inspect proj-api-1 --format '{{.Config.Image}} {{.Image}}'

myapp:1.5.0 sha256:9f2b4c8e1a37d5b60c9e8f14a2d7b3c5e60f81a94d2c7b3e5f81a94d2c7b3e5f

# The digest of the release you are rolling back to
docker image inspect myapp:1.4.2 --format '{{index .RepoDigests 0}}'

# Deploy that exact content
APP_IMAGE='myapp@sha256:REPLACE_WITH_DIGEST_FROM_ABOVE' \
docker compose up -d --wait --wait-timeout 120

# Confirm you got what you asked for
docker inspect proj-api-1 --format '{{.Image}}'

The reason for the digest is specific and it has caused real incidents: tags are mutable. If your pipeline overwrites 1.4.2, or if latest moved, “rolling back to 1.4.2” can deploy the same broken content you are rolling back from — and you will spend the next twenty minutes concluding that the rollback did not help and the deploy was not the cause.

Restore is a sequence, not a command

1. Stop writes to the damaged data before anything else, so the amount you have to recover stops growing.
2. Identify the most recent backup that predates the corruption. The newest backup is not automatically the right one.
3. Restore into a new volume, never over the live one.
4. Verify the restored data before cutting over: row counts, the most recent record, and one real application query.
5. Cut over, and record exactly how much data was lost in wall-clock terms. That number belongs in the postmortem and it is the number leadership will ask for.
6. Keep the damaged volume until the review is complete.

Rebuild when the host is suspect

If the incident involved a compromise, or host-level state you cannot account for, do not clean the host. Rebuild it.

The argument is asymmetric. Cleaning succeeds only if you found everything, and you cannot prove that. Rebuilding succeeds if your host is reproducible — which is a property you either have or need to acquire, and either way the incident has told you something important.

This is also the moment when “we can rebuild this host from configuration in about twenty minutes” stops being a nice-to-have. If the honest answer is “nobody knows how this host was built”, that is the top action item of the review, ahead of whatever caused the incident.

“Resolved” is not “the container is up”

Resolution needs all four of these, not one:

1. A real user transaction succeeds, executed from outside your infrastructure against the public endpoint — not a healthcheck.
2. The error rate has been at baseline for a sustained period. Ten minutes is a common bar; one green scrape is not.
3. Every backlog is drained: queues, retry buffers, dead-letter queues, and any batch that did not run during the outage.
4. Every containment action has been reversed and recorded — the load balancer entry restored, docker update --restart=no undone, the disconnected network reconnected, the paused container unpaused or explicitly retired.

docker inspect proj-api-1 --format \
'restart={{.HostConfig.RestartPolicy.Name}} status={{.State.Status}} health={{.State.Health.Status}} nets={{range $n, $c := .NetworkSettings.Networks}}{{$n}} {{end}}'

docker ps --filter status=paused

That second command has saved more than one team from discovering a paused container three weeks later.

Sanity check