high riskservice affecting~30 min

Runbook: Roll back a Compose stack to a pinned digest

1 · Prerequisites

Confirm every item is in place before any state change.

The digest of the last known-good image is recorded, or recoverable from the registry or a previous deploy log
The Compose file is under version control and you can identify the commit that was deployed before this one
You know whether a database migration ran as part of the bad deployment, because that answer decides whether rollback is safe at all
A recent database backup exists and its restore time is known, in case the migration answer is the wrong one
Deploys are frozen for the duration, so that CI does not redeploy the bad version while you are rolling back
You can reach the registry, or the previous image is still present locally

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· cd /srv/app
· docker compose ps --format 'table {{.Service}}\t{{.Image}}\t{{.Status}}'
· docker compose images
· docker compose config --images
· SERVICE=web
· docker compose ps -q "$SERVICE" | xargs -r docker inspect -f '{{.Config.Image}} {{.Image}}'
· docker image inspect -f '{{json .RepoDigests}}' myorg/myapp:1.4.2
· docker compose config --resolve-image-digests
· git -C /srv/app log --oneline -5 -- compose.yaml

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Freeze deploys in CI before touching anything; expect the pipeline to refuse new runs. Without this, an automated job can redeploy the bad version halfway through the rollback.
2Capture the current state with docker compose ps and docker compose images redirected to a file under /var/tmp; expect a record of exactly which image each service is running right now, so the rollback itself can be undone.
3Resolve the digest currently running with docker compose ps -q "$SERVICE" then docker inspect -f '{{.Image}}'; expect a sha256 value. Record it as the forward digest.
4Determine the target digest for the last known-good release from the previous deploy record or from docker image inspect -f '{{json .RepoDigests}}' on the old image; expect a value of the form myorg/myapp@sha256 followed by 64 hex characters.
5Answer the migration question before changing anything - did this deployment run a schema migration, and is the old code able to read the new schema? Expect a definite yes or no. If the answer is no or unknown, stop and go to the escalation section rather than continuing.
6Pin the target digest in the Compose file or in a rollback override file, replacing the tag entirely; expect docker compose config --images to print the digest form rather than the tag.
7Pull the target image explicitly with docker pull followed by the full digest reference; expect the pull to succeed and report the same digest you asked for.
8Recreate only the affected service with docker compose up -d --no-deps --wait "$SERVICE"; expect Compose to stop and recreate that container and then wait for it to be running or healthy.
9Confirm the running container is the target digest with docker inspect -f '{{.Image}}' on the new container ID; expect it to equal the digest resolved in step 4 and not the forward digest from step 3.
10Verify from outside the stack that the old version is serving - a version endpoint, a build identifier in a response header, or the application log line emitted at startup; expect the previous release identifier, not the bad one.
11Watch error rate and latency for at least one full traffic cycle before unfreezing deploys, and record the forward digest in the incident so the fix-forward deployment can be prepared deliberately.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker compose ps shows every service in state running, and healthy where a health check is defined
✓docker inspect -f '{{.Image}}' on each rolled-back container prints the target digest recorded in step 4
✓docker compose config --images prints digest references and no floating tags for the rolled-back services
✓The application version endpoint returns the previous release identifier - a stack that reports the new version is still running the new image regardless of what the Compose file says
✓The error signature that triggered the rollback is absent from docker compose logs --since 15m for the affected service
✓Error rate and p99 latency have returned to the pre-deployment baseline, measured over a full traffic cycle rather than for two minutes

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶To undo the rollback, re-pin the forward digest recorded in step 3 in the Compose file and run docker compose up -d --no-deps --wait for the same service
↶If a rollback override file was used, remove it from the docker compose -f argument list and run docker compose up -d again to return to the base configuration
↶If deploys were frozen in CI, unfreeze them explicitly; a freeze left in place silently blocks the fix-forward deployment
↶Named volumes are preserved across recreation, so application data survives a rollback. Anonymous volumes are not preserved if -V or --renew-anon-volumes is used - do not use it during a rollback
↶A schema migration is not reversible by this procedure. If the rollback was attempted with an incompatible schema, the recovery path is a database restore from backup, not another docker compose up. Treat that as a separate, escalated operation

6 · Escalation

When the runbook isn't enough, contact:

· A migration ran and the old image cannot read the new schema: escalate to the data team immediately. Rolling the image back will produce errors or, worse, silent data corruption. This runbook stops here
· Whether a migration ran cannot be determined within a few minutes: escalate rather than guessing, and treat the deployment as fix-forward until the answer is known
· The registry is unreachable and the target image is not present on the host: escalate to the platform team, and check every host in the fleet for a local copy before declaring the digest unavailable
· The rolled-back version shows the same fault: escalate to the application team, because the deployment was probably not the cause and rolling further back will not help
· The stack spans more than one host and versions are now mixed: escalate to whoever owns the orchestration, since a partial rollback is its own incident

Rolling back a Compose stack is two decisions and one command. The first decision is which artefact you are rolling back to, and the answer is a digest, never a tag. The second decision is whether rollback is safe at all, and if a schema migration ran the answer may be no — in which case the correct action is not in this runbook and pretending otherwise makes the incident worse.

Symptoms

Errors began at a deployment timestamp, not gradually.
The new version is up and serving, badly.
docker compose ps shows everything running, which is why nobody looked at the deployment first.

Step 1: Freeze, then capture

Read-only / Safecapture current state— Records what is running before anything changes. No state change.

cd /srv/app

# Freeze the pipeline first, in CI. A redeploy mid-rollback is its own outage.

docker compose ps --format 'table {{.Service}}\t{{.Image}}\t{{.Status}}' \
| tee /var/tmp/rollback-before.txt
docker compose images | tee -a /var/tmp/rollback-before.txt
docker compose config --images | tee -a /var/tmp/rollback-before.txt

# The digest actually running, per service
SERVICE=web
docker compose ps -q "$SERVICE" | xargs -r docker inspect \
-f 'config={{.Config.Image}} running={{.Image}}'

That last command prints two different things and the difference is the point. .Config.Image is what was requested — often a tag. .Image is the image ID of what is actually running. When a tag has moved, these describe different artefacts, and only the second one is a fact.

Step 2: Roll back to a digest, not a tag

The tag is what moved under you

myorg/myapp:1.4.2 is a mutable pointer. Somebody can rebuild and repush it. A CI job can retag it. latest moves by design, but so does any tag anyone is willing to overwrite.

So “roll back to 1.4.2” is not a rollback — it is a request for whatever 1.4.2 happens to point at right now, which on a bad day is the same broken image you are trying to escape. Teams have rolled back three times to the same tag and watched the same fault return, and concluded the deployment was not the cause.

A digest is content-addressed and immutable. myorg/myapp@sha256: followed by 64 hex characters resolves to exactly one image forever. Docker’s own documentation puts it plainly: using a digest “pins” an image “to a specific version in time”.

Roll back to the digest. Record the digest at deploy time, every time, so that this step is a lookup rather than an investigation.

Read-only / Safefind the target digest— Reads image metadata locally and from the registry. No state change.

# If the old image is still on this host
docker image inspect -f '{{json .RepoDigests}}' myorg/myapp:1.4.2

# Ask the registry what a tag currently resolves to, without pulling
docker buildx imagetools inspect myorg/myapp:1.4.2

# What Compose would resolve every tag to, right now
docker compose config --resolve-image-digests

RepoDigests is the registry manifest digest — the identifier the registry knows. It is not the same value as the local image ID that docker inspect -f '{{.Image}}' prints, and it is the one to put in a Compose file.

docker compose config --resolve-image-digests pins every tag in the resolved configuration to its current digest. Run it as part of a normal deploy and store the output, and a future rollback becomes a file lookup instead of an archaeology exercise. There is also docker compose config --lock-image-digests, which produces an override file containing those digests.

Step 3: The migration question

Answer this before you change anything. It is the only step in this runbook that can turn an outage into data loss.

If a migration ran and is not backward compatible, do not roll back the image

A deployment that applied a schema migration has changed state that lives outside the container. Rolling the image back does not roll the schema back. The old code then meets a schema it was never written against.

What happens next depends on the migration, and none of the options are good:

A column dropped or renamed by the migration: the old code queries a column that no longer exists. Every affected request fails. This is loud, and it is the best case.
A column added with a NOT NULL constraint: the old code inserts rows without it. Writes fail. Also loud.
A type or semantic change — an enum widened, a unit changed, a value re-encoded: the old code reads and writes successfully and interprets the data incorrectly. This is silent, and it corrupts data for as long as it runs.

There is no command in this runbook that fixes the third case. If the migration is not backward compatible, the rollback is a database restore, which is a different, slower, escalated procedure with its own data-loss window.

Stop here and escalate to the data team. Do not roll the image back “to see”. The honest options are fix forward, or restore from backup and accept the loss between the backup and now.

Read-only / Safedid a migration run— Reads deployment logs and the migration table. Read-only if your migration tool's status command is.

# Did the deploy log a migration?
docker compose logs --since 6h "$SERVICE" | grep -iE 'migrat|schema|alter table'

# Most migration frameworks keep a versions table. Read it.
docker compose exec -T db \
psql -U app -d app -c 'SELECT * FROM schema_migrations ORDER BY 1 DESC LIMIT 5;'

The safe pattern, for next time, is to make migrations backward compatible by construction: add columns, never drop them in the same release; write to both old and new shapes during a transition; remove the old shape only after the release that stops reading it has been stable. Then rollback is always an image change, and this step is a formality rather than a decision.

Step 4: Pin the digest and recreate

Service impact possibleroll back one service— Recreates the service container from the target image. Brief unavailability for that service.

DIGEST=myorg/myapp@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30

# Pull explicitly, so a registry failure surfaces before the recreate
docker pull "$DIGEST"

# Option A: edit compose.yaml so the service reads
#   image: myorg/myapp@sha256:2e863c44b7187...
#
# Option B: a rollback override file, which leaves the base file untouched
cat > /srv/app/compose.rollback.yaml <<'YAML'
services:
web:
  image: myorg/myapp@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30
YAML

# Confirm what Compose will use BEFORE recreating anything
docker compose -f compose.yaml -f compose.rollback.yaml config --images

# Recreate just this service, and wait for it to be running or healthy
docker compose -f compose.yaml -f compose.rollback.yaml \
up -d --no-deps --wait --wait-timeout 120 web

Three flags are doing specific work here:

--no-deps stops Compose starting or recreating the service’s dependencies. Rolling back the web tier should not restart the database.
--wait waits for the service to be running or healthy before the command returns, and it implies detached mode. Without it the command returns as soon as the container is created, which is not the same as serving.
--wait-timeout bounds that wait so the command cannot hang through the rest of your incident.

How Compose decides to recreate

Compose does not blindly restart. Its documentation: “If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker compose up picks up the changes by stopping and recreating the containers (preserving mounted volumes).”

Changing the image reference from a tag to a digest is a change, so up -d recreates. But if the tag and the digest resolve to the same image and nothing else changed, Compose correctly does nothing — which reads as a rollback that silently did not happen. --force-recreate overrides that, and is the right flag when you need a container rebuilt from unchanged configuration.

Mounted volumes are preserved across recreation. That is what makes this safe for data. Do not add -V or --renew-anon-volumes during a rollback: it recreates anonymous volumes instead of reusing the previous container’s, which discards whatever was in them.

Step 5: Verify the rolled-back version is actually serving

This is the step people skip, and it is where silent rollback failures live. A Compose file that says one thing and a container running another is completely undetectable from docker compose ps.

Read-only / Safeprove the old version is serving— Reads container state and queries the application. No state change.

# What is running, by digest
CID=$(docker compose ps -q web)
docker inspect -f 'config={{.Config.Image}} running={{.Image}}' "$CID"
docker inspect -f 'started={{.State.StartedAt}} health={{.State.Health.Status}}' "$CID"

# The application's own claim about its version
curl -sS http://127.0.0.1:8080/version
curl -sSI http://127.0.0.1:8080/ | grep -i 'x-app-version'

# The startup line the old release emits
docker compose logs --since 10m web | head -20

# The error signature that started the incident should be absent
docker compose logs --since 15m web | grep -ciE 'ERROR|exception'

Three independent confirmations, and you want all three:

The digest — .Image on the running container equals the target digest.
The application’s self-report — a version endpoint or a response header returns the previous release identifier. If the stack still reports the new version, it is still running the new image whatever the Compose file says.
The symptom — the error signature that triggered the rollback has stopped, measured over a real traffic cycle rather than for the thirty seconds after the recreate.

Common patterns

Evidence	Diagnosis	Action
Rolled back to a tag, same fault returns	The tag moved; you redeployed the bad image	Roll back to the digest instead
`docker compose up -d` reports nothing to do	Resolved configuration is unchanged	Confirm with `config --images`; use `--force-recreate` if genuinely needed
Container recreated but version endpoint unchanged	The wrong service was rolled back, or a proxy is caching	Check `.Image` on the running container first
Errors persist after a clean rollback	The deployment was not the cause	Stop rolling back; investigate as a fresh incident
Old image not present and registry unreachable	Cannot obtain the artefact	Check other hosts for a local copy before escalating
Database errors immediately after rollback	Schema is ahead of the code	Stop. Escalate to the data team. Do not retry
Whole stack restarted when one service was intended	`--no-deps` was omitted	Use `--no-deps` and name the service explicitly
Data missing after rollback	An anonymous volume was renewed, or `down -v` was run	Restore from backup; treat as a data incident