Runbook: Bad image deployment — roll back to the previous version

1 · Prerequisites

Confirm every item is in place before any state change.

• You have the previous image reference, ideally as an immutable digest rather than a mutable tag
• The previous image is still in the registry and has not been removed by a retention policy
• You have registry pull credentials on the deploy host: docker login registry.example.com printed Login Succeeded
• You know whether the new version applied a database migration. If it did, read the escalation section before doing anything else
• You have authority to roll production back, and an incident channel is open
• Set the variables reused below: SERVICE=api, BAD=myorg/api:2.4.0, GOOD=myorg/api@sha256:REPLACE_ME

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · docker compose ps --format "{{.Service}} {{.Image}} {{.Status}}" prints what is actually deployed right now. Trust this over the pipeline dashboard
• · docker inspect -f "{{.Config.Image}}" "$SERVICE" prints the reference the running container was started from
• · docker image inspect --format "{{index .RepoDigests 0}}" "$BAD" prints the digest currently deployed. Record it in the incident timeline before you change anything
• · docker buildx imagetools inspect "$GOOD" reads the rollback target manifest straight from the registry without pulling it. If this errors, the target no longer exists and this runbook cannot proceed
• · docker compose config --images lists exactly which images this compose file resolves to
• · docker events --since 60m --filter type=container --filter container="$SERVICE" prints the start and die events, giving the exact deploy timestamp
• · docker logs --since 30m --tail 200 "$SERVICE" shows the errors, with their first occurrence
• · Compare the first error timestamp against the container start timestamp. If the errors predate the deploy, stop: this is not a bad deployment and this is the wrong runbook
• · docker inspect -f "{{.RestartCount}}" "$SERVICE" prints the restart count, which distinguishes a crash loop from a running-but-wrong deploy

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Confirm causality first. The first error in the logs must postdate the container start event from the pre-checks. If it does not, stop - you are about to roll back an innocent change and lose time
2. 2Freeze further rollout. Stop the pipeline job or set the deploy lock, so no automation re-applies the bad tag while you work. A rollback that is immediately overwritten is worse than none
3. 3Pull the rollback target before you stop anything: docker pull "$GOOD" prints Status: Downloaded newer image or Status: Image is up to date. If it fails, do not proceed - go to escalation
4. 4Verify you pulled what you intended: docker image inspect --format "{{index .RepoDigests 0}}" "$GOOD" prints a digest, and it must differ from the digest you recorded for the bad build
5. 5Pin the compose file to the digest rather than the tag. Replace the image value for the service with the full myorg/api@sha256:... reference, so a later re-pull cannot silently resolve to a different build
6. 6Confirm the edit resolves as intended: docker compose config --images prints the digest you just pasted, not the old tag
7. 7Deploy only the affected service: docker compose up -d --no-deps --wait "$SERVICE". --no-deps leaves healthy dependencies untouched, and --wait blocks until the container is running or healthy and exits non-zero if it is not
8. 8Confirm the running container is the rollback target: docker inspect -f "{{.Config.Image}}" "$SERVICE" prints the digest reference, not the bad tag
9. 9Watch for the original symptom rather than assuming it is gone: docker logs -f --since 1m "$SERVICE" no longer produces the error string from the incident
10. 10Run the synthetic probe that first alerted, from outside the host rather than from localhost, and confirm it passes three times in a row
11. 11Record in the incident timeline: the bad digest, the good digest, the rollback timestamp, and who authorised it
12. 12Quarantine the bad build so nobody redeploys it. Remove or re-point the moving tag in the registry, then release the deploy lock and open the postmortem ticket

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓docker inspect -f "{{.Config.Image}}" "$SERVICE" prints the rollback digest and does not contain the bad tag
• ✓docker compose ps --format "{{.Service}} {{.Status}}" shows the service as running, with no other service left in a partial state
• ✓docker inspect -f "{{.State.Health.Status}}" "$SERVICE" prints healthy. If it prints starting for longer than the configured start_period, the rollback has not converged
• ✓docker inspect -f "{{.RestartCount}}" "$SERVICE" prints 0 and stays at 0 across five minutes of observation
• ✓The exact error string from the incident is absent from docker logs --since 10m "$SERVICE"
• ✓The synthetic probe that opened the incident returns 200 on three consecutive runs from outside the host
• ✓Error rate and latency return to the pre-deploy baseline, not merely below the alert threshold
• ✓docker compose config --images shows the digest, so the next deploy from this file cannot resolve back to the bad build

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶This runbook is itself a rollback. To undo it and return to the newer image, re-pin the compose file to the bad digest and run docker compose up -d --no-deps --wait "$SERVICE". Do this only on an explicit decision from the incident commander
• ↶Both images remain local until something prunes them: docker image ls confirms the old and new digests are still present, so either direction is one command away
• ↶If you edited the compose file in place without version control, restore it from the git checkout before anything else. An untracked production edit is the usual cause of the next incident
• ↶A database migration applied by the new version is not undone by rolling the image back. Running the old image against the migrated schema can corrupt data. Stop and escalate instead
• ↶If the rollback itself caused a new failure, capture logs before reverting again, or you will have two undiagnosed incidents rather than one
• ↶Release the deploy lock set in step 2 once the incident closes, or the next legitimate deploy will fail for reasons nobody remembers

6 · Escalation

When the runbook isn't enough, contact:

• · The previous image is not in the registry - buildx imagetools inspect errors, or docker pull returns manifest unknown. You cannot roll back. Escalate to the release team to rebuild from the previous commit, and to the incident commander to consider a feature flag or traffic shift instead
• · The new version applied a schema migration: escalate to the data team before rolling back. A backward-incompatible migration turns an image rollback into a data-loss event
• · The rollback would reintroduce a known CVE or reverse a security fix: escalate to the security team for a recorded accept-the-risk decision with a deadline
• · The registry is unreachable rather than missing the image: switch to the registry-unavailable runbook, which covers deploying from the pull-through cache
• · The rollback completed and the symptom persists: the deploy was not the cause. Escalate to the on-call for the dependent service and reopen diagnosis rather than rolling back further
• · Hand over: the bad digest, the good digest, the deploy timestamp, the first-error timestamp, the compose diff, and the current state of the deploy lock

References

1. docker image tag - tagging reference
2. docker compose up: --no-deps, --wait and --pull
3. docker buildx imagetools inspect - read a registry manifest without pulling
4. Compose file reference: services, restart and healthcheck
5. docker inspect