Runbook: Registry unavailable during deploy

1 · Prerequisites

Confirm every item is in place before any state change.

• You know the registry hostname and whether it is a hosted service or one your organisation runs
• You know whether a pull-through cache or mirror is configured for these hosts
• You have registry credentials available in case authentication needs re-testing
• You know which services are mid-deploy and which are stable on already-cached images
• You have authority to pause the deploy pipeline
• Set the variables reused below: REG=registry.example.com, MIRROR=mirror.example.com, IMG=myorg/api:2.4.0, PROJECT=/srv/app

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · curl -sS -o /dev/null -w "%{http_code}\n" https://"$REG"/v2/ returns 200 or 401 when the registry API is healthy. A 401 is healthy and merely unauthenticated; anything else, or a timeout, is the outage
• · curl -sS -o /dev/null -w "%{http_code}\n" https://"$MIRROR"/v2/ tests the pull-through cache the same way
• · Run the same curl from a host outside this network. That single comparison separates a global registry outage from a local network fault
• · dig +short "$REG" confirms the name resolves. NXDOMAIN is a DNS incident, not a registry incident, and has a different owner
• · docker info --format "{{json .RegistryConfig.Mirrors}}" prints the mirrors this daemon is configured to use. An empty list means there is no fallback to reach for
• · docker buildx imagetools inspect "$IMG" reads the manifest without pulling. The error text distinguishes unauthorized from manifest unknown from connection refused, and those are three different incidents
• · docker image ls --format "{{.Repository}}:{{.Tag}} {{.ID}}" lists what is already cached locally and can still start with no registry at all
• · docker compose -f "$PROJECT"/compose.yaml config --images lists exactly which images this deploy needs, so you can tell whether the local cache covers it
• · docker ps --format "{{.Names}} {{.Status}}" confirms running containers are unaffected. A registry outage does not stop what is already running
• · Check the registry provider status page from a network that does not depend on the registry

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Stop the bleeding first: pause the deploy pipeline. A retry loop against a dead registry turns one failed deploy into a queue of them, and can rate-limit you the moment the registry returns
2. 2Classify the fault from the pre-checks into one of four: registry down globally, unreachable from this network only, DNS failure, or authentication failure. Only the first is a registry outage, and the other three have different owners
3. 3If /v2/ returns 401 but a pull fails with unauthorized, this is credentials rather than availability. Run docker login "$REG", expect Login Succeeded, and retry the pull
4. 4Establish what is actually at risk. Running containers keep serving from their local image. The exposure is any container that restarts, any host that is rebuilt, and any scale-out event
5. 5Freeze anything that would force a pull while the outage lasts: do not run docker compose up --pull always, and do not prune images on any affected host
6. 6If a mirror is configured and healthy, use it. docker pull "$IMG" succeeds transparently, because the daemon consults its configured mirrors before the upstream registry
7. 7If the mirror is configured but has never cached the image you need, it cannot serve it. Do not wait on it - move to the next step
8. 8If the image is already cached on the host that needs it, deploy without contacting the registry: docker compose -f "$PROJECT"/compose.yaml up -d --pull never. It fails loudly if an image is genuinely absent, which is the behaviour you want
9. 9If the image exists on one host but not another, move it directly: docker save "$IMG" -o /var/tmp/img.tar on the source, copy the file, then docker load -i /var/tmp/img.tar on the target. Expect Loaded image with the tag printed
10. 10If nothing can supply the image, drain the deploy rather than half-completing it. A release rolled out to some hosts and not others is harder to reason about than a deferred one
11. 11Work the registry outage itself, or escalate it to whoever owns the registry, while the deploy stays frozen
12. 12When the registry returns, verify it before unfreezing: the /v2/ curl returns 200 or 401 on three consecutive attempts, and docker pull "$IMG" completes
13. 13Unpause the pipeline and re-run the deploy from the beginning. Do not resume a partially applied one
14. 14Close the exposure the outage revealed: add "registry-mirrors": ["https://mirror.example.com"] to /etc/docker/daemon.json and run sudo systemctl reload docker. registry-mirrors is one of the reloadable daemon options, so no restart is needed

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓curl -sS -o /dev/null -w "%{http_code}\n" https://"$REG"/v2/ returns 200 or 401 on three consecutive attempts
• ✓docker pull "$IMG" completes and prints Status: Downloaded newer image or Status: Image is up to date
• ✓docker buildx imagetools inspect "$IMG" prints a manifest whose digest matches the one the pipeline expects
• ✓docker info --format "{{json .RegistryConfig.Mirrors}}" lists the mirror and is not an empty array
• ✓A pull of an image the mirror has never seen succeeds, proving the mirror proxies upstream rather than only serving what it already holds
• ✓docker compose -f "$PROJECT"/compose.yaml ps --format "{{.Service}} {{.Status}}" shows every service running, with none left on an unintended image
• ✓docker inspect -f "{{.Config.Image}}" on each redeployed container matches the intended digest, not a stale local tag
• ✓The pipeline is unpaused and its next run completes green, rather than merely being unblocked
• ✓The registry reachability monitor fires on a synthetic /v2/ check, verified by watching a deliberately induced test alert arrive

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Nothing here changes application state, so the risk is in the workarounds rather than the fix
• ↶To undo a hand-carried docker load, remove the image with docker image rm "$IMG" once the registry can supply it, then re-pull so the local copy matches the registry digest
• ↶A deploy started with --pull never may be running an older cached layer under the expected tag. Re-run docker compose up -d --pull always once the registry is back, and confirm the digest
• ↶To undo the registry-mirrors change, remove the key from /etc/docker/daemon.json and run sudo systemctl reload docker. Running containers are unaffected
• ↶Delete any image tarball copied to /var/tmp. It is an unversioned copy of a production image sitting on a host
• ↶Release the pipeline freeze explicitly and confirm it. A forgotten freeze is always discovered at the worst possible moment

6 · Escalation

When the runbook isn't enough, contact:

• · The registry is self-managed and you do not own it: escalate to the registry owner with the /v2/ status codes from inside and outside the network and the exact pull error text
• · The registry is a hosted service reporting an incident: escalate to the incident commander, record the upstream reference, and stop retrying
• · The fault is DNS or network rather than the registry: escalate to the network team. A registry that resolves nowhere is not a registry problem
• · The fault is authentication and the same credentials work elsewhere: escalate to the identity or platform team, and treat repeated unexplained auth failures as a possible security event
• · The registry is permanently gone and the images exist only in local caches: escalate to the platform and release teams to rebuild a registry from those local copies before any host is rebuilt or pruned
• · A deploy must ship despite the outage for a security or contractual reason: escalate to the incident commander for that decision rather than making it alone
• · Hand over: the registry host, the status codes from both vantage points, whether a mirror exists and is populated, which images are cached on which hosts, and the pipeline freeze state

References

1. Run a Docker Hub mirror (pull-through cache) and registry-mirrors
2. Docker Hub repositories
3. Registry HTTP API V2: the /v2/ version check endpoint
4. docker buildx imagetools inspect
5. docker image load
6. dockerd: registry-mirrors is a reloadable daemon option