Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

advancedRegistry~20 min

Break/Fix 13: Registry unavailable during deploy

Reported symptoms

  • ●`docker compose up` fails with `image not found` or `pull access denied`.
  • ●Rolling deploy stalls at the first new replica.
  • ●Old replicas continue serving until they crash or restart.

Evidence

  • · `docker pull registry.example.com/myorg/app:1.0.0` returns an error.
  • · The registry service is unreachable from the deploy host.
  • · A monitoring alert for the registry fires.
Diagnosis and resolutionclick to reveal

Root cause

The registry is unreachable from the deploy host. Causes include: registry host down, network partition, DNS failure, expired registry TLS certificate, expired credentials.

Remediation

(1) Identify the registry failure mode. (2) If the registry is permanently lost, fall back to a local cache. (3) If temporarily unavailable, pause the rollout; existing replicas continue serving their cached images. (4) Restore the registry before continuing the deploy.

Verification

The new deploy completes successfully once the registry is reachable. No service downtime occurred.

Prevention

Mirror critical images to a second registry or local cache. Alert on registry reachability. Test registry failover quarterly.

The “registry outage” decision tree

  1. Confirm scope. Is the registry service down, or just this host’s network? Check from a second host.
  2. Identify the cause. Service down, network partition, DNS, TLS, credentials.
  3. Pause the deploy. Do not continue to create new replicas that cannot start.
  4. Check existing replicas. They are running with cached images; they keep serving until they crash or restart.
  5. Restore the registry. The fix depends on the cause: failover DNS, restore service, refresh credentials.
  6. Resume the deploy. Once the registry is reachable, complete the rollout.

Local cache as a fallback

For critical images, mirror to a local registry or pull-through cache (e.g. registry:2 configured as a pull-through cache against Docker Hub):

# /etc/docker/registry/config.yml
proxy:
  remoteurl: https://registry-1.docker.io

Each host pulls from the local cache. If Docker Hub is down, the cache serves from its local copy.