Unavailable registries — graceful degradation and pull-through cache

Unavailable registries are a common production incident. This lesson walks the failure modes, the Pod symptoms, pull-through cache, multi-registry fallback, and the discipline.

The failure modes

flowchart LR
    A[Failure modes] --> B[Registry down]
    A --> C[Registry rate-limited]
    A --> D[Network partition]
    A --> E[DNS resolution failure]
    A --> F[Authentication failure]

The failure modes:

• Registry down. The registry server is unreachable (process crashed, server offline).
• Registry rate-limited. Docker Hub limits anonymous pulls; bursts hit the limit.
• Network partition. The cluster cannot reach the registry (firewall, route).
• DNS resolution failure. The registry hostname cannot be resolved.
• Authentication failure. The credentials are invalid or expired.

The Pod symptoms

kubectl describe pod myapp-xxx

Events:
  Type     Reason          Age                 From     Message
  ----     ------          ----                ----     -------
  Warning  Failed           5m                 kubelet  Failed to pull image
  Warning  ErrImagePull     5m                 kubelet  Error: ImagePullBackOff
  Normal   BackOff          5m                 kubelet  Back-off pulling image

The Pod symptoms:

• ErrImagePull. The kubelet failed to pull the image (network, auth, or image not found).
• ImagePullBackOff. The kubelet is backing off pulling; the Pod stays in Pending or ContainerCreating.
• CrashLoopBackOff. If the Pod started with a cached image but the image is corrupted, the container crashes.

The Pod is stuck; the workload is unavailable.

Pull-through cache

flowchart LR
    A[kubelet] -->|pull| B[Registry cache in cluster]
    B -->|cache hit| C[Use cached image]
    B -->|cache miss| D[Upstream registry]
    D -->|return image| B
    B -->|return image| A
    E[Upstream down] -->|registry cache still serves| A

A pull-through cache (distribution, Harbor) provides resilience:

• Cache hit. The kubelet pulls from the cache; no upstream call.
• Cache miss + upstream up. The cache pulls from the upstream, caches, returns to kubelet.
• Cache miss + upstream down. The kubelet fails; no cached image to fall back to.

The cache reduces load on the upstream and improves performance; it does not eliminate the dependency on the upstream for uncached images.

Multi-registry fallback

# Pod spec with multiple registries
containers:
  - name: app
    image: registry.example.com/myapp:1.2.3
    imagePullPolicy: Always

# Containerd configuration for mirror
# /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd]
  [plugins."io.containerd.grpc.v1.cri".containerd.mirrors]
    [plugins."io.containerd.grpc.v1.cri".containerd.mirrors."registry.example.com"]
      endpoint = ["https://mirror1.example.com", "https://mirror2.example.com"]

Multi-registry fallback:

• The kubelet (via containerd) tries each mirror in order.
• If the primary is down, the next mirror is tried.
• Production-grade registries use replication across multiple locations for this.

The kubelet retry

flowchart LR
    A[Pod created] --> B[Attempt pull]
    B --> C{Success?}
    C -->|Yes| D[Container starts]
    C -->|No| E[Exponential backoff]
    E --> F{Retry interval}
    F -->|after delay| B

The kubelet retry behaviour:

• After a failed pull, the kubelet waits with exponential backoff (10s, 20s, 40s, … up to 5 minutes).
• After the backoff, another attempt is made.
• The Pod stays in ContainerCreating until the pull succeeds.

Quiz

The operational discipline

Registry outages in production rest on five non-negotiable elements:

• Pull-through cache. Always.
• Multi-registry fallback. Configure mirrors.
• Test registry outage. Quarterly: simulate registry down; verify Pods start from cache.
• Monitor pull metrics. Alert on high pull failure rate.
• Document the fallback chain. In the runbook.

Registry outages are production incidents. The discipline is pull-through cache, multi-registry fallback, and testing.