Registry outage (503 from pull)

Reported symptoms

●Every new pod scheduled during the outage fails with `ImagePullBackOff`; the kubelet logs show `Failed to pull image "<image>:<tag>": rpc error: code = Unavailable desc = Error response from daemon: Head ...: 503 Service Unavailable`
●`docker pull <image>` or `crane manifest <image>` from a developer laptop returns 503 with `Retry-After` set to a few seconds
●`kubectl get pods -A` shows a wave of pods in `ImagePullBackOff` or `ErrImagePull` whose start times cluster within the outage window
●Argo CD Applications that depend on the registry show `Sync Status: OutOfSync` and `Health Status: Degraded`; the diff is a manifest that requires an image the controller cannot pull
●The CI pipeline's image-push job fails with `PUT /v2/<repo>/blobs/uploads/: 503` or `Blob upload invalid: 503`
●The registry's status page (or DNS / load balancer health check) reports a degraded control plane
●A `crane manifest --insecure` from the CI runner shows the same 503, isolating the failure to the registry, not to the runner network
●Previously-pulled images on running nodes are still serving; only newly-scheduled pods are affected

Evidence

· `kubectl describe pod <pod>` Events include `Warning Failed kubelet Failed to pull image "<image>:<tag>": rpc error: code = Unavailable desc = ... 503`
· `crane manifest <registry>/<repo>:<tag>` returns `Error: GET ... 503` with `Retry-After: 30` in headers
· `curl -I https://<registry>/v2/` returns `HTTP/2 503` with `Content-Type: application/json` and the body UNAVAILABLE
· `docker pull <image>` from a workstation returns `Error response from daemon: Head ...: 503 Service Unavailable`
· The registry's load balancer (CloudFront, Akamai, F5) shows a 5xx spike on `/v2/<repo>/manifests/<tag>` and `/v2/<repo>/blobs/uploads/`
· Argo CD Applications list `Health Status: Degraded` for every Application that depends on images from this registry; Applications using local images or alternate registries are unaffected
· The CI pipeline's retry budget was 3 attempts at `backoff: exponential`; all 3 returned 503
· `registry-probe` (a synthetic monitor) reports `Status: down` since the outage started

Diagnosis and resolutionclick to reveal

Root cause

A container registry is on the runtime path of every node: every pod scheduled during the outage window needs to pull its image, and the kubelet has no fallback for a pull that returns 5xx. Argo CD is no different — its `sync` operation may need to reference a freshly-pushed image, and the controller has no alternative source. The structural failure is the absence of a redundant pull path: the kubelet is configured with one registry endpoint, and there is no regional mirror, no cached image on the node that could be reused, and no retry budget that exceeds the outage window. The CI push path is equally single-source: the push job uploads blobs to the same registry that just went 503, with a retry budget of 3 that exhausts in seconds. The result is that every dependency on the registry freezes for the duration of the outage, and the deployment pipeline grinds to a halt because no layer of the pipeline has a fallback.

Remediation

Triage the failure layer first: DNS, network, or the registry itself. `dig <registry> +short` and `curl -I https://<registry>/v2/` from a runner host confirm whether the registry is reachable. If the registry itself is degraded, the only recovery is to wait for the provider RTO — but the in-cluster pull should failover to the regional mirror. Configure the kubelet `containerd` mirror to prefer the mirror: place a mirror block in `/etc/containerd/config.toml.tmpl` that points at the regional mirror first, the primary second. Configure CI push and GitOps sync with a retry budget that exceeds the typical outage window: at least 5 attempts with exponential backoff capped at 60s, total budget 5 minutes. For the CI push, `docker buildx --push` and `crane push` both accept retry semantics; verify the retry budget in a staging cluster by replaying the outage window. For the GitOps controller, raise `argocd-cmd-params-cm` server timeouts if necessary, and ensure the controller manifest references the mirror image pull secret. Add a synthetic monitor that pulls a known small image from the registry every 30 seconds and alerts on first failure. Promote images by digest in production manifests so a tag move during an outage cannot change what the cluster pulls.

Verification

Registry status is restored: `curl -I https://<registry>/v2/` returns `HTTP/2 200`. `crane manifest <reg>/<repo>:<tag>` returns the manifest. New pods scheduled now succeed (`kubectl get pods -l app=<name>` shows `Running` within the readiness window). Argo CD Applications return to `Health Status: Healthy` after one reconcile cycle. The CI image-push job completes. The retry budget of 5 attempts with exponential backoff was sufficient to ride out the outage — verify this by replaying the outage window in a staging cluster with the same backoff settings. The regional mirror is reachable from each cluster region and the kubelet `containerd` config points at it.

Prevention

A registry is a single point of failure for every image pull. The structural fix is replication: deploy a regional mirror of the primary registry (Harbor, ECR with cross-region replication, GCR with dual-region buckets, Quay with geo-mirroring), and configure kubelet to prefer the mirror via a per-cluster image pull policy or a `containerd` mirror config. Configure CI and the GitOps controller to retry on 5xx with `crane retry` semantics: at least 5 attempts with exponential backoff capped at 60s, total budget 5 minutes. Stand up a synthetic monitor that pulls a known image every 30 seconds from each region; alert on first failure. Track RTO and RPO for the registry itself: the registry should be on the same DR tier as the cluster control plane (RTO minutes, RPO seconds), and the drill should include pulling an image while the primary is degraded. For long-term immutability, promote by digest: the Application manifest references `image: <reg>/<repo>@sha256:<digest>`, not `:tag`, so a tag move during the outage cannot change what the cluster pulls. The principle is that image pull is a runtime dependency of every node, and a runtime dependency without a redundant path is an outage waiting to be scheduled.

A registry is on the runtime path of every node and every deploy. Treat it like a control-plane dependency: replicate it, monitor it, pin images by digest so a tag move cannot change what the cluster pulls, and drill the failure. A 503 from pull is a deployment freeze, not an inconvenience, and the only fix is a redundant path the kubelet can use without operator action.