The registry failure scenario — when pulls return 503 and the deploy falls over

An artifact registry failure is the loss of the store that sits between the workflow that pushes an image and the deployment that pulls it. The registries that production systems rely on — Harbor, ECR, GHCR, a private Distribution registry — are HTTP services that serve the OCI Distribution Spec. A failure of the service is a failure of every deploy that references it, in the same instant, even though the images themselves are still on disk inside the registry storage layer.

flowchart LR
    A["registry outage"] --> B["manifest endpoint down"]
    A --> C["blob endpoint down"]
    A --> D["auth endpoint down"]
    A --> E["write API down"]
    B --> F["pulls return 503"]
    C --> G["pulls hang on blob fetch"]
    D --> H["pulls return 401"]
    E --> I["pushes return 503"]

The DR question this lesson opens with is: what does a registry failure look like, and how do we know it is the registry and not the network?

The four scopes of failure

An OCI registry exposes four scopes, and each can fail independently. The production discipline is to identify the scope before deciding what to do.

• Manifest endpoint. The /v2/<name>/manifests/<ref> endpoint returns the manifest the client uses to enumerate the layers. A failure here produces a 503 Service Unavailable or a TLS handshake timeout on every pull. The blast radius is every deploy.
• Blob endpoint. The /v2/<name>/blobs/<digest> endpoint returns the layer bytes. A failure here allows the manifest to be fetched but stalls the blob fetch with an EOF or a reset connection. The deploy log shows a successful manifest pull followed by a failed blob GET.
• Auth endpoint. The token endpoint that issues the bearer token the registry expects. A failure here produces a 401 Unauthorized on every request, including pulls that would otherwise succeed. The blast radius is every pull from every credential.
• Write API. The push-side endpoints. A failure here breaks CI workflows that push artifacts but leaves deploys intact for as long as the artifacts already in the registry remain reachable.

The four scopes share storage but fail independently. An engineer who treats “the registry is down” as a single condition misses the distinction: a blob-endpoint-only outage allows manifests to be fetched while every layer pull fails; a write-API-only outage allows deploys to continue while CI cannot publish new artifacts.

What a 503 looks like in the deploy log

The deploy log is the first place the failure shows up. The three signatures that classify a registry failure:

# 1. Manifest endpoint — 503 Service Unavailable
docker pull registry.example.com/app:$DIGEST
# Error response from daemon: failed to resolve reference
# ...: failed to do request: Head
# "https://registry.example.com/v2/app/manifests/$DIGEST":
# 503 Service Unavailable

# 2. Blob endpoint — connection reset mid-fetch
docker pull registry.example.com/app:$TAG
# $LAYER: Pulling fs layer
# error pulling image configuration:
# ... unexpected EOF

# 3. Auth endpoint — 401
docker pull registry.example.com/app:$TAG
# Error response from daemon: Head
# "https://auth.example.com/token?...": 401 Unauthorized

Separating registry failure from network failure

A deploy that fails to pull an image can fail for one of two reasons: the registry is down, or the network path to the registry is down. The two recoveries differ — a registry recovery requires the registry operator to act; a network recovery requires the network operator to act. The production discipline is to distinguish the two before paging the wrong team.

The probe:

# Probe 1: digest pin (verifies the manifest endpoint)
DIGEST=$(crane manifest digest registry.example.com/app:$TAG)
docker pull registry.example.com/app:$DIGEST

# Probe 2: a second registry (verifies the network path)
docker pull ghcr.io/acme/canary:$TAG

# Probe 3: the registry /v2/ endpoint (verifies the registry itself)
curl -fsS https://registry.example.com/v2/

If probe 1 fails but probe 3 succeeds, the manifest endpoint is the scope of the failure. If probe 3 fails, the registry itself is unreachable. If probes 1 and 2 both fail but probe 3 succeeds, the network path to the registry is the scope, not the registry.

Production discipline

1. A registry failure is classified by the HTTP status in the deploy log before the runbook is opened. 503 from the manifest endpoint is scope one; EOF on blob fetch is scope two; 401 from the auth endpoint is scope three; push-side 503 is scope four.
2. A registry failure is separated from a network failure with a digest-pin probe and a second-registry probe. Paging the wrong team costs the team the recovery window.
3. The /v2/ endpoint is probed directly, bypassing the CDN edge. A CDN-edge 503 is a CDN problem; a registry 503 is a registry problem.

Cross-course references

• Git, CI/CD & GitOps — Part XLV-02 (Digests and Content-Addressing) covers the digest pin the probe uses.
• Git, CI/CD & GitOps — Part XCVII-05 (Recovering the Artifact Registry) covers the recovery paths this lesson’s scenario opens.
• Container Security for Production Sysadmins — Part VII (Registry Backup) covers the registry backup mechanics that the recovery relies on.

Quiz