Impact on deployment — what stops working, what keeps working

A registry outage cascades into the deployment pipeline along four paths: image pull, Helm chart pull, signature verification, and GitOps reconciliation. Not all four paths fail at once, and not all four paths have the same criticality. The impact is a partial degradation, not an all-or-nothing outage. The production discipline is to distinguish what stops working from what keeps working, because the answer determines whether the failover decision is taken or the team waits out the outage.

flowchart LR
    A["registry outage"] --> B["image pull fails"]
    A --> C["Helm chart pull fails"]
    A --> D["signature verify fails"]
    A --> E["GitOps sync fails"]
    B --> F["new pods blocked"]
    C --> G["Helm installs fail"]
    D --> H["cosign rejects image"]
    E --> I["drift detection pauses"]
    F --> J["in-flight pods survive"]
    I --> K["source repo unaffected"]

The four paths that depend on the registry

A deployment pipeline touches the registry at four points. Each point has a different criticality, a different recovery behaviour, and a different signature in the log.

• Image pull. Every new pod that references the registry must pull the image. A failure here blocks new pods with ImagePullBackOff or ErrImagePull. The blast radius is every new pod and every restart that needs to re-pull the image.
• Helm chart pull. Helm installations that pull the chart from an OCI-based registry fail at the chart-pull step. The blast radius is every Helm release that uses an OCI chart and every helm upgrade that re-pulls.
• Signature verification. The admission controller that verifies cosign signatures against the registry’s signature store fails open (accepting unsigned images) or fails closed (rejecting every image), depending on the policy. The blast radius is the chain of trust.
• GitOps reconciliation. The Argo CD or Flux controller that reads the manifests from Git and reconciles the cluster still works — the manifests come from Git, not the registry — but a controller that references the registry to resolve image tags to digests pauses reconciliation until the registry returns.

What keeps working

A registry outage is partial, not total. The paths that keep working are the ones that do not depend on a fresh fetch from the registry.

• In-flight pods. A pod that has already pulled its image and is running continues to run. The kubelet does not re-pull an image unless the image pull policy is Always or the pod is restarted with a new generation.
• Cached layers on the node. The kubelet’s image cache holds every image the node has ever pulled. A pod that requests an image the node has pulled before resolves from the cache without touching the registry. The trade-off is that the cache is per-node; a new node that has never pulled the image has nothing to fall back on.
• Source repositories. Git is not the registry. The GitOps controller reads manifests from Git, and the source-of-truth repositories are unaffected by a registry outage. The manifests, the values files, the Kustomize overlays are all reachable through the Git path.
• Drift detection. A GitOps controller that detects drift between the manifests in Git and the cluster state can still detect drift — the diff is computed from Git, not the registry. The reconciliation that applies the fix may fail, but the detection works.

# Inspect a node's image cache
crictl images | grep registry.example.com/app

The partial-degradation model

The production discipline is to treat a registry outage as a partial degradation with tiers, not as a single binary state.

• Tier 1 — minor. New deploys on existing nodes with warm caches succeed. New nodes or restarted pods that need to re-pull fail. The deploy log shows intermittent ImagePullBackOff. No page.
• Tier 2 — significant. New deploys fail cluster-wide. In-flight pods continue. Helm releases fail at the chart-pull step. The cluster is functional but cannot roll forward. The page fires.
• Tier 3 — critical. Signature verification fails. GitOps reconciliation pauses. Cached images continue but no new artifact can be admitted. The cluster is frozen at its last good state. The failover decision is taken.

The failover trigger is tier 3, not tier 2. A tier-2 outage can be absorbed by the existing buffer (in-flight pods continue, cached images serve) for the duration of the registry RTO. A tier-3 outage cannot.

Production discipline

1. Cached images keep working; new pulls do not. A registry outage freezes new deploys, not in-flight pods.
2. The failover trigger is tier 3, not tier 2. Tier 2 is absorbed by the buffer; tier 3 requires failover.
3. Signature verification fails closed when the registry is unreachable. Failing open during an outage admits unsigned images.
4. The cluster is only as warm as its coldest node. A pre-warmed image cache is the per-node defence.

Cross-course references

• Git, CI/CD & GitOps — Part XLV-02 (Digests and Content-Addressing) covers the digest pin that the tier-3 escalation depends on.
• Git, CI/CD & GitOps — Part XCVII-05 (Recovering the Artifact Registry) covers the recovery that the tier-3 failover triggers.
• Container Security for Production Sysadmins — Part VII (Registry Backup) covers the signature-store recovery the verification depends on.

Quiz