Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

advancedkubernetes-registry~30 min

Registry outage

Reported symptoms

  • ●A node was lost to a hardware fault at 02:07; the Pods it held were rescheduled, and roughly a third of them never started
  • ●Which workloads are stuck looks arbitrary - the payments API came back in seconds, the checkout API did not, and neither did the ingress controller
  • ●The failing Pods do not agree on an error: some report a 429 from the registry, some report ImagePullBackOff with no detail, and some have sat in ContainerCreating for eight minutes saying nothing
  • ●The internal registry is healthy and every engineer on the call can pull the failing images by hand from their own laptop
  • ●Nothing was deployed, no manifest changed, and no registry maintenance was scheduled
  • ●The first few retries appeared to work, so the incident was called transient and the on-call deleted the stuck Pods to force a retry, after which more of them failed

Evidence

  • · kubectl describe pod on a stuck Pod shows Failed to pull image with a 429 Too Many Requests and a pull rate limit message
  • · Every failing image reference begins with docker.io; every image that started cleanly comes from the internal registry
  • · kubectl get pods -A with a go-template over container images shows how many distinct docker.io references the cluster actually runs
  • · curl against the internal registry v2 endpoint from inside a Pod succeeds, so cluster egress and DNS are working
  • · The containerd registry configuration on the nodes has a mirror entry for the internal registry host and none for docker.io
  • · The surviving nodes are running the same images happily, and none of them has pulled anything since the images were first cached
  • · The cluster egresses through a single NAT address, so every node presents the same source address to an external registry
Diagnosis and resolutionclick to reveal

Root cause

Nothing was wrong with any registry. The cluster depends on Docker Hub for a set of third-party images - the ingress controller, a cache, several sidecars - and it pulls them anonymously, directly, with no mirror entry for docker.io in the node registry configuration. In steady state that dependency is invisible, because a running Pod never pulls: the images are in each node local image store and nothing goes near the network. Losing a node converted the whole invisible dependency into a burst. Every rescheduled Pod that had no cached copy of its image on its new host needed a pull, all of them within a minute or two of each other, and every one of those pulls presented the same source address because the cluster egresses through one NAT gateway. A shared anonymous quota that a trickle never touches is exhausted immediately by a crowd. From there the symptom set writes itself. The workloads that came back in seconds are the ones whose images come from the internal registry, which was never involved. The Pods with no error yet are at an earlier stage of the same failure, because the kubelet pull timeout is long and its retries are quiet. And the pull that works from a laptop works because a laptop is a different source address with its own quota, which makes it a test of the image rather than a test of the cluster. The retry storm that followed was not a contributing factor added later; it is the same cause, fed.

Remediation

The first action is to stop causing pulls. Deleting a stuck Pod restarts the kubelet backoff from the beginning and spends more of a quota that is already exhausted, so the well-meant remedy is the thing making recovery slower. Say out loud on the call that nobody drains, restarts or force-deletes anything until the pull path is fixed, and mean it - the kubelet exponential backoff is correct behaviour and needs no help. Then give the pulls an identity or a local source, in that order of speed. An imagePullSecret on the affected namespaces makes the pulls authenticated so they are attributed to an account rather than to a shared address; what the limits then are is the provider business and they change, so read their current terms rather than a number someone wrote in a runbook last year. The durable fix is that no recovery path should reach outside the estate at all: add a docker.io mirror pointing at the in-cluster pull-through cache to every node pool registry configuration, and host the third-party images the cluster genuinely cannot start without in the internal registry under your own names. Holding is a real option for part of this. Workloads still running on surviving nodes with cached images are not at risk unless something makes them restart, so the safest posture while the fix is prepared is deliberate stillness, with a named owner and a stated end time, rather than activity that looks like progress.

Verification

A pull that succeeds is not evidence that the fix works, because a pull can succeed for the wrong reason - a cached layer, a quota window that has rolled over, a laptop. Verify on a node, with the image removed from that node local store first, and confirm from the mirror own access log that the mirror served it. Then verify the property rather than the instance: block egress to docker.io on one node, start a Pod there that depends on an external image, and require it to run. That is a short exercise and it is the only check that actually tests the dependency this incident was about. Inventory every image reference in the cluster afterwards and require every one to resolve through the internal registry or the mirror; a single remaining direct reference is the next incident. Confirm the previously stuck Pods are Running rather than merely no longer backing off, confirm the replacement node is carrying its share of the workload, and confirm the mirror configuration is present on every node pool including any pool created since the change.

Prevention

Treat every external image as a production dependency with an owner, and keep the inventory somewhere a human reads rather than deriving it during an incident. Mirror or re-host everything the cluster needs in order to recover, because recovery is precisely the moment an external dependency cannot be afforded - a registry you can live without at 15:00 on a Tuesday is one you cannot live without while a node pool is being replaced. Put the mirror configuration in the node image or machine configuration rather than applying it to nodes afterwards, so a new pool cannot come up without it; a mirror that exists on the old nodes and not the new ones is the same outage with a different trigger. Test it deliberately: block the registry on one node and confirm Pods still start. That exercise almost always finds something, usually an imagePullPolicy of Always on something critical or a pool with a cold cache. Alert on image pull failures as their own signal rather than letting them arrive as Pods not becoming ready, because the two have very different fixes and only one of them is urgent at 02:00. And write down that force-deleting a Pod during a pull failure makes it worse, because somebody will reach for it again.

Reported symptoms

A worker node failed at 02:07 with what the hardware team later called a backplane fault. That part went to plan: the node went NotReady, the node controller marked its Pods for deletion, and the schedulers placed them elsewhere. The cluster is built for this and has done it before.

By 02:20 roughly a third of the rescheduled Pods had still not started, and the picture on the call was confusing.

The failures look unrelated to each other. The payments API came back inside ten seconds. The checkout API did not. The ingress controller did not. A Redis sidecar in three different namespaces did not, and the message broker did not. There is no pattern by team, by namespace, by priority class or by node.

The Pods do not agree on what is wrong. Some report a 429 from a registry. Some sit in ImagePullBackOff with a message that says only that the back-off is happening. Several have been in ContainerCreating for eight minutes and have produced no event at all.

The registry is fine. The internal registry answers its /v2/ endpoint. Two engineers pull the failing images by hand from their laptops while the call is running. Both succeed, immediately, every time.

Nothing changed. No release went out. No manifest was edited. No registry maintenance was scheduled. The only event of the night was a node dying.

And there is one more thing, offered apologetically at 02:35: the first two Pods the on-call force-deleted came back fine, so the problem was called transient and the rest were force-deleted too. More of them failed after that than before.

Evidence collected

Read-only / Safea rate limit, not an outage
$ kubectl -n shop describe pod checkout-7d9f4c8b6-x2k9p | tail -8
Events:
Type     Reason   Age                 From     Message
----     ------   ----                ----     -------
Normal   Pulling  9m                  kubelet  Pulling image "docker.io/library/redis:7.2"
Warning  Failed   9m                  kubelet  Failed to pull image "docker.io/library/redis:7.2": 429 Too Many Requests - You have reached your pull rate limit
Warning  Failed   9m                  kubelet  Error: ErrImagePull
Normal   BackOff  4m (x12 over 9m)    kubelet  Back-off pulling image "docker.io/library/redis:7.2"

Illustrative output

Read-only / Safe37 containers in the estate pull from Docker Hub
$ kubectl get pods -A -o go-template='{{range .items}}{{range .spec.containers}}{{println .image}}{{end}}{{end}}' | cut -d/ -f1 | sort | uniq -c | sort -rn
    412 registry.example.com
   37 docker.io
    6 quay.io

Illustrative output

Read-only / Safethe replacement node: three stalled, one running, same node and same minute
$ kubectl get pods -A --field-selector=spec.nodeName=worker-11
NAMESPACE   NAME                                   READY   STATUS              RESTARTS   AGE
billing     billing-cache-0                        0/1     ImagePullBackOff    0          9m
ingress     traefik-6b8f9c47d5-qz4mn               0/1     ImagePullBackOff    0          9m
search      search-api-5c4d7f9b8-m8vqx             2/2     Running             0          9m
shop        checkout-7d9f4c8b6-x2k9p               1/2     ContainerCreating   0          9m

Illustrative output

Read-only / Safethe same four Pods, and the images each one is trying to pull
$ kubectl get pods -A --field-selector=spec.nodeName=worker-11 -o custom-columns='NAME:.metadata.name,IMAGES:.spec.containers[*].image'
NAME                                   IMAGES
billing-cache-0                        docker.io/library/redis:7.2
traefik-6b8f9c47d5-qz4mn               docker.io/library/traefik:v3.1
search-api-5c4d7f9b8-m8vqx             registry.example.com/search/api:4.8.1
checkout-7d9f4c8b6-x2k9p               registry.example.com/shop/checkout:2.14.0,docker.io/library/redis:7.2

Illustrative output

Read-only / Safeegress and DNS work; the internal registry is healthy from inside the cluster
$ kubectl -n shop run probe --rm -it --restart=Never --image=registry.example.com/tools/curl:1 -- curl -sS -o /dev/null -w '%{http_code}' https://registry.example.com/v2/
200

Illustrative output

Read-only / Safeone mirror entry, and it is not for docker.io
# grep -A2 'registry.mirrors' /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.example.com"]
endpoint = ["https://harbor.internal.example.com/proxy-internal"]

Illustrative output

Work the evidence before reading on

Three Pods on the replacement node stalled and one on that same node came up in seconds. That is the most useful pair of facts on the page.

  1. List the image references of the Pods that failed and the Pods that succeeded, side by side. Ignore everything else about them - the team, the namespace, the priority - and look only at the registry host.
  2. The surviving nodes are running these same images right now, quite happily. What is different about the node that cannot start them?
  3. Two engineers pulled the failing image successfully from their laptops. What does a laptop have that a cluster node does not, from the point of view of the thing counting requests?
  4. Read the containerd mirror configuration again. Which registry host is covered, and which one is not?
  5. The force-deletes made it worse. If the constraint were an outage, would retrying cost anything?

Before continuing: nothing was deployed and no registry went down. What changed at 02:07 that made a dependency visible which had been invisible for months?

Root cause

1. In steady state, nothing pulls

A running Pod does not talk to a registry. The image is unpacked in the node local content store, and every restart, every probe failure, every container crash-loop reuses what is already there. Whether the cluster can reach a registry at all is a question that simply does not come up while the cluster is healthy.

That is why 37 containers pulling from Docker Hub had never been noticed. The dependency is real and continuously present in the manifests, and it is exercised approximately never.

2. A node failure converts the whole dependency into one burst

At 02:07 every Pod on the dead node needed a new home, and a new home means a host that has probably never run that image. The kubelet on the replacement node pulled for all of them at once, and so did the kubelets on the other nodes that picked up the overflow.

Months of accumulated, unexercised external dependency arrived as a single burst of requests inside about ninety seconds.

3. Every one of those requests came from the same place

The cluster egresses through a single NAT address. To an external registry counting anonymous requests, the whole estate is one client. A quota that a trickle never approaches is exhausted immediately by a crowd, and once exhausted it stays exhausted for as long as the window lasts - which is exactly the window in which the cluster is trying to recover.

This is the piece that makes the laptop test misleading rather than merely useless. The laptop is a different source address with its own budget. Pulling the image successfully from it proves the image exists and is readable. It proves nothing whatsoever about the cluster, and it was the single most confidently offered piece of evidence on the call.

4. The scattered symptoms are one cause seen at different stages

Once you sort the Pods by registry host, the arbitrariness disappears. Everything from registry.example.com started. Everything from docker.io did not. search-api was running on the same replacement node throughout, because its image comes from the internal registry, and checkout sits at 1/2 because one of its two containers comes from each.

The Pods with no events yet are not a different problem. The kubelet pull timeout is generous and its retries are quiet, so a Pod can sit in ContainerCreating for many minutes before anything is written down. ImagePullBackOff is a later stage of the same story, not a worse one.

Resolution

  1. Stop causing pulls, explicitly and out loud. No force-deletes, no drains, no rollouts, no node replacements until the pull path is fixed. Name the owner of that hold and the time it will be reviewed so that stillness reads as a decision rather than as inaction.
  2. Sort the failing Pods by registry host. This takes one command and it converts an arbitrary-looking incident into a bounded list of affected workloads: everything from the external host, nothing else.
  3. Confirm the constraint is a quota rather than an outage by reading the actual error rather than the Pod status. A 429 is a refusal to serve you specifically; a timeout or a connection failure would point somewhere entirely different and would change the whole plan.
  4. Make the pulls authenticated as the fastest available relief. An imagePullSecret on the affected namespaces attributes the requests to an account instead of to a shared egress address. Check the provider current terms for what that actually buys rather than trusting a number in an old runbook.
  5. Let the backed-off Pods recover on their own. They will retry on their existing schedule and start succeeding; they need no intervention, and intervening is what caused the second wave.
  6. Add a mirror entry for the external registry host to the containerd registry configuration, pointing at the in-cluster pull-through cache, and roll it one node at a time. Verify on the first node with crictl pull before touching the second.
  7. Re-host the third-party images the cluster cannot start without in the internal registry, under your own names, and update the manifests. This is the change that makes the next node failure uneventful rather than survivable.
  8. Only once pulls are working again, bring the replacement node fully back into service and confirm it is carrying a normal share of the workload.

Verification

  1. The previously stuck Pods are Running, not merely no longer backing off. Check readiness and check that their containers have not restarted since starting.
  2. A pull is served by the mirror, and you have proof of it rather than an inference. Remove the image from one node local store, pull it again with crictl pull, and find the request in the mirror own access log.
  3. The dependency itself is gone, which is a stronger claim than the pull working. Block egress to the external registry host on one node, start a Pod there that references an external image, and require it to run.
  4. No direct external reference remains. Inventory every image in the cluster by registry host and require each one to resolve through the internal registry or through the mirror. One survivor is the next incident.
  5. The mirror configuration is present on every node pool, including any pool created after the change. A configuration applied to existing nodes by hand is a configuration the next pool will not have.
  6. Nothing is pinned to a pull policy that defeats the cache. Find workloads with imagePullPolicy: Always on external images and decide, deliberately, whether each one is worth the loss of cache fallback.
  7. The alert exists and fires. Cause a pull failure on a scratch workload and confirm an image-pull alert arrives as its own signal, distinct from a Pod-not-ready alert.
  8. The runbook records that force-deleting Pods during a pull failure prolongs it. This is the part of the incident that will be repeated by somebody who was not on the call.

Prevention

  • Inventory external image dependencies and give each one an owner. The list took one command to produce during the incident and would have taken the same command a year earlier, when there was time to act on it.
  • Mirror or re-host everything needed to recover. Recovery is exactly when an external dependency cannot be afforded, and it is the only time it will be exercised at scale.
  • Put the mirror configuration in node provisioning, not on the nodes afterwards. A mirror that exists on the old pool and not the new one is this same outage with a different trigger, and it will look equally mysterious.
  • Test the registry dependency deliberately. Block it on one node and confirm Pods still start. The exercise is short and it reliably finds something.
  • Alert on image pull failures as a distinct signal. Arriving as “Pods not ready” mixes them with a dozen unrelated causes at the moment when the distinction matters most.
  • Know what imagePullPolicy is doing. Always on an external image means no cache fallback and a pull on every single start, including every start during an incident.
  • Write down that force-deleting a stuck Pod resets the backoff. It is the most natural action available and it is the wrong one, so it needs to be somewhere other than in the memory of the people who were awake.