ImagePullBackOff

Reported symptoms

The nightly ETL window opens at 22:00 and the team scales etl-loader in prod-etl from 8 replicas to 12. Four Pods never start:

NAME                          READY   STATUS             RESTARTS   AGE    NODE
etl-loader-7f4c8b9d6-2wq7x    0/1     ImagePullBackOff   0          4m     node-13
etl-loader-7f4c8b9d6-5nkzt    1/1     Running            0          9d     node-02
etl-loader-7f4c8b9d6-8mpcv    0/1     ImagePullBackOff   0          4m     node-14
etl-loader-7f4c8b9d6-b4rjw    0/1     ImagePullBackOff   0          4m     node-13
etl-loader-7f4c8b9d6-hs2dq    1/1     Running            0          9d     node-05
etl-loader-7f4c8b9d6-x9gtn    0/1     ImagePullBackOff   0          4m     node-14

The events say 401 Unauthorized, which reads as a credential problem, and the team’s first three checks all argue against that reading:

• The registry is fine. Its health endpoint is green, and two other namespaces pulled images from it in the last ten minutes.
• The credential works. An engineer pulls the exact same image by hand on node-13, using their own registry login, and it succeeds in eleven seconds.
• Nothing changed. The image tag is 2.8.4, unchanged for five weeks. The Deployment has not been touched. The four healthy Pods are running that image right now, on the same cluster, against the same registry.

There is one other data point, and nobody connects it for the first half hour. Last Friday a deploy of 2.9.0 failed on every node with the same message, was rolled back after twenty minutes, and was written up as “registry flapping during the release window.”

Evidence provided

$ kubectl describe pod etl-loader-7f4c8b9d6-2wq7x -n prod-etl | sed -n '/Events:/,$p'

Events:
Type     Reason     Age                 From      Message
----     ------     ----                ----      -------
Normal   Scheduled  4m                  default-scheduler  Successfully assigned prod-etl/etl-loader-7f4c8b9d6-2wq7x to node-13
Normal   Pulling    4m (x3 over 4m)     kubelet   Pulling image "registry.example.com/etl-loader:2.8.4"
Warning  Failed     4m (x3 over 4m)     kubelet   Failed to pull image "registry.example.com/etl-loader:2.8.4": failed to authorize: failed to fetch anonymous token: unexpected status from GET request: 401 Unauthorized
Warning  Failed     4m (x3 over 4m)     kubelet   Error: ErrImagePull
Normal   BackOff    2m (x9 over 4m)     kubelet   Back-off pulling image "registry.example.com/etl-loader:2.8.4"
Warning  Failed     2m (x9 over 4m)     kubelet   Error: ImagePullBackOff

$ kubectl get nodes --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp | tail -4

node-11   2026-05-02T09:14:22Z
node-12   2026-05-02T09:16:03Z
node-13   2026-08-10T16:41:55Z
node-14   2026-08-10T16:44:10Z

$ kubectl get deployment etl-loader -n prod-etl -o yaml | grep -E 'serviceAccountName|image:|imagePullPolicy|imagePullSecrets'

        image: registry.example.com/etl-loader:2.8.4
serviceAccountName: etl-runner

$ kubectl get sa etl-runner default -n prod-etl -o custom-columns=NAME:.metadata.name,PULLSECRETS:.imagePullSecrets[*].name

NAME         PULLSECRETS
etl-runner   <none>
default      regcred

$ kubectl get pod etl-loader-7f4c8b9d6-2wq7x -n prod-etl -o yaml | grep -c imagePullSecrets

0

$ kubectl rollout history deployment/etl-loader -n prod-etl

REVISION  CHANGE-CAUSE
11        bump etl-loader to 2.8.4
12        use dedicated serviceaccount etl-runner (rbac hardening)

# crictl images | grep etl-loader

registry.example.com/etl-loader   2.8.4   4f1b9a3c72e8   412MB

# crictl images | grep -c etl-loader

0

Work the evidence before reading on

A credential error, a working credential, and a workload that has been running happily for five weeks. Two of those three observations are about different things, and separating them is the whole exercise.

1. The engineer pulled the image successfully on node-13 using their own login. Whose credential does the kubelet use when it pulls an image for a Pod, and is it the same one? What exactly did that manual pull prove?
2. The Pod spec has no imagePullSecrets and neither does its ServiceAccount. The eight healthy Pods have the same Pod spec and the same ServiceAccount. How, then, did they ever get their image onto their nodes?
3. The tag is 2.8.4, and the Pod template sets no imagePullPolicy. What is the default policy for a tag that is not latest, and what does the kubelet do first under that policy?
4. Revision 12 is five weeks old. node-13 is one day old. Last Friday’s failed deploy was a new tag. Put those three facts on a timeline and say what each one changed.

Before continuing: the credential is valid, the registry is up, and the Pod gets a 401. Which credential is the kubelet actually offering, and where was it supposed to come from?

Root cause

1. The workload has had no pull credential for five weeks

Revision 12 moved etl-loader from the namespace’s default ServiceAccount to a dedicated etl-runner, as part of an RBAC hardening exercise. That change was correct and worth doing.

The imagePullSecrets reference, however, lived on the default ServiceAccount

• which is the common way to give a whole namespace access to a private registry without editing any workload manifest. It was not copied to etl-runner, and nothing in Kubernetes objects to a ServiceAccount without one. From revision 12 onwards, every Pod this Deployment created was created with no registry credential.

The manual pull on node-13 did not contradict this. It proved the node has a network path to the registry and that a human credential is accepted. The kubelet does not use a human’s credential; it uses the credentials referenced by the Pod, which in this case are none, so it fell back to an anonymous request and the registry refused it. That is precisely what failed to fetch anonymous token in the event message is telling you, and it is the single most under-read phrase in this class of failure.

2. The node cache hid it, and did so completely

Nothing failed for five weeks, and the reason is the default pull policy.

The image reference is etl-loader:2.8.4, an ordinary tag. When a Pod spec sets no imagePullPolicy, the default is IfNotPresent for any tag other than latest. Under IfNotPresent the kubelet asks the runtime whether the image is already on the node, and if it is, no pull happens - and therefore no credential is needed and no error is possible.

Every node in the cluster had 2.8.4 cached, because every node had pulled it back when the workload still used the default ServiceAccount and its credential. Pods rescheduled after node drains, after evictions, after scale-ups

• all of them landed on nodes that already had the layers. The broken configuration was real, continuous, and completely invisible.

That state is not stable, it is merely quiet. It ends the first time a Pod is scheduled somewhere the image is not already present. There are exactly two ways that happens: a new node, or a new tag.

3. Both ways happened, and the first one was thrown away

Last Friday’s deploy of 2.9.0 was a new tag, so no node could serve it from cache, so every Pod had to perform a real pull, so every Pod got a 401. The diagnosis was available in full, on every node, in the same event message the team is reading tonight.

It was rolled back to 2.8.4 after twenty minutes. Rolling back restored a tag that every node had cached, the symptom vanished immediately and completely, and “registry flapping” became the accepted explanation because it fit the shape of what everyone had seen. The rollback did not fix anything. It re-armed the same trap and bought four days.

Yesterday’s capacity expansion was the second way. Two nodes joined with an empty image cache, tonight’s scale-up put four Pods on them, and the same defect surfaced from the other direction.

Resolution

1. Confirm what the Pod is actually carrying, not what you expect it to carry: kubectl get pod etl-loader-7f4c8b9d6-2wq7x -n prod-etl -o yaml | grep -A3 imagePullSecrets. An empty result here is the finding; everything after this is acting on it.
2. Check the Secret is right before assuming it is: kubectl get secret regcred -n prod-etl -o jsonpath={.data} | base64 -d and confirm the auths key matches the registry host in the image reference exactly. A credential keyed for the wrong host fails identically to no credential.
3. Decide between fixing now and holding. Cordoning node-13 and node-14 puts the four replicas back onto cached nodes and restores the ETL window in minutes. That is a reasonable choice at 22:00, and it is only reasonable with an owner, a stated end time, and an explicit note that the fault is hidden rather than fixed.
4. Add the pull secret to the ServiceAccount in the repository that owns it, not with kubectl patch. A cluster-side patch is reverted by the next GitOps sync, and this workload has already survived one change that only half landed.
5. Apply it, then confirm the object: kubectl get sa etl-runner -n prod-etl -o yaml shows the imagePullSecrets entry.
6. Delete the four failing Pods. Patching the ServiceAccount does not alter existing Pods, because the reference is merged in at Pod creation - the replacements are what pick it up. Deleting Pods that are already contributing nothing costs nothing.
7. Leave the eight healthy Pods alone. They are serving, their spec is identical, and a rolling restart to make everything consistent buys no correctness and risks the window.
8. Uncordon the new nodes only after a real pull has succeeded on them, and prove it with the test in the verification section rather than by inference from the four replacements.
9. Reopen last Friday incident record with the actual cause. A rollback that removed the symptom without explaining it is an open question, and closing it here is what stops the next release from hitting the same wall.
10. Audit the rest of the hardening exercise: kubectl get sa -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PULLSECRETS:.imagePullSecrets[*].name and look for dedicated ServiceAccounts with none. One migration rarely touched one workload.

Verification

1. The four replacements are Running, and their events show Successfully pulled image rather than another BackOff. This is necessary and it is not proof.
2. A pull that cannot use the cache succeeds. Run a short-lived Pod in prod-etl with serviceAccountName: etl-runner, imagePullPolicy: Always, and nodeName: node-13. Its events must show a successful pull. This is the only check that exercises the credential rather than the cache.
3. The check can fail. Run the identical Pod with a ServiceAccount that has no pull secret and confirm it does return 401 Unauthorized. If both variants pass, the test is measuring the node cache and tells you nothing.
4. The Pod spec carries the reference. kubectl get pod -o yaml on a replacement shows imagePullSecrets in the spec, which is what the kubelet reads.
5. The fix survives a sync. Run the GitOps reconciliation, or re-apply from the repository, and re-check the ServiceAccount. A hand-patched ServiceAccount is one sync away from the state you started in.
6. A new tag deploys. Deploy 2.9.0 again - the release that was rolled back on Friday - into a canary or a single replica. Nothing was ever wrong with it, and a successful pull of an uncached tag is the strongest available evidence that the credential path works.
7. The namespace is clean. The ServiceAccount audit shows no other workload running with a dedicated ServiceAccount and no pull secret.

Prevention

• Treat pull credentials as part of the workload identity. A ServiceAccount is not finished when its RBAC is correct. Whatever checklist or module creates one should ask whether Pods using it will pull from a private registry, and fail if the answer is unanswered.
• Never let the node cache be load-bearing. IfNotPresent is the right default and it will happily run a misconfigured workload for months. Any configuration that only works because an image happens to be cached is a fault with a delayed trigger, and the trigger is pulled by capacity events you do not schedule.
• Make new nodes prove they can pull. Add a bootstrap smoke test that pulls one private image through the same credential path a workload would use, and do not mark the node schedulable until it passes. A node that has never pulled from the registry is a node whose ability to do so is an assumption, and this incident is what that assumption costs.
• Alert on image-pull failures as a cluster-wide class. ErrImagePull and ImagePullBackOff events aggregated across all namespaces turn four Pods in one corner of the cluster into a signal somebody sees, rather than something discovered by the team that happened to scale up.
• Require an explanation, not just a green board, before closing a rollback. Friday’s rollback removed the symptom and the cause survived it. A rollback that nobody can explain is an open incident with the pager switched off.
• Prefer one long-lived Secret name over new names per rotation. Rotating the contents of a stable Secret leaves every ServiceAccount and Pod reference correct; rotating the name means every reference is a place the rotation can be left half-finished - which is the same class of defect as this one, arriving from a different direction.