low riskservice affecting~20 min

Runbook: Investigate an ImagePullBackOff Pod

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the Pod events: kubectl describe pod <name> -n <ns> | sed -n "/Events:/,$p"
· Capture the image the Pod is trying to pull: kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[*].image}'
· Capture the ServiceAccount and any imagePullSecrets: kubectl get pod <name> -n <ns> -o jsonpath='{.spec.serviceAccountName}{" "}{.spec.imagePullSecrets[*].name}'
· Confirm the registry is reachable from a node: crictl pull <image> on any worker
· Confirm DNS resolves the registry from a node: dig +short <registry> @<node-dns>
· Confirm the manifest pins the image to a digest or a specific tag (not latest)

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the events; the first Warning is the kubelet view of the pull failure
2Classify the failure: registry unreachable, image not found, authentication failure, manifest invalid, rate-limited, signature verification failure
3For registry unreachable: test from the node that hosts the Pod: nc -vz <registry> 443 and dig +short <registry>
4For image not found: verify the image exists with the exact tag/digest: crane manifest <image> (use the cluster-allowed tooling)
5For authentication failure: verify the imagePullSecret is mounted and valid: kubectl get secret <name> -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq
6For manifest invalid: test the pull with crictl pull on the node hosting the Pod and read the verbose error
7For rate-limited: check the registry response headers (-I on a curl) and confirm the cluster is using a pull-through cache
8For signature verification: identify the admission controller enforcing signatures (cosign, Notary, Kyverno) and verify the image is signed by an accepted identity
9Apply the smallest fix: correct the image name, replace the secret, fix the NetworkPolicy that blocks the registry, pre-pull on the node
10Delete the Pod to retry: kubectl delete pod <name> -n <ns> (the controller will recreate)
11Verify the new Pod pulls successfully: kubectl wait --for=condition=Ready pod/<name> -n <ns> --timeout=120s

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl describe pod <name> -n <ns> shows Events with Pulling and Pulled (no Back-off)
✓kubectl get pod <name> -n <ns> reports Running and Ready
✓kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].imageID}' matches the expected <registry>/<repo>@sha256:<digest>
✓kubectl logs <name> -n <ns> shows the application starting
✓No new Warning events for the Pod in the last 5 minutes
✓A second Pod with the same image on a different node also pulls successfully

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix involved modifying a Secret, revert via Git and kubectl apply
↶If the fix involved adding a NetworkPolicy exception for the registry, document the exception and review it in the next change window
↶If a node-level pre-pull was used as a workaround, that workaround must be promoted to a proper cluster-wide pull mechanism (DaemonSet, registry mirror) before rollback
↶If the original image is permanently unavailable, the fix is a new image in the manifest; the old Pod is replaced via the Deployment rollout, not by leaving a stale Pod running
↶Capture the failing pull response (HTTP status, headers, body) to the incident record before deletion

6 · Escalation

When the runbook isn't enough, contact:

· Registry is reachable from the bastion but not from the node: a NetworkPolicy or egress firewall is blocking the registry; escalate to network/platform
· Authentication consistently fails even with a fresh Secret: the registry credentials have been rotated; check the cluster secret manager or vault
· Image exists in the registry but the cluster pulls a stale manifest: pull-through cache issue; escalate to platform ownership
· Rate limit from a public registry (Docker Hub, GHCR, Quay): the cluster is using the public endpoint without authentication; switch to a pull-through cache or authenticated pulls; escalate to platform ownership
· Signature verification failure on an image that was previously accepted: signing identity rotated or admission controller misconfigured; escalate to security

ImagePullBackOff means the kubelet has tried to pull the image, failed, and is backing off. The reason for the failure is in the events. The fix is whichever smallest change unblocks the pull.

1. Read the events

Read-only / SafeRead the events


# Common event messages:
#   Failed to pull image "<image>": rpc error: ... = pull access denied
#   Failed to pull image "<image>": ... = not found
#   Failed to pull image "<image>": ... = i/o timeout
#   Failed to pull image "<image>": ... = toomanyrequests
#   Failed to pull image "<image>": ... = failed to verify signature

2. Reachability

Read-only / SafeReachability


NODE=$(kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}')
REGISTRY=$(echo <image> | cut -d/ -f1)

ssh "$NODE" -- bash -c "
dig +short $REGISTRY
nc -vz $REGISTRY 443
curl -sI https://$REGISTRY/v2/ | head
"

If dig fails, the cluster DNS is the problem (see kubernetes-rb-troubleshoot-coredns). If nc -vz fails, the egress path to the registry is blocked (NetworkPolicy or firewall — see kubernetes-rb-troubleshoot-networkpolicy). If both work, the registry is reachable.

3. Image exists

Read-only / SafeImage exists


crane manifest <image> 2>&1 | head -20
# Or:
curl -sI https://<registry>/v2/<repo>/manifests/<tag>

# If a digest is used, verify it is in the repo
curl -sI https://<registry>/v2/<repo>/manifests/sha256:<digest>

A 404 means the image or tag does not exist. A 401/403 means the cluster cannot authenticate to read the manifest. The fix differs based on which.

4. Authentication

Read-only / SafeAuthentication


kubectl get secret <pull-secret-name> -n <ns>

# Decode and inspect (the JSON is base64-encoded)
kubectl get secret <pull-secret-name> -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 -d | jq

# Test the credentials against the registry
SERVER=$(kubectl get secret <pull-secret-name> -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 -d | jq -r '.auths | to_entries[0].key')
USER=$(kubectl get secret <pull-secret-name> -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 -d | jq -r '.auths | to_entries[0].value.auth' | base64 -d | cut -d: -f1)
curl -u "$USER" -sI "https://$SERVER/v2/" | head

A 401 here means the credentials are wrong or expired. Rotate the secret in the secret manager and re-create the Kubernetes Secret.

5. Pull from the node directly

Read-only / SafePull from the node directly


NODE=$(kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- sudo crictl pull <image> 2>&1 | tail -30

# Verbose error trail
ssh "$NODE" -- sudo crictl pull -v <image> 2>&1 | tail -30

The CRI’s error message is usually more specific than the kubelet’s. toomanyrequests confirms a rate limit; unauthorized confirms a credential issue; not found confirms the image name is wrong.

6. Apply the fix

Read-only / SafeApply the fix


# A. Wrong image name
git revert <bad-commit>
git push
kubectl rollout restart deploy/<name> -n <ns>

# B. Expired secret - rotate and patch
kubectl create secret docker-registry <pull-secret-name> \
--docker-server=<registry> --docker-username=<user> --docker-password=<pw> \
-n <ns> --dry-run=client -o yaml | kubectl apply -f -

# C. NetworkPolicy blocks the registry
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-registry-egress
namespace: <ns>
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
  - ipBlock:
      cidr: 0.0.0.0/0
      except:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16
  ports:
  - protocol: TCP
    port: 443
YAML

# After the fix, delete the Pod to retry the pull
kubectl delete pod <name> -n <ns>

7. Verify

Read-only / SafeVerify


kubectl get pod -l app=<name> -n <ns> -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'
kubectl describe pod -l app=<name> -n <ns> | grep -E 'Pulled|Pulling|Back-off' | tail

Common pitfalls

Symptom	Cause	Action
`Failed to pull image` with `i/o timeout`	Egress firewall or NetworkPolicy blocks the registry	Test from the node; add an explicit egress rule
`pull access denied` with a fresh Secret	ServiceAccount does not reference the Secret	Patch the SA or use `imagePullSecrets` in the Pod spec
`toomanyrequests` from Docker Hub	Public registry rate limit	Switch to authenticated pulls or a pull-through cache
Image pulls on a new node but not on others	Older nodes have stale `/etc/containers/registries.conf`	Audit the configuration; the cluster should converge on a managed template
`failed to verify signature` after a registry migration	Signing identity rotated	Update the admission policy’s accepted identities

An ImagePullBackOff is almost always a configuration issue, not a Kubernetes issue. Fix the manifest, the secret, or the network path; do not patch around the cluster’s policy to “make it pull”.