Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

intermediatekubernetes-kubelet~35 min

kubelet cannot start pods

Reported symptoms

  • ●Every Pod newly scheduled to worker-04 sits in ContainerCreating and never leaves it - no crash, no ImagePullBackOff, restart count zero
  • ●Pods that were already running on worker-04 before Tuesday are healthy and serving traffic
  • ●The node has been continuously Ready and reports ample allocatable CPU and memory
  • ●The stuck Pods have no IP in `kubectl get pods -o wide`, which sends the incident to the network team
  • ●`kubectl logs` against a stuck Pod returns an error rather than output, so the application team concludes the Pod does not exist yet
  • ●containerd is running on the node and `crictl ps` answers instantly, so the platform team concludes the runtime is fine
  • ●Three other nodes were patched in the same maintenance window and none of them behave like this

Evidence

  • · `kubectl describe pod` shows a `FailedCreatePodSandbox` warning, and the image named in it is not the application image
  • · The application image is already present on the node and pulls fine when tested by hand
  • · `crictl images` on worker-04 lists an older sandbox image but nothing matching the tag in the error
  • · Pulling the image named in the error from the node times out; pulling the application image from the internal registry succeeds
  • · `grep sandbox_image /etc/containerd/config.toml` returns a different value on worker-04 than on a healthy peer node
  • · The containerd package was upgraded on worker-04 during the maintenance window and the package manager left a saved copy of the previous configuration file beside the new one
  • · `journalctl -u kubelet` on the node repeats the same RunPodSandbox failure for each stuck Pod, once per sync attempt
Diagnosis and resolutionclick to reveal

Root cause

The runtime upgrade replaced `/etc/containerd/config.toml` with the package default and discarded the node's local override of `sandbox_image`, so containerd now tries to fetch the Pod sandbox image from an upstream registry the node is not permitted to reach instead of from the internal mirror. Every Pod on a node is built around that sandbox: the kubelet's first CRI call for a new Pod is RunPodSandbox, and the runtime cannot create a sandbox without that image. The failure therefore lands before any application container is considered - no application image is pulled, no container is created, no log stream exists, and the Pod has no network namespace and so no IP. That is why every diagnostic aimed at the workload comes back clean and why the evidence points at three different teams. Pods already running are untouched because their sandboxes were created before the upgrade and are not recreated. The node stays Ready throughout because the kubelet's status loop runs independently of its Pod sync loop: a kubelet that cannot start a single Pod will still cheerfully report that the node is healthy, and the scheduler will keep sending it work.

Remediation

Cordon the node first. It is advertising capacity it cannot use, and every minute it stays schedulable it collects more Pods that will not start; that single command stops the bleeding and costs nothing. Then restore the local value of `sandbox_image` in the runtime configuration - the package manager saved the previous file beside the new one, so the correct value is on the node rather than in somebody's memory - and diff the two files rather than restoring wholesale, because the new default may carry changes you want to keep. Restart containerd, drain the stuck Pods so the kubelet retries them cleanly, and confirm one Pod reaches Running before uncordoning. Restarting the runtime under a live workload is a step to take deliberately rather than casually, which is a further reason to have drained the node before touching it. Holding is a legitimate alternative if the maintenance window has closed: leave the node cordoned, let its existing workload keep serving, and schedule the repair - but assign an owner and an end time, because a cordoned node is capacity the cluster is paying for and not using.

Verification

The only verification that means anything is a Pod that starts. Schedule a throwaway Pod onto the node by name and watch it reach Running, because that exercises RunPodSandbox, the image fetch, the CNI attachment and container start in the order they actually happen. Confirm the sandbox image is now resolvable from the node itself rather than trusting the configuration file, and confirm the value in the file matches a healthy peer node. Check that the test Pod received an IP, since a sandbox that is created but not networked is a different fault wearing the same symptom. Then look wider: the same upgrade ran on other nodes, so grep the same setting across the fleet and confirm no other node is one scale-up away from the same incident. Finally, confirm the stuck Pods have cleared rather than assuming they will, since some will have been waiting long enough that their owning controller has given up on them.

Prevention

The defect is not the setting; it is that a package upgrade could silently revert node configuration and nothing noticed for a day. Make configuration management own the runtime configuration and run after the package upgrade, so the node converges rather than depending on a file surviving. Add a post-change validation that actually starts a Pod on every node touched by a maintenance window - a node that has not run a new Pod since it was patched has not been tested, and the DaemonSet that reports 4/4 is reporting a sandbox created last week. Uncordon on the strength of that test rather than on the strength of the node reporting Ready. Alert on Pods stuck in ContainerCreating for longer than a few minutes, grouped by node, because that single signal separates a node-local fault from a workload fault immediately. Finally, treat runtime configuration as cluster configuration: it decides whether Pods can start at all, and it deserves the same review and the same drift detection as anything held in the API server.

Reported symptoms

Tuesday’s maintenance window patched four worker nodes. Wednesday afternoon a batch namespace misses its SLA: none of its jobs have started. Someone notices they are all on worker-04.

By Wednesday evening the shape is clear and nobody can explain it:

  • Every Pod scheduled onto worker-04 since the window sits in ContainerCreating indefinitely. No crashes. No ImagePullBackOff. Restart count zero.
  • Every Pod that was already running on worker-04 before the window is healthy and serving traffic.
  • The node is Ready, has never been anything else, and reports plenty of free CPU and memory - so the scheduler keeps choosing it.
  • The stuck Pods have no IP address. That routes the incident to the network team, who spend an afternoon on the CNI and find nothing wrong with it.
  • kubectl logs on a stuck Pod returns an error instead of output. The application team reads that as “the Pod has not been created yet” and escalates it as a scheduling problem.
  • containerd is running. crictl ps answers instantly and lists the node’s running containers. The platform team concludes the runtime is healthy and hands the ticket back.

Three other nodes were patched in the same window with the same playbook, and none of them do this. The DaemonSet that is supposed to prove node health is reporting 4/4.

Evidence provided

Read-only / Safethe image in the error is not the application image
$ kubectl describe pod batch-import-9k2mv -n batch | tail -8
Events:
Type     Reason                  Age                    From     Message
----     ------                  ----                   ----     -------
Normal   Scheduled               21m                    default-scheduler  Successfully assigned batch/batch-import-9k2mv to worker-04
Warning  FailedCreatePodSandbox  19m (x9 over 21m)      kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.10": failed to pull image "registry.k8s.io/pause:3.10": context deadline exceeded

Illustrative output

Read-only / Safeno IP on the new Pods - and one Pod from last week is perfectly happy
$ kubectl get pods -n batch -o wide | head -4
NAME                  READY   STATUS              RESTARTS   AGE   IP        NODE
batch-import-9k2mv    0/1     ContainerCreating   0          21m   <none>    worker-04
batch-import-c4t8p    0/1     ContainerCreating   0          21m   <none>    worker-04
batch-export-tt61w    1/1     Running             0          6d    10.244.4.31   worker-04

Illustrative output

Read-only / Safea sandbox image is cached, but not the one the error names
# crictl images | grep -i pause
IMAGE                         TAG      IMAGE ID       SIZE
harbor.example.com/k8s/pause  3.9      e6f181688397   744kB

Illustrative output

Read-only / Safethe internal registry works from this node; the upstream one does not
# crictl pull registry.k8s.io/pause:3.10; crictl pull harbor.example.com/batch/import:2.4.1
FATA[0030] pulling image: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Image is up to date for sha256:9d2f0e4b1c...

Illustrative output

Read-only / Safethe package upgrade replaced the file, and kept the old one
# grep -n sandbox_image /etc/containerd/config.toml /etc/containerd/config.toml.dpkg-old
/etc/containerd/config.toml:61:    sandbox_image = "registry.k8s.io/pause:3.10"
/etc/containerd/config.toml.dpkg-old:58:    sandbox_image = "harbor.example.com/k8s/pause:3.9"

Illustrative output

Read-only / Safethe kubelet is not stuck - it is retrying, every sync, and failing every time
# journalctl -u kubelet --since '30 min ago' | grep -c RunPodSandbox
214

Illustrative output

Work the evidence before reading on

The kubelet is running. The runtime is running. The node is Ready. The scheduler placed the Pods correctly. And no Pod has started on this node for a day.

  1. Read the FailedCreatePodSandbox message again and identify which image it names. Is that image anywhere in the Pod spec?
  2. The old Pods on this node are fine and the new ones are not. What does a running Pod already have that a new one has to be given?
  3. crictl ps works. What exactly does that prove, and what does it not prove?
  4. The stuck Pods have no IP. Which step in Pod startup assigns one, and does it run before or after the step that is failing?

Before continuing: the node has run this workload for six days. What changed on Tuesday that a Pod created on Monday would never notice?

Root cause

1. The sandbox is a Pod’s first dependency, and it is not in the Pod spec

The kubelet’s first CRI call for a new Pod is RunPodSandbox. The sandbox is the container that owns the Pod’s namespaces - the thing the Pod’s application containers are later joined to. It is built from an image the runtime chooses, configured on the node, and named nowhere in any manifest anyone writes.

Tuesday’s containerd upgrade replaced /etc/containerd/config.toml with the package default. That default names the upstream sandbox image. The node’s egress policy does not permit the upstream registry - which is the entire reason the internal mirror was configured in the first place - so the pull times out, the sandbox is never created, and the Pod stops there.

The old value was not lost. The package manager saved it beside the new file, where nobody looked, because nobody had a reason to suspect the runtime configuration of a workload problem.

2. Failing that early makes the Pod invisible to workload diagnostics

Everything an operator reaches for by reflex assumes a container exists.

kubectl logs needs a container to read from, so it errors. The Pod has no application image pull to fail, so there is no ImagePullBackOff to see. There is no restart count, because nothing has run and therefore nothing has restarted. And the Pod has no IP, because the network is attached during sandbox creation - so a failure at that step reads exactly like a CNI failure to anyone who checks the IP first and the events second.

Four teams each looked at the layer they own, found it healthy, and were right. The failure sits in the one layer no team was watching: the node’s own runtime configuration, which is not in the API server and does not appear in any kubectl output.

3. Ready is a claim about the kubelet, not about the node’s usefulness

The kubelet runs its status update loop independently of its Pod sync loop. The status loop reports node conditions and heartbeats on its own schedule; the sync loop reconciles Pods. Neither waits on the other.

So a kubelet that has failed to start every Pod for twenty-four hours reports a healthy node the entire time, and the scheduler - which reads that report - keeps sending it work. The node is not lying about anything it was asked. It was simply never asked the question that mattered.

Resolution

  1. Cordon the node before anything else. It is advertising capacity it cannot use, and every minute it stays schedulable it collects more Pods that will never start. This costs nothing and it stops the incident growing while you work.
  2. Drain the node so the running workload moves somewhere that works. The remaining steps restart the container runtime, and that is not a thing to do underneath live traffic when you have the option not to.
  3. Diff the saved configuration against the new package default. Take the local settings the node needs - the sandbox image, any registry mirrors - and leave the new defaults alone. Do not copy the old file back over the new one.
  4. Restart containerd and confirm it comes back cleanly in its own journal before looking at Kubernetes at all. A runtime that failed to parse its configuration is a worse position than the one you started from.
  5. Delete the stuck Pods so their controllers recreate them and the kubelet retries from a clean state. Some will have been waiting long enough that their owning controller has already given up.
  6. Start one throwaway Pod pinned to this node and watch it reach Running with an IP before uncordoning. Ready is not the gate; a Pod that started is the gate.
  7. Uncordon, then check the same setting on every node the maintenance window touched. Three other nodes ran the same upgrade and may be one scale-up away from the same fault - they have simply not been asked to start a Pod yet.
  8. Record what the package replaced and why configuration management did not put it back. That is the defect worth fixing; the setting itself is only the symptom that made it visible.

Verification

  1. A new Pod pinned to the node reaches Running. This is the whole verification, because it is the only check that exercises RunPodSandbox, the image fetch, the network attachment and container start in the order they actually occur.
  2. The test Pod has an IP. A sandbox that is created but not networked is a different fault presenting with the same symptom, and only the IP separates them.
  3. The sandbox image resolves from the node itself, not merely in the configuration file. Pull it by hand and confirm; a value that is correct and unreachable fails identically to a value that is wrong.
  4. The setting matches a healthy peer node. Compare rather than inspect - the fastest way to be confident a node is repaired is that it now looks like the nodes that never broke.
  5. No other node in the fleet carries the reverted value. Run the same check everywhere the upgrade ran, because a node that has not been asked to start a Pod since the window has not been tested.
  6. The previously stuck Pods have cleared and their controllers report the expected replica counts, rather than being assumed to have recovered on their own.
  7. The post-maintenance procedure now ends with a Pod-start test and the node is uncordoned on that result. A procedure change that is agreed but not written down has not happened.

Prevention

  • Make configuration management own the runtime configuration and run after the package upgrade. The setting was correct until a package replaced the file. Nothing put it back because nothing was watching. Convergence after the upgrade is the fix; hoping the file survives is not.

  • End every maintenance window with a Pod that starts. A node that has been patched and not asked to run a new Pod has not been tested, whatever its conditions say:

NODE=worker-04
kubectl run smoke-"$NODE" --image=harbor.example.com/ops/busybox:1.36 \
  --restart=Never --overrides='{"spec":{"nodeName":"'"$NODE"'"}}' -- true
kubectl wait --for=condition=Ready pod/smoke-"$NODE" --timeout=120s

  • Uncordon on the strength of that test, not on the node reporting Ready. Ready is the kubelet’s opinion of its own health. It is not a statement that the node can do the job it is about to be given.

  • Alert on Pods in ContainerCreating beyond a few minutes, grouped by node. One signal, grouped that way, separates a node-local fault from a workload fault instantly - and it is the alert that would have turned a day-long four-team investigation into a five-minute one.

  • Distrust read-only health checks of write paths. crictl ps answering is not evidence that the runtime can create anything. Where a component is checked by reading, know what the check omits.

  • Treat node runtime configuration as cluster configuration. It decides whether Pods can start at all, and it lives outside the API server where no Kubernetes-native tool will show you drift. It deserves review and drift detection like anything else that can stop the cluster working.