Container runtime unavailable · Break/Fix · Kubernetes

Reported symptoms

At 09:31 the on-call is paged for worker-07 NotReady in the production cluster. Twenty minutes later the node is still NotReady, and none of the usual conclusions fit.

The applications on worker-07 are still running. Their log lines are still arriving in the central pipeline, timestamped seconds ago, from Pods the control plane has given up on. Whatever is wrong with this node, it has not stopped the workload.

kubectl exec into a Pod on worker-07 hangs and then fails. The same command against a Pod on any other node returns instantly. kubectl get pod and kubectl describe pod answer normally for the very same Pods.

Every Pod the scheduler has placed on worker-07 since 09:12 is in ContainerCreating and has never left it. There are eleven, and they are all on this one node.

A Pod deleted at 09:20 is still Terminating.

The Deployments that had Pods here show the right number of Ready replicas: replacements appeared on other nodes about five minutes after the node went NotReady. The StatefulSet member that was on the node was never replaced, and its Pod has read Terminating for a quarter of an hour.

The kubelet on worker-07 is alive. It is renewing its lease, it is reporting node conditions, and its journal is producing several lines a second.

That last fact is what makes the page confusing. A dead kubelet explains a NotReady node. A live one does not.

Evidence provided

Read-only / Safeone node, and only one

$ kubectl get nodes

NAME        STATUS     ROLES           AGE    VERSION
worker-06   Ready      <none>          214d   v1.34.1
worker-07   NotReady   <none>          214d   v1.34.1
worker-08   Ready      <none>          214d   v1.34.1

Illustrative output

Read-only / Safethree states at once on one node

$ kubectl get pods -A --field-selector spec.nodeName=worker-07

NAMESPACE   NAME                        READY   STATUS              RESTARTS   AGE
prod        checkout-6f9c4d-2xk8w       0/1     ContainerCreating   0          19m
prod        checkout-6f9c4d-9tzqr       1/1     Running             0          31d
prod        ledger-0                    1/1     Terminating         0          17d
prod        reporting-58b7f9-h4mnp      0/1     ContainerCreating   0          18m

Illustrative output

Read-only / Safeon worker-07, over SSH

$ systemctl status containerd

● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled)
   Active: failed (Result: exit-code) since 09:12:41 UTC

Illustrative output

Read-only / Safea clean stop, then a start that never completes

$ journalctl -u containerd -n 20 --no-pager

systemd[1]: Stopping containerd container runtime...
systemd[1]: Starting containerd container runtime...
containerd[3141]: failed to load TOML from /etc/containerd/config.toml
systemd[1]: containerd.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: containerd.service: Failed with result 'exit-code'.

Illustrative output

The exact wording of a configuration-load failure varies between containerd releases. The load-bearing parts of this journal do not: the unit was stopped cleanly, the restart failed while reading the configuration file, and the process exited before it could create its socket.

Read-only / Safenothing is listening on the CRI socket

$ crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

FATA[0002] validate service connection: validate CRI v1 runtime API for endpoint
"unix:///run/containerd/containerd.sock": rpc error: code = Unavailable

Illustrative output

Read-only / Safethe file is four days old; the unit has no successful start to report

$ stat -c %y /etc/containerd/config.toml; systemctl show containerd -p ActiveEnterTimestamp

2026-08-14 02:41:18.402 +0000
ActiveEnterTimestamp=

Illustrative output

Work the evidence before reading on

Three questions, in this order. Each one is cheaper than the next.

The kubelet is alive and the node is NotReady. Name every operation the kubelet performs that requires the runtime, and check each reported symptom against that list. Does anything remain unexplained?
The applications are still serving. Which part of the request path for an already-running Pod involves containerd at all?
The configuration file is dated 14 August. The unit was restarted on 18 August. What is the relationship between those two timestamps, and what would kubectl get events from the day of the outage show about the change that caused it?

Before continuing, answer the question that decides the first action: is it safe to force-delete the Pod that has been Terminating for fifteen minutes? Justify the answer from what a force-delete does and what it does not do.

Root cause

1. containerd is dead, and it died four days after it broke

/etc/containerd/config.toml was edited on 14 August to add a registry mirror. The block was appended to the end of the file, and it re-declares a table that already appears earlier in it. TOML forbids that, so the file no longer parses.

Nothing happened on 14 August. containerd reads its configuration when it starts, and it was already running with the previous file loaded in memory. The broken file sat on disk for four days, inert.

At 09:12 on 18 August the monthly package upgrade restarted the unit. That restart is where the four-day-old edit took effect, and containerd has not started since.

This gap is the whole reason the incident is hard to read. The change log for the day of the outage is empty of anything relevant. The change that caused it is four days upstream, in a window that closed green.

2. One dead daemon, four symptoms

Everything on the page follows from a single division of labour: the kubelet knows the cluster’s desired state, the runtime knows the local containers, and the CRI is the only bridge between them.

Symptom	The CRI call the kubelet cannot make
Pods stuck in `ContainerCreating`	`RunPodSandbox`, then `CreateContainer`
Pod stuck in `Terminating`	`StopContainer`, then `RemovePodSandbox`
`kubectl exec` fails	`Exec` on the RuntimeService
Node reports NotReady	the kubelet cannot manage containers and says so

And the symptom that is not on that list is the one that misleads everybody: the applications keep serving, because serving a request needs none of it.

3. The Deployment healed and the StatefulSet did not

Five minutes after the node’s Ready condition went False, the node controller’s node.kubernetes.io/not-ready:NoExecute taint reached the end of its default 300-second toleration and the Pods on the node were marked for deletion.

For the Deployment, that was enough: the Pods have deletion timestamps, so the ReplicaSet created replacements elsewhere. They are Ready. The Deployment looks healthy, and the original Pods are still running on worker-07, still holding their connections to everything downstream.

For the StatefulSet, the deletion never completed - completing it requires the runtime - so the Pod object still exists, and the controller will not create a second ledger-0 while the first one is unaccounted for. That is the guarantee working correctly. It is also why the member is simply missing rather than duplicated.

Both behaviours are correct. They point in opposite directions, which is why reading either one alone sends the investigation somewhere else.

Under the hood

containerd does not hold the container processes as children. It starts a per-container shim process which becomes the supervisor, and the shim survives the death of the daemon that started it. That is a deliberate design property: it is what lets containerd be upgraded or restarted without killing every workload on the host.

The consequence during this incident is that the node has two truths at once. On the host, the container processes are alive, holding their sockets, and answering. In the API server, their Pods are marked for deletion, their replacements are Ready, and the node is NotReady.

Traffic to a Pod IP is forwarded by rules the kube-proxy already wrote, across a network namespace the CNI already configured, into a process that is already running. None of those three things asks containerd for permission. That is why the data path is unaffected, and why every check that reads the data path will report the node as healthy for as long as the outage lasts.

The force-delete is the trap

The instinct on a Pod that has been Terminating for fifteen minutes is --force --grace-period=0. That instinct is correct on a healthy node whose application is ignoring SIGTERM. Here it is the worst available action.

A force-delete does not stop a container. It removes the Pod object from the API server without waiting for the kubelet to confirm the container is gone - and on this node the kubelet cannot confirm anything, because confirming requires the runtime that is down.

The container keeps running. Its Pod object no longer exists, so nothing in the cluster is tracking it, and the StatefulSet controller is now free to create a second ledger-0 elsewhere. Two members with the same identity, both writing, and the only record that the first one exists is a process table on a node the control plane has stopped talking to.

A force-delete on a NotReady node requires evidence that the process is dead. There are exactly two kinds of evidence that qualify: a runtime that answers and confirms it, or a node that is powered off.

Resolution

Decide, and say out loud, that nothing on this node gets force-deleted until the runtime answers or the node is off. Everything below depends on that decision holding under time pressure.
Cordon the node with kubectl cordon worker-07. The not-ready taint already stops scheduling, but it is removed the moment the node recovers, and the recovery is not the moment you want new work arriving.
Read the containerd journal for the parse failure and the file it names. This is the step that converts a node incident into a configuration incident.
Diff the file against a healthy node in the same pool: ssh worker-08 cat /etc/containerd/config.toml against the local copy. The difference is the change, and it carries its own date.
Preserve the broken file. Copy it aside before restoring; it is the only evidence of what was written and when, and the restore overwrites it.
Restore the known-good configuration, then validate it with containerd config dump before restarting anything. That command reads the same file the daemon reads and fails the same way, so it answers the parse question without asking systemd to answer it in production.
Restart containerd and watch the journal through the start rather than checking the exit code afterwards. A unit that starts and then fails a second later reports success to an impatient reader.
Confirm the runtime before looking at Kubernetes: crictl info must answer, and crictl ps must list the sandboxes that were running before the incident. If containerd came back with no state, the recovery is a different and much longer procedure.
Let the kubelet reconnect on its own. It retries on its sync interval; there is no step here, and restarting the kubelet to hurry it along only adds a variable.
Watch the pending work drain: the eleven ContainerCreating Pods start, and the deletions the kubelet has been holding since 09:20 finally complete.
Reconcile the workloads. For each affected Deployment, compare running Pods against declared replicas and confirm the duplicates are gone; confirm ledger-0 was recreated exactly once.
Uncordon the node only after a canary Pod has reached Running on it.

If the configuration cannot be recovered inside the window, holding is a real option and should be treated as one: the workload is serving, and a node that is NotReady with healthy Pods is not an emergency. It is not free either - the cluster is running with one node of lost capacity, the StatefulSet is a member short, and every Deployment on that node is carrying a duplicate. Give the hold an owner and an end time, and write down what happens at that time.

If the node must instead be abandoned, power it off first. Powering it off is the only action available that actually stops the containers, and it is what makes the subsequent force-deletes safe rather than merely fast.

Verification

The runtime answers. crictl info returns, and crictl ps lists the pre-incident sandboxes. Check this before any kubectl command, because the node condition lags the daemon and a green kubectl get nodes read too early proves nothing.
The kubelet journal has stopped reporting runtime connection failures. A single successful crictl call does not prove the kubelet reconnected.
The node reports Ready and the node.kubernetes.io/not-ready taint is gone from kubectl describe node worker-07.
The create path works, not just the socket. Schedule one canary Pod to the node and watch it reach Running - that exercises RunPodSandbox, CreateContainer and StartContainer, none of which crictl info demonstrates.
The streaming path works. kubectl exec into a Pod on the node returns. This was one of the reported symptoms and is the only check that closes it.
The stuck deletions completed. The Pod that had been Terminating since 09:20 is gone from kubectl get pods, and it left because the kubelet stopped it rather than because somebody forced it.
No duplicates remain. For each affected workload, the running Pod count equals the declared replica count, and ledger-0 exists exactly once.
The guard can fail. On a test node, write a file with a duplicated table and confirm the pre-restart validation refuses it. A validation step that has only ever passed has not been tested.

Prevention

Parse the configuration in the same run that writes it. A file whose only validator is a production restart is a file that is validated in production, four days late, by an outage.
Compare the file mtime against the service start time as a standing fleet check. stat -c %Y /etc/containerd/config.toml newer than the ActiveEnterTimestamp from systemctl show containerd means a configuration is staged and untested. One command per node, and it finds this entire class of fault - including the version where a human edited the file and forgot.
Alert on the combination, not the condition. NotReady alone is a page that gets acknowledged and ignored, because on this failure every dashboard stays green. NotReady while the node’s Pods are still producing logs is the runtime signature and it has no other cause worth guessing at.
Alert on ContainerCreating older than a few minutes, grouped by node. A cluster of stuck sandbox creations on one node is a runtime failure; the same count spread across the cluster is a registry or CNI problem. The grouping is what makes the alert diagnostic rather than noisy.
Write down the force-delete rule before you need it, and put it in the same runbook as the node-recovery procedure, because that is the page somebody will have open at 09:31.
Restart containerd deliberately and watch it. It is not a fire-and-forget command: the node condition lags, so an operator who restarts the daemon and moves on will be told nothing for a minute or more.

What if the kubelet had died instead of containerd?

The node would still read NotReady, and almost nothing else would match.

A dead kubelet stops renewing the node lease, so the node controller stops hearing from the node entirely and the condition becomes Unknown rather than False - which kubectl get nodes still prints as NotReady, and which is why that column is a starting point rather than a diagnosis. There would be no kubelet journal to read, no runtime connection errors, and no new Pods stuck in ContainerCreating, because nothing would be accepting them in the first place.

The containers would keep running in that case too, for the same reason they do here: neither daemon is in the request path. So the two failures share their most misleading symptom and differ in every diagnostic one. Ready False means the kubelet is alive and telling you something is wrong. Unknown means nobody is telling you anything. They need opposite investigations - one on the node, one on the path to it - and the first useful question on any NotReady page is which of the two you have.

Why this matters

Twenty minutes of a node the control plane had written off, running production traffic the whole time, with a Deployment reporting full health because it had quietly doubled and a StatefulSet member simply absent.

The mechanism worth carrying is the split between the control path and the data path. Creating, stopping, inspecting and exec-ing into a container all go through the runtime; serving a request goes through rules and namespaces that were programmed before the runtime died. When the runtime fails, everything in the first list breaks at once and nothing in the second one moves, which is exactly the shape that makes a single fault look like four.

The operational lesson is narrower and sharper. On a node in this state, the API server’s view and the node’s view have come apart, and every command that acts on the API server’s view alone - a force-delete above all - widens the gap instead of closing it. Fix the runtime, or turn the node off. Those are the two ways to make the two views agree again.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. containerd is dead, and it died four days after it broke

2. One dead daemon, four symptoms

3. The Deployment healed and the StatefulSet did not

Resolution

Verification

Prevention