Node NotReady

Reported symptoms

The first page fires at 02:14, during a scheduled control-plane patch window. Two worker nodes are NotReady. At 02:31 three more. At 02:49 a sixth. By the time the day team arrives, six of fifty workers are NotReady and have stayed that way.

What the on-call found overnight:

• SSH to every affected node connects instantly. Load average is normal. Disks are not full. systemctl status kubelet says active and running, and the process has been up for 34 days.
• kubectl logs and kubectl exec against Pods on those nodes fail at once with a dial-backend error. This looked like an API server problem, so the first twenty minutes went into the API server, which was healthy the whole time.
• The infrastructure dashboard shows all six nodes green. Their node-exporter metrics never stopped arriving.
• Five minutes after each node was marked NotReady, its Pods were evicted and rescheduled onto other nodes. The resulting scheduling surge paged the capacity team, who opened a separate incident about an unexplained scale-up.
• Customer-facing traffic never dipped. Right up to the moment they were evicted, the Pods on the failing nodes were serving requests normally.
• The six nodes are in three different racks, are two different instance types, were built across four months, and sit in the same availability zone as forty healthy nodes. Every correlation the team looked for came up empty.

At 02:20 the on-call rebooted the first NotReady node, which is the standard reflex and the thing to think hardest about in this scenario. It did not come back Ready. It did kill six Pods that were serving traffic at that moment, five minutes before the cluster would have moved them gracefully.

Evidence provided

$ kubectl get nodes --no-headers | grep -c ' NotReady '

6

$ kubectl describe node worker-23 | sed -n '/Conditions/,/Addresses/p'

Conditions:
Type             Status    LastHeartbeatTime                 Reason
----             ------    -----------------                 ------
MemoryPressure   Unknown   Fri, 15 Aug 2026 02:14:07 +0000   NodeStatusUnknown
DiskPressure     Unknown   Fri, 15 Aug 2026 02:14:07 +0000   NodeStatusUnknown
PIDPressure      Unknown   Fri, 15 Aug 2026 02:14:07 +0000   NodeStatusUnknown
Ready            Unknown   Fri, 15 Aug 2026 02:14:07 +0000   NodeStatusUnknown

$ kubectl get lease -n kube-node-lease worker-23 -o jsonpath='{.spec.renewTime}'

2026-08-15T02:14:03.000000Z

# systemctl show kubelet -p ActiveState -p ActiveEnterTimestamp -p NRestarts

ActiveState=active
ActiveEnterTimestamp=Wed 2026-07-12 09:41:22 UTC
NRestarts=0

# journalctl -u kubelet --since 02:10 --until 02:20 | grep -i lease | tail -3

Aug 15 02:14:13 worker-23 kubelet[1187]: E0815 02:14:13.882 controller.go:145] "Failed to ensure lease exists, will retry" err="Get https://10.20.0.10:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-23: dial tcp 10.20.0.10:6443: i/o timeout"
Aug 15 02:14:20 worker-23 kubelet[1187]: E0815 02:14:20.104 controller.go:145] "Failed to ensure lease exists, will retry" err="dial tcp 10.20.0.10:6443: i/o timeout"
Aug 15 02:14:27 worker-23 kubelet[1187]: E0815 02:14:27.339 controller.go:145] "Failed to ensure lease exists, will retry" err="dial tcp 10.20.0.10:6443: i/o timeout"

# APISERVER=10.20.0.10; curl -sS -m 5 -k https://$APISERVER:6443/livez; echo; hostname -I

curl: (28) Connection timed out after 5001 milliseconds
10.20.4.31

# APISERVER=10.20.0.10; curl -sS -m 5 -k https://$APISERVER:6443/livez; echo; hostname -I

ok
10.20.1.62

$ kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name,IP:.status.addresses[0].address,READY:.status.conditions[-1].status | grep Unknown

worker-19   10.20.4.14   Unknown
worker-23   10.20.4.31   Unknown
worker-27   10.20.4.44   Unknown
worker-31   10.20.4.58   Unknown
worker-34   10.20.4.71   Unknown
worker-38   10.20.4.92   Unknown

Work the evidence before reading on

The team spent the night on the nodes and on the API server. Both were healthy. Three things in the evidence say the fault is neither.

1. The Ready condition is Unknown, not False. Those are different states with different meanings and different investigations. False is the kubelet reporting a problem. Unknown is the node controller saying nothing has renewed the lease - which is a statement about hearing, not about health.
2. The kubelet is running, has never restarted, and is logging a dial timeout to a specific address and port. A process that describes its own failure precisely is not the broken component.
3. The infrastructure dashboard and kubectl disagree about the same six machines. They cannot both be wrong, and they are measuring over different network paths.

Before reading on: the node-exporter scrape reaches these nodes and the lease renewal does not. Both are IP traffic. What is different about the two paths, and what would have to be true of the network for exactly one of them to fail?

Then look at the last piece of evidence again and ask what those six addresses have in common that the forty-four healthy ones do not.

Root cause

1. Unknown means the control plane stopped hearing, not that the node stopped working

The kubelet renews a Lease object in the kube-node-lease namespace roughly every ten seconds. The node controller watches those leases; when one is not renewed within the grace period, the controller marks the node Ready=Unknown and applies the node.kubernetes.io/unreachable taint, which starts the eviction timer for the Pods on it.

Every one of those steps is performed by the control plane, using the absence of a signal. None of them involves asking the node anything. So Ready=Unknown carries exactly one piece of information: the lease is stale. It says nothing about the kubelet, the runtime, the disk or the workload, and in this incident all four were fine.

The distinction against Ready=False is the whole first branch of the diagnostic. False is the kubelet actively reporting a problem, which means it is reachable, which means you investigate on the node. Unknown means the kubelet is not being heard, which means you investigate the path between the node and the control plane. The team investigated on the node for four hours because kubectl get nodes prints NotReady for both.

2. The path was removed a week earlier and did nothing for a week

A firewall change replaced a rule that referenced the worker security group with an explicit list of subnet CIDRs. The list was assembled by hand from the original build and omits 10.20.4.0/22, the subnet added when the cluster was expanded. Because the expansion stayed inside the same availability zone, none of the checks by zone, rack, instance type or build date found a pattern - the correlating dimension was the subnet, and nobody groups nodes by subnet.

The change was applied a week before the incident and had no visible effect at the time. That is not luck. On a stateful firewall, removing a rule stops new flows from being permitted; flows already tracked continue. The kubelets on those six nodes held established connections to the API server, and those connections kept working exactly as before.

3. The control-plane restart is what tested the change

The three API servers were restarted one at a time during the patch window at 02:12, 02:29 and 02:47. Each restart forced the kubelets connected to that instance to open a new connection. The nodes in 10.20.0.0/22 reconnected. The nodes in 10.20.4.0/22 were refused by the firewall and never got another connection.

That is the entire explanation for the shape of the incident. The six failures at 02:14, 02:31 and 02:49 are not a progressive hardware fault, a memory leak, or a cascading failure. They are six nodes discovering, in three groups, a rule change that had been latent for a week, at the moment something forced them to ask permission again.

Resolution

1. Stop touching the affected nodes. They are serving. Anything you do to them costs Pods and gains nothing until the path is understood.
2. Confirm the direction of the failure from the node, not from the control plane. On an affected node, set APISERVER to the control-plane address and run curl -sS -m 5 -k https://$APISERVER:6443/livez. A timeout means the node cannot reach the control plane; run the identical command from a healthy node to prove the control plane is fine.
3. Characterise the failing set before fixing anything. List the NotReady nodes with their internal addresses and look for the dimension that separates them from the healthy ones. Here it is the subnet, and it takes one kubectl get nodes -o custom-columns to see.
4. Correlate the failure timestamps with the change log for the window. The three groups line up with the three API server restarts, which is what tells you the trigger was a reconnect rather than a node-side degradation.
5. Restore the firewall rule for the omitted subnet, covering both directions: node to control plane on 6443, and control plane to kubelet on 10250. Fixing only the first leaves kubectl exec and kubectl logs broken and the incident half-closed.
6. Let the nodes rejoin on their own. The kubelet is already retrying; once the path exists the lease renews within seconds and the node controller clears the condition and the taint without help. Nothing needs to be rebuilt.
7. Repair the rule properly rather than by adding one more CIDR. Express it against the tag or security group that defines the worker role, so the next subnet is in scope automatically, and record why the CIDR list existed in case something depended on it.

Verification

1. The path works from every node in the previously blocked subnet, not from the one you tested. Run the same curl against the API server from all six, because a hand-edited CIDR list can be wrong in more than one place.
2. The reverse direction works: kubectl exec and kubectl logs succeed against a Pod on each recovered node. This exercises the control-plane-to-kubelet path on 10250, which is a different rule from the one you probably just fixed.
3. The lease is advancing. kubectl get lease -n kube-node-lease worker-23 -o jsonpath for the renewTime, sampled twice thirty seconds apart, must show it moving. This is the direct evidence that the heartbeat is back, and it recovers before the node conditions do.
4. All fifty nodes report Ready, and the node.kubernetes.io/unreachable taint is gone from the six. Check the taint explicitly: a node can read Ready while a stale taint still repels workload.
5. The evicted Pods have rescheduled and the cluster is back to its normal distribution. Confirm the capacity team can close their scale-up incident, since it was a downstream effect of this one.
6. The test that can fail: restart one API server deliberately, in hours, and confirm no node goes NotReady. That is the exact event that exposed the fault, and it is the only check that proves the fix holds against the trigger rather than against the symptom.
7. The change record names the subnet, both port directions and the rule form that replaced the CIDR list. A fix recorded as "opened 6443" will be re-broken by the next person tidying firewall rules.

Prevention

• Write firewall rules against the group, tag or role that defines a set of hosts, never against a hand-maintained list of CIDRs. A list is correct on the day it is written and silently incomplete from the next subnet onward.
• Treat a stateful firewall change as untested until something forces a reconnect. The change applied cleanly, broke nothing for a week, and then took out six nodes during an unrelated maintenance. Schedule a deliberate reconnect - or at minimum watch node readiness through the next control-plane restart - as part of closing any network change.
• Alert on the node lease directly rather than only on the node Ready condition. A stale lease is earlier, unambiguous, and names the failing direction, whereas NotReady is the same word for two opposite problems.
• Know which path each of your health signals traverses. When node-exporter says healthy and kubectl says dead, the answer is in the difference between those routes, and having written that down in advance turns a four-hour investigation into a five-minute one.
• Group nodes by subnet when characterising a failing set, not only by zone, rack, instance type and age. A cluster that has been expanded has more network boundaries than it has zones, and the newest boundary is the one least represented in everyone’s mental model.
• Write down, in the on-call runbook, that an Unknown node with running Pods is a hold-and-investigate situation and not a reboot-and-move-on one. The reflex is fast, feels productive, and in this failure mode costs workload while gaining nothing.