NotReady and unknown nodes — the heart of the worker

A node that stops reporting does not take its Pods with it: they keep running, keep serving traffic, and keep their volumes attached while the control plane works out whether the node is broken or merely unreachable. A Ready condition of False means the kubelet is alive and saying something is wrong; Unknown means nothing has renewed the node lease within the grace period and the control plane has stopped hearing from it at all. kubectl get nodes prints NotReady for both, and the two need opposite investigations — one on the node, one between you and it.

The node heart

A node’s heart is the kubelet’s lease. The kubelet renews the lease every 10s (default). The node controller checks the lease; if the lease is not renewed for 40s (default), the node is marked NotReady.

flowchart TD
    A[kubelet] -->|Renew lease every 10s| B[Node lease]
    B --> C{Lease renewed?}
    C -->|Yes| D[Node Ready]
    C -->|No| E[Mark NotReady]
    E --> F[Evict Pods]

A NotReady node is one whose kubelet has not renewed the lease.

The node conditions

The node’s conditions are the cluster’s view of the node’s state:

# Substitute your own value before running:
NODE=worker-03

kubectl describe node "$NODE"

A real node’s conditions:

Conditions:
  Type             Status  LastHeartbeatTime                 Reason                       Message
  ----             ------  -----------------                 ------                       -------
  Ready            True    Fri, 16 Aug 2026 04:23:01 +0000   KubeletReady                 kubelet is posting ready status
  MemoryPressure   False   Fri, 16 Aug 2026 04:23:01 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 16 Aug 2026 04:23:01 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 16 Aug 2026 04:23:01 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available

The diagnostic is the Ready condition. If Ready is False, the node is NotReady.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
NODE=worker-03

# 1. Check the node's conditions
kubectl describe node "$NODE"

# 2. Check the kubelet's logs (run on the node itself)
journalctl -u kubelet -n 200

# 3. Check the runtime's logs (run on the node itself)
journalctl -u containerd -n 200

# 4. Check the node's resources
kubectl top node "$NODE"

# 5. Check the node's events
kubectl get events --field-selector involvedObject.name="$NODE"

The diagnostic is the node conditions, the kubelet logs, the runtime logs, and the resources.

NotReady vs Unknown

A NotReady node is one whose kubelet has reported NotReady. The kubelet is running but the node is not in a usable state.

An Unknown node is one whose kubelet has not reported at all. The kubelet cannot be reached. The node is unreachable from the control plane.

flowchart TD
    A[Node status] --> B{kubelet reachable?}
    B -->|Yes| C[NotReady]
    B -->|No| D[Unknown]
    C --> E[Fix kubelet]
    D --> F[Fix network or kubelet]

The diagnostic is the kubelet’s reachability.

Common causes of NotReady

• Kubelet failing. The kubelet is in CrashLoopBackOff.
• Runtime failing. The containerd is failing.
• Resource pressure. The node is in MemoryPressure or DiskPressure.
• Network partition. The node cannot reach the API server.

Common causes of Unknown

• Network partition. The node cannot reach the API server.
• Kubelet is dead. The kubelet process is crashed.
• Node is dead. The node is unreachable (e.g., power off).

The remediation

The remediation depends on the cause:

# Substitute your own values before running. NODE is the name the API server
# knows the node by; NODE_SSH is the address you reach it on:
NODE=worker-03
NODE_SSH=worker-03.example.com

# Option 1: Restart the kubelet (run on the node itself)
systemctl restart kubelet

# Option 2: Restart the containerd (run on the node itself)
systemctl restart containerd

# Option 3: Drain and replace the node
kubectl drain "$NODE" --ignore-daemonsets --force

# Option 4: Reboot the node
ssh "$NODE_SSH" "sudo reboot"

The remediation is the node recovery.

Production discipline

A NotReady or Unknown node is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the node, identify the cause, apply the remediation. The node is the cluster’s worker; the remediation is the node recovery.

• Check the node conditions. The conditions are the cluster’s view.
• Check the kubelet logs. The kubelet is the node’s voice.
• Check the runtime logs. The runtime is the node’s container engine.

Quiz