Unknown and NotReady nodes — partial observability

The Node object has a Ready condition that resolves to True, False, or Unknown. The False state is NotReady; the Unknown state is Unknown. The two states have different diagnostics; the two states have different recoveries. This lesson walks the states, the controller’s transitions, and the operational patterns.

The Ready condition states

The Ready condition is a tri-state:

The True and False states are set by the kubelet. The Unknown state is set by the node controller when the controller cannot reach the kubelet.

The cluster’s components react to the Ready status. A Pod’s spec.nodeName is the node the Pod is running on; the Pod’s status is the cluster’s view of the Pod.

The NotReady state

The NotReady state (Ready=False) is set by the kubelet when the kubelet detects a problem. The kubelet reports:

NodeCondition {
  type: "Ready",
  status: "False",
  reason: "KubeletNotReady",
  message: "kubelet is not posting ready status"
}

A node in NotReady is a node whose kubelet is running but is reporting a problem. The kubelet’s status update is still being delivered to the API server.

The diagnostic:

# Substitute your own value before running:
NODE=worker-03.example.com

kubectl describe node "$NODE"

The events on the node indicate the failure. The fix is to investigate the kubelet’s logs and the node’s local state.

The recovery:

journalctl -u kubelet | tail -50

The kubelet’s logs show the failure. The fix is to restart the kubelet or to fix the underlying problem.

The Unknown state

The Unknown state (Ready=Unknown) is set by the node controller when the Lease is not renewed for the grace period. The controller’s report:

NodeCondition {
  type: "Ready",
  status: "Unknown",
  reason: "NodeStatusNeverUpdated",
  message: "kubelet has not reported ready status for 5m"
}

A node in Unknown is a node whose kubelet is not reachable. The kubelet may be down, the network may be partitioned, or the kubelet’s process may have crashed.

The diagnostic:

# Substitute your own value before running:
NODE=worker-03.example.com

kubectl get lease -n kube-node-lease "$NODE" -o yaml

The renewTime shows the last renewal. A renewTime that is older than the grace period is a stale Lease.

The recovery:

1. Check the node’s network. The kubelet may be unreachable because of a network partition.
2. Check the kubelet’s process. The kubelet may have crashed.
3. Restart the kubelet. systemctl restart kubelet.
4. Wait for the kubelet to recover. The kubelet updates the Lease; the node controller removes the Unknown state.

The transitions

The cluster’s transitions for the Ready condition:

stateDiagram-v2
    [*] --> Ready: kubelet registered
    Ready --> NotReady: kubelet fails
    NotReady --> Ready: kubelet recovers
    Ready --> Unknown: lease stale
    Unknown --> Ready: lease renewed
    Unknown --> NotReady: kubelet reports NotReady
    NotReady --> Unknown: lease stale

The transitions are:

• Ready → NotReady: the kubelet reports a problem.
• NotReady → Ready: the kubelet recovers.
• Ready → Unknown: the Lease is stale.
• Unknown → Ready: the Lease is renewed.
• Unknown → NotReady: the kubelet reports a problem (concurrency).
• NotReady → Unknown: the kubelet stops reporting (concurrency).

The most common transition is Ready → NotReady → Ready (a transient kubelet failure). The Unknown state is rare; it indicates a more severe problem.

The cluster’s view of the Pods

The cluster’s view of the Pods on a NotReady or Unknown node is the same as the cluster’s view of the Pods on a Ready node until the eviction timeout. The Pods are still running; the cluster’s services do not route to them (because the node is NotReady).

# Substitute your own value before running:
NODE=worker-03.example.com

kubectl get pods -o wide | grep "$NODE"

The Pods are listed as Running (until the eviction timeout). The cluster’s controllers do not consider the Pods as failed until the eviction timeout.

The Pod’s eviction

The Pod’s eviction is performed by the cluster’s eviction logic. The eviction is triggered when the Pod’s spec.nodeName is on a NotReady node for longer than the eviction timeout.

The eviction timeout is the time the Pod is allowed to be on a NotReady node before eviction. The default is 5 minutes.

The eviction is performed by the Pod’s controller (the Deployment, StatefulSet, etc.). The controller creates a replacement Pod; the replacement Pod is scheduled by the scheduler.

The Pod’s eviction is destructive. The Pod’s state is lost. The Pod’s volume is preserved (the PVC is still bound).

The diagnostic workflow

The diagnostic workflow for a NotReady or Unknown node:

flowchart TD
    A[Node NotReady or Unknown] --> B[Read events]
    B --> C[Read kubelet logs]
    C --> D[Check Lease]
    D --> E{Lease stale?}
    E -->|Yes| F[Read kubelet process]
    E -->|No| G[Read node conditions]
    F --> H[Restart kubelet]
    G --> I[Investigate conditions]

The workflow:

1. Read the node’s events. The events indicate the failure.
2. Read the kubelet’s logs. The log shows the local failure.
3. Check the Lease. The Lease’s renewTime is the last heartbeat.
4. Read the node’s conditions. The conditions indicate the failure type.
5. Restart the kubelet. If the kubelet is failing, the restart may recover the node.

The recovery

The recovery for a NotReady node:

1. Identify the failure. The kubelet’s logs show the cause.
2. Fix the underlying problem. Restart the failing service, free the disk, or address the resource pressure.
3. Restart the kubelet. systemctl restart kubelet.
4. Wait for the node to be Ready. The kubelet reports Ready=True.

The recovery for an Unknown node:

1. Identify the network failure. The kubelet may be unreachable because of a network partition.
2. Restore the network. The fix is to restore the network on the node.
3. Restart the kubelet. A kubelet that lost its API server connection re-registers and renews the Lease on start.
4. Wait for the kubelet to renew the Lease. The node controller removes the Unknown state.

Quiz

Production discipline

• A NotReady node is a node whose kubelet is failing. The diagnostic is the kubelet’s logs.
• An Unknown node is a node whose Lease is stale. The diagnostic is the Lease’s renewTime.
• The cluster’s view of the Pods is the same as before the failure. The Pods are still running until the eviction timeout.
• The Pod’s eviction is destructive. The Pod’s state is lost. The Pod’s volume is preserved.
• Monitor the Ready condition. The cluster’s alerts should fire on Ready=False for more than 5 minutes.
• Audit the recovery at every node repave. A new node that joins the cluster with the wrong configuration is a node that is failing silently. The audit catches the failure.
• Test the kubelet’s restart in non-production. A kubelet that does not recover from a restart is a kubelet that is failing silently.