Node controller — heartbeats, NotReady, eviction

The node controller is the controller-manager component responsible for tracking node health. When a kubelet stops sending heartbeats, the node controller marks the node NotReady, applies the NoExecute taint, and the eviction subsystem removes Pods from the failed node. This lesson walks the controller’s role, the timeline of a node failure, and the production discipline of tuning.

The node controller in one sentence

The node controller watches kubelet lease renewals; when the lease expires, the node controller marks the node NotReady, applies the node.kubernetes.io/not-ready NoExecute taint, and the eviction loop removes Pods from the failed node.

sequenceDiagram
    autonumber
    participant K as kubelet on node
    participant AS as API server
    participant NC as Node controller
    K->>AS: lease renew every 10s
    Note over NC: lease is current
    K-->>K: kubelet fails
    Note over K: lease stops renewing
    NC->>AS: check lease
    AS-->>NC: lease expired
    NC->>AS: mark node Ready=False
    NC->>AS: apply NoExecute taint
    NC->>AS: eviction loop removes Pods
    K-->>AS: kubelet recovers
    NC->>AS: remove NoExecute taint

The controller’s logic is bounded by two flags: --node-monitor-grace-period and --node-eviction-timeout.

The grace period

# Default in 1.34
--node-monitor-grace-period=40s
--node-eviction-timeout=5m0s

The grace period is the time the kubelet can be silent before the node is considered NotReady. The eviction timeout is the time after NotReady before Pods are kicked.

gantt
    title Node failure timeline
    dateFormat HH:mm
    axisFormat %H:%M
    section Healthy
    Lease renews :a1, 00:00, 60s
    section NotReady
    Lease expires :b1, 00:01, 30s
    Mark NotReady :c1, after b1, 5s
    Apply NoExecute taint :d1, after c1, 5s
    section Eviction
    Eviction loop :e1, after d1, 60s
    Pods removed :f1, after e1, 90s

The grace period default (40s) is calibrated for typical kubelet latency. The eviction timeout default (5m) is the “time between NotReady and actual Pod loss”.

The eviction subsystem

When the NoExecute taint is applied, the eviction loop:

flowchart LR
    T[Taint node] -->|kubelet watches| QE[Queue new eviction]
    QE -->|notifies Pods on node| NG[Pod termination grace period]
    NG -->|30s default| TERM[Terminate Pod]
    TERM -->|kubelet PATCHes status to API server| API

The eviction is kubelet-driven: the kubelet watches its own taints and evicts Pods on the node. The controller’s role is to apply the taint; the kubelet’s role is to do the eviction.

PodDisruptionBudget interaction

The eviction respects PDB:

kubectl delete pod web-0 -n prod  # would normally succeed
# but PDB is minAvailable: 2 / 3 replicas -> webhook returns 403

If a PDB has minAvailable or maxUnavailable, the eviction may be rejected by the PDB webhook. The node controller’s loop sees the rejection and skips the eviction.

A PDB-blocked eviction is logged but not retried at the node controller level.

Cordoning (operator action)

The node controller does not handle cordoning; that’s an operator action via kubectl cordon:

kubectl cordon cp-3

A cordoned node is spec.unschedulable=true. The node controller does not evict Pods on a cordoned node; the operator uses kubectl drain to evict Pods.

The NotReady recovery

When the kubelet comes back:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant AS as API server
    participant NC as Node controller
    K->>AS: lease renew (after recovery)
    NC->>AS: check lease
    AS-->>NC: lease renewed
    NC->>AS: mark node Ready=True
    NC->>AS: remove NoExecute taint
    K->>AS: list Pods (still Terminated / Evicted)
    NC->>AS: scheduler runs (Pod placements for the now-Ready node)

The recovery is asymmetric: the taint removal is fast, but Pod scheduling is up to the scheduler. The cluster recovers by the scheduler placing new Pods (the deleted ones are gone; controllers re-create them).

The race conditions

A node can transition NotReady and back to Ready quickly:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant NC as Node controller
    K-->>NC: lease expires
    NC->>NC: schedule eviction (timer)
    K->>NC: lease renews (kicked back)
    NC->>NC: cancel eviction
    Note over NC: the eviction timer fires
    NC->>K: taint applied (race)
    K-->>NC: lease renews
    NC->>K: taint removed (recovery)

A flaky network that makes the lease briefly expire triggers an eviction that may or may not have actually run. The grace period balances this — short grace causes false positives; long grace causes real failures to be slow.

The flag interactions

A high-availability cluster tunes these for the network:

# Multi-AZ cluster, WAN delays
--node-monitor-grace-period=90s
--node-eviction-timeout=10m

A cloud-managed cluster may use:

# Single AZ, low-latency network
--node-monitor-grace-period=30s
--node-eviction-timeout=3m

The metrics

- alert: NodeNotReady
  expr: kube_node_status_condition{condition="Ready",status="true"} == 0
  for: 5m
  labels:
    severity: critical

- alert: NodeEvictionLatency
  expr: time() - max(node_controller_evictions_total{result="success"})

A growing rate of NotReady nodes or evictions indicates infrastructure trouble.

The production failure mode

A common failure mode: the cloud provider has transient storage issues that slow kubelet. The node controller sees the lease as expired. The eviction loop fires; Pods are removed; the cloud recovers; the scheduler re-lands the Pods.

The resilience strategy:

• Tune the grace period for the cluster’s typical latency.
• Set the eviction timeout high enough to avoid premature eviction.
• Monitor for eviction storms (a spike in evictions is suspicious).

$ kubectl describe node cp-1 | grep -E 'Conditions:|Ready|NodeStatus'

...

The NodeStatus reflects the node controller’s view of the node.

Quiz

Production discipline

• Tune the grace period for the network. Default 40s; raise for slow networks.
• Watch eviction rates. Spikes indicate real infrastructure trouble.
• Document the timeout values. A cluster’s time-to-NotReady is operational; document it in the runbook.
• Test the recovery path. Kill a kubelet and observe the eviction timeline.
• Respect tolerations. Pods with NoExecute tolerations stay on the node; design tolerations deliberately.

The node controller is the cluster’s mechanism for identifying failed hosts and removing their workload. Operating it well is keeping the time-to-NotReady short for real failures and long enough to tolerate blips.