The node controller — cluster-level node lifecycle management

The node controller is the cluster-level manager of node lifecycle. It runs as a goroutine in the kube-controller-manager; it monitors the cluster’s nodes, applies the taints, assigns the Pod CIDRs, and triggers the Pod evictions. This lesson walks the controller’s responsibilities, the taints it applies, the Pod CIDR assignment, and the operational patterns.

The node controller’s responsibilities

The node controller’s primary responsibilities:

flowchart TD
    A[Node controller] --> B[Monitor Lease]
    A --> C[Apply taints]
    A --> D[Set conditions]
    A --> E[Assign Pod CIDR]
    A --> F[Trigger Pod eviction]
    B --> G[Detect NotReady]
    C --> H[not-ready, unreachable]
    D --> I[Ready, NetworkUnavailable]
    E --> J[spec.podCIDR]
    F --> K[Eviction timeout]

• Monitor Lease: watch the kube-node-lease namespace for stale Leases.
• Apply taints: add the not-ready and unreachable taints when the Lease is stale.
• Set conditions: update the Node’s Ready and NetworkUnavailable conditions.
• Assign Pod CIDR: assign a CIDR to new nodes.
• Trigger Pod eviction: evict Pods on a NotReady node after the eviction timeout.

The controller’s behaviour is the cluster’s view of the node. The controller’s taints and conditions are what the cluster’s components react to.

The controller’s monitor loop

The node controller’s monitor loop runs every --node-monitor-period (default 5s). The loop iterates over the cluster’s nodes and checks the Lease for each:

sequenceDiagram
    autonumber
    participant NC as Node controller
    participant L as Lease
    participant API as API server

    NC->>L: watch Lease
    Note over NC: every 5s
    NC->>NC: check renewTime
    Note over NC: now > renewTime + grace period?
    NC->>API: mark node NotReady
    NC->>API: add not-ready taint

The loop is simple: for each node, check the Lease’s renewTime. If the renewTime is older than the grace period, the node is NotReady.

The loop is bounded by the number of nodes. A cluster with 1000 nodes has 1000 Lease checks every 5s; the controller’s CPU usage is bounded by the number of nodes.

The controller’s taints

The node controller adds the following taints:

• node.kubernetes.io/not-ready: applied when the Lease is stale. Effect: NoExecute. The tolerationSeconds is 300 (the default).
• node.kubernetes.io/unreachable: applied when the Lease is stale. Effect: NoExecute. The tolerationSeconds is 300.
• node.kubernetes.io/network-unavailable: applied when the CNI has not yet configured the node. Effect: NoSchedule. Removed when the CNI reports ready.

The two NoExecute taints are the cluster’s eviction trigger. The default tolerationSeconds of 300 gives the Pod 5 minutes to be evicted.

# Substitute your own value before running:
NODE=worker-03

kubectl describe node "$NODE" | grep -A 5 "Taints"

Taints:             node.kubernetes.io/not-ready:NoExecute for 300s
                    node.kubernetes.io/unreachable:NoExecute for 300s

The for 300s indicates the taint’s tolerationSeconds. A Pod that does not tolerate the taint is evicted after 300s.

The controller’s conditions

The node controller updates the Node’s conditions:

NodeCondition {
  type: "Ready",
  status: "False",
  reason: "NodeStatusNeverUpdated",
  message: "kubelet has not reported ready status for 5m"
}

The controller’s Ready overrides the kubelet’s Ready when the Lease is stale. The cluster’s view of the node’s Ready is the most recent update.

The controller also updates the NetworkUnavailable condition. The condition is set to True when the CNI has not yet configured the node; the condition is set to False when the CNI agent reports ready.

The controller’s Pod CIDR assignment

The node controller assigns a Pod CIDR to new nodes. The assignment is based on the cluster’s ClusterCIDR specification:

spec:
  clusterCIDR: 10.244.0.0/16
  clusterCIDRMaskSize: 24

The controller assigns a CIDR from the cluster’s CIDR range. The CIDR is added to the node’s spec.podCIDR.

The assignment is performed when the node is registered. The CIDR is removed when the node is deleted.

The controller’s CIDR assignment is the cluster’s mechanism for partitioning the Pod IP space. The CNI plugin uses the CIDR to assign Pod IPs.

The controller’s Pod eviction

The controller triggers the Pod eviction when the node is NotReady. The eviction is performed by the Pod’s controller (the Deployment, StatefulSet, etc.), not by the node controller. The node controller’s role is to mark the node as NotReady; the eviction timeout is the time the Pod is allowed to be on the NotReady node.

The eviction timeout is configurable:

kube-controller-manager \
  --pod-eviction-timeout=5m

The default is 5 minutes. The timeout is the time the Pod is allowed to be on a NotReady node before the cluster’s garbage collector evicts it.

The eviction timeout is independent of the grace period. A NotReady node may still have its Pods running for the duration of the eviction timeout.

The controller’s cloud integration

In a cloud-managed cluster, the node controller’s behaviour is complementary to the cloud-controller-manager. The cloud-controller-manager handles the cloud-specific operations (the IP address, the instance metadata); the node controller handles the cluster-level operations (the taints, the conditions, the CIDR).

The two controllers are independent. They both watch the Node objects; they both update the Node object. The updates are reconciled by the cluster’s serialization.

The controller’s failure modes

The node controller’s failure modes:

The diagnostic:

# The control-plane node whose kube-controller-manager static Pod you want:
CP_NODE=cp-01

kubectl logs -n kube-system "kube-controller-manager-$CP_NODE" | grep node

The logs show the controller’s actions. The fix is to investigate the controller’s logs and the API server.

The controller’s configuration

The controller’s flags:

• --node-monitor-period: the time between Lease checks. Default 5s.
• --node-monitor-grace-period: the time the Lease is allowed to be stale before the node is NotReady. Default 40s.
• --pod-eviction-timeout: the time the Pod is allowed to be on a NotReady node before eviction. Default 5m.
• --node-eviction-rate: the maximum number of nodes evicted per second. Default 0.1 (one node every 10 seconds).

The flags are set on the kube-controller-manager. The configuration is the cluster’s tuning for the node lifecycle.

The controller’s API rate limits

The controller’s API rate limits are critical. The controller watches every node and every Lease; the controller’s API calls are bounded by the cluster’s size.

A cluster with 1000 nodes has 1000 Lease checks every 5s; the controller’s API calls are 200/s. The cluster’s API server must handle the rate.

The controller’s --node-eviction-rate flag limits the rate of node evictions. A cluster that evicts a large number of nodes at once is a cluster that overwhelms the API server. The production rule is to limit the eviction rate.

Quiz

Production discipline

• The node controller is the cluster’s view of the nodes. The controller’s taints and conditions are what the cluster’s components react to.
• Tune the grace period for the workload. A latency-sensitive workload may want a 5-second grace period; a batch workload may want a 60-second grace period.
• Tune the eviction timeout for the workload. The default is 5 minutes; a workload with a long startup time may want a longer timeout.
• Monitor the controller’s metrics. The node_collector_* metrics expose the controller’s health. The operator should alert on the controller’s failure.
• Audit the controller’s flags at every release. A controller that is misconfigured is a cluster that is not updating the nodes.
• The controller’s API rate limits are critical. The controller’s API calls are bounded by the cluster’s size. The production rule is to limit the eviction rate.