Node observers and the node controller — who decides status

The Node object’s status is the consensus of multiple observers: the kubelet reports its own state, the node controller monitors the cluster’s view, the cloud controller adds topology information, and the cluster’s admission plugins validate. Each observer contributes a piece; no single observer owns the truth. This lesson walks the roles, the authority, and the operational patterns for diagnosing conflicts.

The observers

The four primary observers of a Node object:

flowchart LR
    A[kubelet] -->|Status update| B[API server]
    C[Node controller] -->|Taints, conditions| B
    D[Cloud controller] -->|Labels, addresses| B
    E[Lease] -->|Heartbeat| C
    B --> F[Node object]
    F --> G[Scheduler read]
    F --> H[Operator read]

• kubelet: the primary observer. Reports the node’s addresses, capacity, conditions, and active Pods.
• node controller: the cluster’s monitor. Reports taints and conditions based on the lease and the kubelet’s status.
• cloud controller: the cloud-integration observer. Reports the cloud’s metadata (instance type, zone, region).
• Lease (kube-node-lease): the heartbeat. The kubelet renews the lease every 10 seconds; the node controller watches the lease.

The four observers update different fields of the Node object. The kubelet updates status.addresses, status.capacity, status.allocatable, and the conditions typed by the kubelet (Ready, MemoryPressure, etc.). The node controller updates the conditions typed by the controller (the NodeNetworkUnavailable and the Ready condition when the kubelet is not reachable). The cloud controller updates the metadata.labels for cloud-specific labels.

The kubelet as an observer

The kubelet reports the node’s status every 10 seconds (the --node-status-update-frequency flag). The status includes:

• Addresses: the node’s InternalIP, Hostname, and optional ExternalIP and ExternalDNS.
• Capacity: the node’s total CPU, memory, pods, ephemeral-storage.
• Allocatable: the capacity minus the kubelet’s reserved resources.
• Conditions: the kubelet’s view of the node (Ready, MemoryPressure, DiskPressure, PIDPressure).
• Images: the cached images.
• Active Pods: the Pods running on the node.

The kubelet’s status is the primary source of truth for the node’s runtime state. The node controller reads the status and reacts to changes.

The node controller as an observer

The node controller runs in the kube-controller-manager. The controller’s responsibilities:

• Monitor the lease. The controller checks the kube-node-lease/<node-name> lease every --node-monitor-period (default 5s).
• Set the not-ready taint. If the lease is not renewed for --node-monitor-grace-period (default 40s), the controller sets the not-ready NoExecute taint.
• Set the unreachable taint. If the lease is not renewed for --node-monitor-grace-period, the controller also sets the unreachable NoExecute taint.
• Update the Ready condition. If the lease is missing, the controller sets Ready=False. The kubelet’s Ready is the primary; the controller’s overrides when the kubelet is unreachable.
• Assign Pod CIDR. When a new node is registered, the controller assigns a CIDR from the cluster’s spec.clusterCIDR.
• Evict Pods. The controller evicts Pods on a NotReady node after the grace period.

The node controller is the cluster’s judge: it decides what the cluster’s view of the node is, regardless of what the kubelet reports.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant L as Lease
    participant NC as Node controller
    participant API as API server

    K->>L: renew lease every 10s
    NC->>L: watch lease
    K->>API: status update every 10s
    NC->>API: check last status update
    Note over K,NC: grace period 40s
    K->>L: lease stale
    NC->>API: set Ready=False, taint not-ready
    K->>L: lease renewed
    NC->>API: set Ready=True, remove taint

The grace period is the cluster’s tolerance for transient network blips. A kubelet that is unreachable for 40s is presumed dead; the controller takes over.

The cloud controller as an observer

The cloud-controller-manager runs the cloud’s node lifecycle controller. The controller:

• Reads the cloud’s metadata service for the node’s instance type, zone, region.
• Updates the node.kubernetes.io/instance-type label.
• Updates the topology.kubernetes.io/zone and topology.kubernetes.io/region labels.
• Updates the ExternalIP and ExternalDNS addresses.
• Removes the node.cloudprovider.kubernetes.io/uninitialized taint when the controller has finished.

The cloud controller is the cluster’s translator: it takes the cloud’s metadata and converts it to the cluster’s labels.

The cloud controller’s updates are infrequent — the labels and addresses are set once at node registration and only change if the node moves. The controller’s heartbeat is the node’s lease, not its own.

The status fields and their authority

A Node object’s status fields are updated by different observers. The rule:

The authority is the cluster’s contract: each field is the responsibility of one observer, and conflicts are resolved by the cluster’s serialization (the most recent update wins).

Authority conflicts

A node whose status is divergent between observers is a node with an authority conflict. The conflict is not visible to the cluster’s view (the most recent update wins), but the operator can detect it by reading the observers’ logs.

# Substitute your own values before running. CP_NODE is the control-plane node
# whose kube-controller-manager static pod you are reading; NODE is the node
# under investigation:
CP_NODE=cp-1
NODE=node-1

journalctl -u kubelet | grep "Status"
kubectl logs -n kube-system "kube-controller-manager-$CP_NODE" | grep "$NODE"

The kubelet and the node controller should agree on the node’s Ready condition. If the kubelet reports Ready but the node controller reports Ready=False, the controller’s heartbeat is lagging the kubelet’s status.

A more common conflict: the kubelet and the cloud controller disagree on the ExternalIP. The kubelet reports the local interface’s IP; the cloud controller reports the cloud’s metadata. The two values should match; if they differ, the kubelet is running on a different interface than the cloud expects.

The node’s audit log

The cluster’s audit log records every change to the Node object. The audit shows who changed what:

# Substitute your own value before running:
NODE=node-1

kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name="$NODE"

The events include:

• The node’s status update (with the kubelet’s identity).
• The taint changes (with the node controller’s identity).
• The label changes (with the cloud controller’s or the operator’s identity).

The audit log is the operator’s primary source of truth for “who changed the node’s status.” A node that is failing should have a corresponding audit log entry.

The status-sync period

The kubelet’s --node-status-update-frequency (default 10s) is the time between status updates. The node controller’s --node-monitor-period (default 5s) is the time between lease checks. The two are independent.

A cluster that wants faster failure detection can:

• Lower the kubelet’s --node-status-update-frequency to 5s.
• Lower the node controller’s --node-monitor-period to 2s.
• Lower the node controller’s --node-monitor-grace-period to 20s.

The cost is more API server traffic; the benefit is faster detection of a failing node.

The lease as a heartbeat

The kube-node-lease namespace holds the leases. Each node has a lease object named after the node:

# Substitute your own value before running:
NODE=node-1

kubectl get lease -n kube-node-lease "$NODE" -o yaml

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: node-1
  namespace: kube-node-lease
spec:
  holderIdentity: node-1
  leaseDurationSeconds: 40
  renewTime: "2026-08-16T10:00:00Z"

The lease is renewed by the kubelet every 10 seconds. The node controller watches the lease’s renewTime. If the lease is not renewed for leaseDurationSeconds (default 40s), the controller treats the node as unreachable.

The lease is a cheap heartbeat. The kubelet updates the Node object’s status every 10s; the lease is updated separately. The two are independent. The lease is the cluster’s primary signal for liveness.

The role of the cluster’s admission

The cluster’s admission plugins validate the Node object. The validation plugins:

• Validate the node’s spec: the podCIDR and podCIDRs shapes.
• Validate the node’s status: the conditions and addresses.
• Validate the node’s labels: the well-known labels.

The cluster’s NodeRestriction admission plugin constrains what labels a kubelet can set on its own Node object. The kubelet can only set labels with the kubernetes.io/ or node.kubernetes.io/ prefix, and only on its own node. This prevents a kubelet from labelling other nodes.

Quiz

Production discipline

• The kubelet is the primary observer. The kubelet’s status is the cluster’s view of the node. The operator should monitor the kubelet’s status-update frequency.
• The node controller is the cluster’s judge. The controller’s verdict on the node’s Ready condition is the cluster’s truth. The operator should monitor the controller’s grace period.
• The cloud controller is the cluster’s translator. The cloud’s labels are the cluster’s labels. The operator should audit the cloud’s labels.
• The lease is the cluster’s heartbeat. A stale lease is a node that is failing. The operator should monitor the lease’s renewTime.
• The status fields have authority. A node whose observers disagree is a node that is partially observed. The operator should resolve the conflict.
• Tune the failure detection for the workload. A financial-trading cluster may want 5s detection; a batch workload may want 60s. The defaults are reasonable but not universal.