Node heartbeats — the lease mechanism and the controller's grace period

The node heartbeat is the kubelet’s signal that it is alive. The kubelet renews a Lease object in the kube-node-lease namespace every 10 seconds; the node controller monitors the Lease and detects a failing node when the Lease is not renewed. This lesson walks the Lease mechanism, the node controller’s grace period, and the operational patterns.

The Lease object

The Lease is a Kubernetes object (apiVersion: coordination.k8s.io/v1, kind: Lease) in the kube-node-lease namespace. The object is named after the node:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: node-1
  namespace: kube-node-lease
spec:
  holderIdentity: node-1
  leaseDurationSeconds: 40
  renewTime: "2026-08-16T10:00:00Z"

The kubelet renews the Lease by patching the spec.renewTime field. The kubelet’s renewal is the heartbeat.

The Lease is a separate object from the Node. The Lease is small (a few bytes); the controller can watch many Leases efficiently. The Lease is the cluster’s primary signal for liveness.

The kubelet’s renewal

The kubelet’s renewal loop runs every 10 seconds (the --node-status-update-frequency flag, default 10s). The loop:

1. Reads the current Lease object.
2. Compares the renewTime with the current time.
3. If the renewTime is older than 10 seconds, sends a PATCH to update the renewTime.

The renewal is a Kubernetes API call. The kubelet authenticates with its client certificate; the API server validates the certificate and updates the Lease.

The renewal is independent of the Node’s status update. The kubelet can update the Node’s status without renewing the Lease; the kubelet can renew the Lease without updating the Node’s status. The two are independent.

The node controller’s grace period

The node controller in kube-controller-manager monitors the Lease. The controller’s logic:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant L as Lease
    participant NC as Node controller

    K->>L: renew lease
    NC->>L: watch lease
    Note over K,NC: now < renewTime + grace period
    K->>L: renew lease
    K->>L: renew lease
    K->>L: lease not renewed
    Note over K,NC: now > renewTime + grace period
    NC->>NC: mark node NotReady
    NC->>NC: add not-ready taint
    K->>L: renew lease (resumes)
    NC->>NC: mark node Ready

The grace period is the time the node controller waits before declaring the node NotReady. The default is 40 seconds.

The grace period is configurable:

kube-controller-manager \
  --node-monitor-period=5s \
  --node-monitor-grace-period=40s \
  --pod-eviction-timeout=5m

• --node-monitor-period: the time between Lease checks. Default 5s.
• --node-monitor-grace-period: the time the Lease is allowed to be stale before the node is NotReady. Default 40s.
• --pod-eviction-timeout: the time the Pod is allowed to be on a NotReady node before eviction. Default 5m.

The defaults are calibrated to the typical heartbeat period (10s) and the typical network blip duration (40s).

The failure detection latency

The failure detection latency is the sum of:

1. The kubelet’s heartbeat period (default 10s).
2. The node controller’s monitor period (default 5s).
3. The grace period (default 40s).

The total is approximately 40s + 5s + 10s = 55s. The node controller detects a failing node approximately 55 seconds after the kubelet stopped.

The total latency is the cluster’s failure detection time. The Pod eviction latency is the time from the detection to the Pod’s eviction. The Pod eviction timeout is the time the Pod is allowed to be on a NotReady node before eviction (default 5m).

The total time from kubelet failure to Pod eviction is approximately 55s + 5m = 5m 55s. The cluster’s failure detection is dominated by the Pod eviction timeout.

The Lease’s failure modes

The Lease’s failure modes:

The diagnostic:

NODE=worker-03            # the Lease is named after the node

kubectl get lease -n kube-node-lease "$NODE" -o yaml

The renewTime shows the last renewal. A renewTime that is older than the grace period is a stale Lease.

The eviction timeout

The Pod eviction timeout is the time the Pod is allowed to be on a NotReady node before the cluster’s garbage collector evicts it. The default is 5 minutes.

kube-controller-manager \
  --pod-eviction-timeout=5m

The eviction timeout is independent of the grace period. A NotReady node may have its Pods evicted after the eviction timeout, even if the kubelet recovers.

The eviction timeout gives the kubelet time to recover. A workload that tolerates a 5-minute node failure is a workload that the cluster can re-create after eviction.

The Lease in production

The Lease is a small object; the cluster can watch many Leases efficiently. The node lifecycle controller reads the renewTime on every sync and compares it against --node-monitor-grace-period.

The operational patterns:

• Monitor the Lease’s renewTime. The cluster’s metrics expose the Lease’s renewTime. The operator should alert on a stale Lease.
• Tune the grace period for the workload. A latency-sensitive workload may want a 5-second grace period; a batch workload may want a 60-second grace period.
• Audit the Lease at every node repave. A new node that joins the cluster with the wrong Lease is a node that is failing silently. The audit catches the failure.
• The kubelet’s clock must be synchronized. A kubelet whose clock is skewed by more than the grace period is a kubelet that is reporting false heartbeats. The production rule is to use the cluster’s NTP service.

The Lease and the static Pod

The kubelet’s static Pods are not Lease-renewed. The static Pods are managed by the kubelet directly; the kubelet’s Lease renewal covers the static Pods.

The static Pods are not in the cluster’s view; the kubelet’s heartbeat is the only signal that the node is running. A kubelet whose Lease is stale is a node whose static Pods are not running.

Quiz

Production discipline

• The Lease is the cluster’s heartbeat. The kubelet renews the Lease every 10 seconds; the node controller monitors the Lease. A stale Lease is a NotReady node.
• The grace period is the cluster’s tolerance for transient failures. The default is 40 seconds. The production rule is to tune the grace period for the workload’s sensitivity.
• The eviction timeout is the cluster’s tolerance for failed nodes. The default is 5 minutes. The production rule is to tune the eviction timeout for the workload’s startup time.
• The kubelet’s clock must be synchronized. A clock skew can cause false NotReady alerts. The production rule is to use the cluster’s NTP service.
• Monitor the Lease’s renewTime. The cluster’s metrics expose the Lease’s renewTime. The operator should alert on a stale Lease.
• Audit the Lease at every node repave. A new node that joins the cluster with the wrong Lease is a node that is failing silently. The audit catches the failure.