Node shutdown and Pod termination — graceful vs forceful

Node shutdown is a special case of Pod termination. The kubelet may not have time to gracefully terminate Pods if the node is powered off quickly. This lesson covers graceful node shutdown, how kubelet integrates with systemd inhibitors, and what happens when shutdown is not graceful.

The shutdown problem

When a node shuts down:

• systemd sends SIGTERM to all services, including kubelet.
• The kubelet’s shutdown handler begins terminating Pods.
• The kubelet has limited time before systemd sends SIGKILL (default 90s, configurable via TimeoutStopSec in the systemd unit).
• If the kubelet is SIGKILL’d before all Pods are terminated, Pods are lost without graceful shutdown.

sequenceDiagram
    participant Systemd
    participant Kubelet
    participant Pods
    participant Node as Node hardware

    Systemd->>Systemd: shutdown initiated
    Systemd->>Kubelet: SIGTERM
    Note over Kubelet: shutdown handler starts
    Kubelet->>Pods: terminate gracefully
    par graceful termination
        Pods-->>Pods: SIGTERM, drain, exit
    and
        Systemd->>Systemd: countdown TimeoutStopSec
    end
    alt kubelet finishes
        Kubelet-->>Systemd: exit
        Systemd->>Node: power off
    else timeout
        Systemd->>Kubelet: SIGKILL
        Systemd->>Node: power off (Pods lost)
    end

Without graceful node shutdown, the kubelet has the TimeoutStopSec window (default 90s) to terminate all Pods. For nodes with many Pods and short grace periods, this is not enough time.

Graceful node shutdown

The kubelet has a graceful node shutdown feature (beta in 1.21+, GA in 1.27+). The kubelet:

1. Listens for systemd’s inhibitor lock release (which signals impending shutdown).
2. On inhibitor release, starts terminating all Pods on the node.
3. Holds the inhibitor lock until termination is complete or the kubelet’s shutdown grace period expires.

The systemd side:

• The kubelet takes a shutdown inhibitor lock when it starts. systemd waits for the inhibitor to be released before powering off.
• The kubelet releases the inhibitor after it has finished terminating Pods (or after the kubelet’s shutdown grace period).

This coordination gives kubelet the time it needs to gracefully terminate Pods, without racing the kernel’s shutdown.

Configuring kubelet for graceful shutdown

The kubelet flags:

• --shutdown-grace-period: total time the kubelet has to terminate all Pods (default 30s).
• --shutdown-grace-period-critical-pods: time the kubelet has to terminate critical Pods (default 10s). Critical Pods are those with priorityClassName: system-cluster-critical or system-node-critical.

The total shutdown grace period is --shutdown-grace-period (regular Pods) + --shutdown-grace-period-critical-pods (critical Pods after the regular ones).

# /etc/kubernetes/kubelet-config.yaml
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 30s

These translate to kubelet flags. For a maintenance window:

• The systemd TimeoutStopSec must be ≥ the kubelet’s total shutdown grace period.
• The kubelet must be able to take the shutdown inhibitor (default in modern systemd).

Systemd integration

systemd’s inhibitor locks let services delay or prevent system shutdown. The kubelet takes a shutdown inhibitor during its lifetime; when systemd begins shutdown, it waits for the inhibitor to be released.

sequenceDiagram
    participant Systemd
    participant Kubelet
    participant Pods

    Kubelet->>Systemd: acquire shutdown inhibitor
    Systemd->>Systemd: shutdown initiated
    Systemd->>Kubelet: inhibitor release signal
    Kubelet->>Pods: terminate gracefully
    Pods-->>Kubelet: all terminated
    Kubelet->>Systemd: release inhibitor
    Systemd->>Systemd: proceed with shutdown

Without the inhibitor:

• systemd sends SIGTERM to kubelet.
• kubelet begins termination.
• systemd’s TimeoutStopSec expires (default 90s).
• systemd sends SIGKILL to kubelet; pods may not be fully terminated.

With the inhibitor:

• systemd sends SIGTERM to kubelet.
• kubelet begins termination.
• systemd waits for kubelet to release the inhibitor.
• kubelet finishes termination, releases the inhibitor.
• systemd proceeds with shutdown.

Pod termination ordering

The kubelet terminates Pods on shutdown in this order:

1. Critical Pods first (priorityClassName: system-cluster-critical or system-node-critical).
2. Regular Pods next.
3. Within each group, the kubelet tries to terminate all Pods simultaneously (it sends SIGTERM to each in parallel).
4. Wait for --shutdown-grace-period to expire or all Pods to exit.
5. SIGKILL any remaining Pods.

The --shutdown-grace-period-critical-pods controls the time the kubelet gives critical Pods; this is shorter than the regular shutdown grace period because critical Pods (typically add-ons like kube-proxy, CNI) are designed to shutdown quickly.

Forceful shutdown

When a node is powered off forcefully (hard power cycle, hardware failure, kernel panic), there is no opportunity for graceful shutdown. The Pods are lost:

• In-flight requests are cut off.
• In-memory state is lost.
• The kubelet’s last reported state may be stale; the API server has not received termination events.

After forceful shutdown:

• The node is NotReady (the kubelet cannot report).
• After node-monitor-grace-period (default 50s), the node is marked NotReady by the node controller.
• After pod-eviction-timeout (default 5 minutes), the Pods are evicted (deleted) and recreated on other nodes.

The application sees a Pod termination that looks like forced deletion: exit code 137, no graceful shutdown, lost work.

Production patterns

Maintenance window with graceful shutdown:

# 1. Cordon the node (no new Pods scheduled)
kubectl cordon node-3

# 2. Drain the node (PDB-protected; voluntary)
kubectl drain node-3 --ignore-daemonsets

# 3. Perform the maintenance (kernel upgrade, hardware swap)

# 4. Reboot — kubelet takes the shutdown inhibitor;
#    Pods are terminated gracefully

# 5. After reboot, uncordon
kubectl uncordon node-3

The drain handles the voluntary disruption; the reboot handles the unavoidable shutdown. PDBs protect the drain; graceful shutdown protects the reboot.

Forced shutdown (last resort):

# Power off immediately, no graceful shutdown
sudo systemctl poweroff --force

# Or hard reboot
sudo reboot --force

Use this only when the node is unresponsive or the maintenance is urgent. Pods on the node are lost; the cluster will recreate them after pod-eviction-timeout.

Cross-course references

• The Linux course part IX-Linux-Boot covers systemd and shutdown; graceful node shutdown is the cluster-level equivalent.
• The Proxmox course part XXXV-Linux-Storage covers cluster shutdown; the kubelet’s inhibitor is the cluster-level equivalent.
• The Docker course part XXX-Docker-Lifecycle covers container shutdown; node-level shutdown is the cluster-level equivalent.

Quiz

Production discipline

• Enable graceful node shutdown on every node. Configure --shutdown-grace-period and --shutdown-grace-period-critical-pods to match your longest Pod shutdown time.
• Verify systemd’s TimeoutStopSec is ≥ the kubelet’s shutdown grace period. Otherwise systemd SIGKILLs the kubelet before it finishes.
• Test graceful shutdown in staging. Trigger a maintenance reboot, verify all Pods terminate cleanly.
• Distinguish drain from shutdown in runbooks. Drain is operator-controlled with PDBs; shutdown is system- controlled without PDBs.
• Plan for forceful shutdown. Hardware failures and kernel panics bypass graceful shutdown; have a recovery procedure for the after-state.