Node-level kernel and network — the host underneath

Below the kubelet is a host that Kubernetes assumes rather than manages: a kernel version, a set of interfaces and routes, a conntrack table, and a cgroup hierarchy. When one of those is wrong the node usually stays Ready and lies — a kubelet and a containerd configured with different cgroup drivers each build their own hierarchy, so the memory limits you set are written where nothing enforces them. Nothing in the API server describes this layer, which is why the evidence comes from uname, ip, journalctl, and the node’s own configuration files.

The node’s host

A node’s host is the Linux kernel, the network interfaces, the routes, and the file system. The Kubernetes components (kubelet, runtime, CNI) run on top of the host.

flowchart TD
    A[kubelet] --> B[containerd]
    B --> C[Linux kernel]
    C --> D[Network interfaces]
    C --> E[File system]
    C --> F[cgroups]

The host is the node’s foundation.

The diagnostic

The canonical diagnostic:

# 1. Check the kernel
uname -a
cat /etc/os-release

# 2. Check the network interfaces
ip addr show
ip route show

# 3. Check the file system
df -h
mount | grep -i overlay

# 4. Check the kernel logs
dmesg | tail -200

# 5. Check the cgroup driver
cat /proc/self/cgroup | head -10

# 6. Check the kubelet's host view
journalctl -u kubelet | grep -i host

The diagnostic is the kernel, the network, the file system, and the kubelet’s host view.

Common failures

• Kernel version mismatch. The kernel version is too old for the kubelet. The remediation is to upgrade the kernel.
• Network interface down. The node’s primary network interface is down. The remediation is to bring the interface up.
• Route missing. The node’s route to the cluster network is missing. The remediation is to add the route.
• File system full. The node’s root file system is full. The remediation is to clean up the disk.
• cgroup driver mismatch. The kubelet’s cgroup driver does not match the runtime’s. The remediation is to align the drivers.

flowchart TD
    A[Node failure] --> B{Kernel version OK?}
    B -->|No| C[Upgrade the kernel]
    B -->|Yes| D{Network OK?}
    D -->|No| E[Fix the network]
    D---|Yes| F{File system OK?}
    F -->|No| G[Clean up the disk]
    F -->|Yes| H{cgroup driver OK?}
    H -->|No| I[Align the drivers]
    H -->|Yes| J[Unknown]

The cgroup driver

The cgroup driver is the cluster’s resource isolation layer. The kubelet and the runtime must use the same cgroup driver (systemd or cgroupfs).

# Check the kubelet's cgroup driver
cat /var/lib/kubelet/config.yaml | grep cgroupDriver

# Check the containerd's cgroup driver
cat /etc/containerd/config.toml | grep SystemdCgroup

The cgroup driver must match.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
IFACE=ens18
ROUTE_CIDR=198.51.100.0/24
GATEWAY=192.0.2.1

# Option 1: Upgrade the kernel
sudo apt upgrade linux-image-generic

# Option 2: Bring the network interface up
sudo ip link set "$IFACE" up

# Option 3: Add a route
sudo ip route add "$ROUTE_CIDR" via "$GATEWAY"

# Option 4: Clean up the disk
sudo crictl rmi --prune

# Option 5: Align the cgroup drivers
# Edit kubelet config and containerd config
sudo systemctl restart kubelet
sudo systemctl restart containerd

The remediation is the host recovery.

Production discipline

A node-level failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the node’s host, identify the cause, apply the remediation. The host is the node’s foundation; the remediation is the host recovery.

Quiz