The worker node — kubelet, runtime, CNI, CSI, and the service dataplane

A Kubernetes worker node is a Linux host that runs Pods. The “host” itself can be a physical server, a VM, a cloud instance, or a bare-metal node — Kubernetes treats them uniformly through the Node object and the kubelet agent. This lesson walks the components on a worker node and the interfaces between them.

The components on a worker node

flowchart TB
    API[API server] --> K[kubelet]
    K -->|CRI| R[Runtime<br/>containerd]
    K -->|CNI| C[CNI plugin<br/>Cilium / Calico / Flannel]
    K -->|CSI| S[CSI driver<br/>node plugin]
    K --> KP[kube-proxy / dataplane]
    KP --> API
    R --> KRN[Linux kernel<br/>namespaces + cgroups]
    C --> KRN
    S --> KRN

• kubelet — the agent; talks to the API server and the runtime, CNI, and CSI plugins.
• container runtime (containerd / CRI-O) — pulls images, unpacks layers, starts processes in namespaces.
• CNI plugin — allocates Pod IPs, sets up the veth pair, programs routes.
• CSI driver (node plugin) — mounts volumes into Pods.
• kube-proxy (or a CNI-managed dataplane) — programs iptables/IPVS/eBPF rules for Service traffic.

The Linux kernel sits underneath all of it, providing namespaces, cgroups, the network stack, and the storage stack.

kubelet in detail

Kubelet is the agent on every worker. Its responsibilities:

flowchart TB
    API[API server] -->|watch Pods<br/>assigned to this node| K[kubelet]
    K --> P[Pod sync worker]
    K --> H[Node lease renewer]
    K --> L[Probes manager<br/>liveness/readiness/startup]
    K --> V[Volume manager<br/>CSI]
    K --> N[Network plugin manager<br/>CNI]
    K --> R[Runtime manager<br/>CRI]
    P --> API
    H --> API
    L --> API

Each piece is a goroutine. The Pod sync worker is the core loop: it watches the API server for Pods with spec.nodeName == <this-node>, compares to what’s running, and reconciles.

Kubelet’s startup sequence

systemctl status kubelet

1. Read /var/lib/kubelet/config.yaml (configured by kubeadm).
2. Read --node-ip, --node-labels, --register-with- tainted from flags or config.
3. Open gRPC connection to the runtime’s CRI socket.
4. Register the Node object with the API server (if --register-node).
5. Start the lease renewer (every --node-status-update- frequency seconds, default 10s).
6. Start the Pod sync worker.

A kubelet that cannot reach the CRI socket at startup will not register the Node and will not start any Pods. The kubelet logs will show the CRI error.

Kubelet’s reconcile loop for a Pod

When kubelet sees a new Pod assigned to its node, it walks:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server
    participant R as Runtime (CRI)
    participant N as CNI
    participant V as CSI

    K->>API: GET Pod (spec)
    K->>K: Validate (admission policy, security context)
    K->>R: CRI PullImage (if not cached)
    R-->>K: image ready
    K->>V: CSI NodeStageVolume (mount volume on host)
    V-->>K: mount point /var/lib/kubelet/pods/<uid>/volumes/...
    K->>R: CRI CreateSandbox (pause with namespaces)
    K->>N: CNI ADD (allocate Pod IP, veth, routes)
    N-->>K: Pod IP, sandbox ready
    K->>R: CRI CreateContainer (with cgroup limits)
    K->>R: CRI StartContainer
    R-->>K: container PID
    K->>API: PATCH /pods/<name>/status (Running)
    K->>K: Start probe goroutines

The Pod’s status transition through this sequence:

• Pending → ContainerCreating (kubelet sees the Pod)
• Running (all containers started and at least one is ready)

If any step fails, the Pod stays in ContainerCreating and the Pod event names the failed step.

Kubelet’s volume manager

Kubelet manages volumes via the CSI driver (or the in-tree volume plugins, deprecated in 1.26+). For each Pod with volumes:

1. Stage the volume on the host (NodeStageVolume): mount the underlying device (or NFS export, or iSCSI LUN) at a kubelet-managed path.
2. Mount the volume into the Pod’s mount namespace (NodePublishVolume): bind-mount from the staging path to the Pod’s volume mount point.
3. On Pod deletion: reverse the steps (NodeUnpublishVolume, NodeUnstageVolume).

The kubelet’s volume manager tracks mounted volumes and unmounts them when the Pod is deleted. If a volume is stuck mounted, the Pod deletion stalls.

Kubelet’s probe manager

For every Pod, kubelet runs three types of probes:

• Liveness — if it fails, kubelet restarts the container.
• Readiness — if it fails, kubelet removes the Pod from the Service’s EndpointSlices (no traffic).
• Startup — like liveness, but only during the initial start; disabled after success.

Probes are kubelet-side, not API-server-side. A failing readiness probe does not show up as a Pod event — it shows up in kubectl describe pod as Ready: False with a message “Readiness probe failed: …”.

The CNI plugin

The Container Network Interface (CNI) is invoked by kubelet when a Pod is created (and when it is deleted). The CNI plugin:

1. Receives the Pod’s namespace, the container ID, and the network configuration.
2. Allocates a Pod IP from the cluster CIDR.
3. Creates a veth pair: one end in the Pod’s network namespace, one end on the host.
4. Programs routes on the host so the Pod IP is reachable from other nodes.
5. Returns the Pod IP and interface name to kubelet.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant CNI as CNI plugin
    participant KRN as Linux kernel

    K->>CNI: CNI ADD (Pod netns path, container ID, config)
    CNI->>KRN: ip link add veth... (veth pair)
    CNI->>KRN: ip addr add <pod-ip>/<mask> dev eth0 (in Pod netns)
    CNI->>KRN: ip route add <cluster-cidr> via <gateway> (on host)
    CNI->>KRN: iptables/ebpf rules (NetworkPolicy)
    CNI-->>K: success, IP, interface

The CNI plugin is a binary on the host (/opt/cni/bin/...) or a daemon (Cilium agent, Calico Felix). It runs on every worker node. A failure of CNI on a single node stops Pods on that node from getting IPs; a failure of CNI cluster-wide stops every Pod.

Common CNIs:

• Cilium — eBPF-based; default for many production clusters; provides NetworkPolicy via eBPF.
• Calico — supports both iptables and eBPF modes; provides BGP and NetworkPolicy.
• Flannel — simplest overlay (VXLAN); no NetworkPolicy enforcement; rarely used in production.

The CSI driver (node plugin)

The Container Storage Interface (CSI) is invoked by kubelet for mount/unmount (node-side) and by the controller manager for provision/attach (control-plane side). The node plugin:

1. Receives a Pod with a PVC.
2. Calls the storage backend (cloud API, Ceph, NFS) to attach the volume to the node.
3. Mounts the device on the host filesystem.
4. Bind-mounts the device into the Pod’s volume mount point.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant CN as CSI node plugin
    participant CP as CSI controller plugin
    participant ST as Storage backend

    Note over CP,ST: Provisioning (control plane)
    CP->>ST: CreateVolume (in zone matching PVC)
    ST-->>CP: volume handle
    CP-->>API: bind PV to PVC

    Note over K,ST: Attach + Mount (worker)
    K->>CN: NodeStageVolume (mount on host)
    CN->>ST: Attach (e.g., attach EBS volume)
    ST-->>CN: attached
    K->>CN: NodePublishVolume (bind-mount into Pod)
    CN-->>K: mounted

    Note over K,ST: Unmount + Detach (Pod deletion)
    K->>CN: NodeUnpublishVolume
    K->>CN: NodeUnstageVolume
    CN->>ST: Detach

The CSI driver has two components: a controller plugin (runs on the control plane) and a node plugin (runs on every worker). The controller handles provisioning and attaching; the node plugin handles mounting.

kube-proxy / service dataplane

kube-proxy runs on every worker node. Its job is to program the local node’s network so that Service ClusterIPs (virtual IPs) route to the right Pods.

flowchart LR
    S[Service 10.96.0.10:80] -->|clusterIP| KP[kube-proxy on node X]
    KP -->|iptables/IPVS/eBPF rule| P1[Pod A 10.244.1.5]
    KP -->|iptables/IPVS/eBPF rule| P2[Pod B 10.244.2.7]
    KP -->|iptables/IPVS/eBPF rule| P3[Pod C 10.244.3.2]

The kube-proxy modes:

• iptables — default; programs iptables rules per Service. Linear scan as Services grow.
• IPVS — programs IPVS virtual server rules; better scaling for many Services.
• eBPF — replaces iptables with eBPF maps; the modern recommendation (Cilium, Calico eBPF).

A failure of kube-proxy on a node means Service IPs do not work from that node (they may still work from other nodes). A failure of kube-proxy on every node means cluster-wide Service failure.

Many modern CNIs (Cilium) replace kube-proxy with their own dataplane. The function is preserved.

The Linux kernel underneath

The kernel provides:

• Namespaces — netns, mountns, pidns, ipcns, utsns (Part I)
• Cgroups — resource limits (Part I)
• Network stack — veth, bridge, routing tables, iptables, ipvs, eBPF
• Storage stack — mount, bind-mount, filesystem drivers
• Process management — PID 1 semantics, signals

The worker node’s components are all user-space processes that ask the kernel to do work. A kernel regression (e.g., a buggy OverlayFS, a buggy veth driver) surfaces as Pod incidents.

Failure domains of a worker node

The right HA response to a worker-node failure is to let the control plane reschedule the Pods onto other nodes. The Pods will come up; the in-flight requests will fail. Application HA (replicas, anti-affinity) is what makes this tolerable.

How to inspect a worker node

# Substitute your own value before running:
NODE=node-03

# Node status from the API server's view
kubectl get node "$NODE" -o yaml

# From the node itself
systemctl status kubelet containerd
journalctl -u kubelet --since "5 min ago" | head -50
crictl ps
crictl pods
crictl images

# Network state
ip link show
ip route show
iptables -t nat -L KUBE-SERVICES | head

# Volume state
mount | grep kubelet
ls /var/lib/kubelet/pods/

Cross-course references

• The Docker course part XXVII-Docker-Install covers the container runtime installation that kubelet integrates with.
• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives (namespaces, cgroups) the worker composes.
• The Linux course part XIX-Linux-NetFoundations covers the network stack that CNI programs.
• The Observability course part IX-Observability-Exporters covers the node-level metrics the operator monitors.

Quiz

Production discipline

• Treat each worker component as a separate dependency: a Pod may be stuck because of the runtime, the CNI, the CSI, or the registry. Read the Pod event.
• Monitor every worker component independently: kubelet lease, runtime image pull latency, CNI pod creation latency, CSI mount latency, kube-proxy sync latency.
• Run CSI drivers as DaemonSets; ensure the DaemonSet’s nodeSelector and tolerations cover every worker node.
• Plan for node loss as routine. Application HA (replicas, anti-affinity, PDBs) is what makes a single-node loss tolerable.
• Back up node-level configuration (kubelet flags, CNI config, CSI driver version) so a new node can come up identically.