The components on a node — kubelet, runtime, CNI, kube-proxy

A Kubernetes worker node is a Linux host running four long-lived processes that cooperate to run Pods: the kubelet, the container runtime, the CNI plugin, and kube-proxy. Each has a distinct role; each has a distinct failure mode. This lesson walks the four, the data flow between them, and the production patterns for diagnosing each.

The four processes

flowchart LR
    A[kubelet] -->|CRI gRPC| B[container runtime]
    B -->|creates| C[Pod sandbox]
    A -->|CNI gRPC| D[CNI plugin]
    D -->|configures| C
    E[kube-proxy] -->|iptables/IPVS/| F[Service dataplane]
    A -->|node status| G[API server]
    B -->|runtime status| G
    D -->|network status| G
    E -->|rules status| G

The four:

• kubelet: the Kubernetes agent on the node. Registers the node, runs the Pod sync loop, exposes the probe endpoints, reports node status.
• Container runtime: the engine that runs the containers. Implements the Container Runtime Interface (CRI). In production, containerd is the default.
• CNI plugin: the network agent that configures the Pod’s network namespace. Calico, Cilium, Flannel are the common choices.
• kube-proxy: the Service dataplane. Watches the API server for Service and Endpoint changes, configures iptables / IPVS / eBPF rules to implement the Service IP load-balancing.

Plus the system-level components:

• Linux kernel: the cgroups, namespaces, network stack, and seccomp/apparmor primitives that the runtime and the CNI use.
• systemd: the supervisor that starts the kubelet and the runtime.

The kubelet

The kubelet is the cluster’s agent on the node. It is a single binary (kubelet) that runs as a systemd service. The kubelet’s primary responsibilities:

• Register the node with the API server. The kubelet creates a Node object with the node’s addresses and capacity.
• Sync Pods. The kubelet watches the API server for Pods assigned to its node (spec.nodeName == nodeName). For each new Pod, the kubelet calls the CRI to create the Pod sandbox and the containers.
• Run probes. The kubelet runs the Pod’s liveness, readiness, and startup probes. The results feed the Pod’s Status.Conditions.
• Report status. The kubelet periodically sends the node’s status (conditions, capacity, allocatable) and the Pods’ status to the API server. The default sync period is 10 seconds for node status, 1 second for Pod status.

The kubelet is the only Kubernetes component that runs as a systemd service on the node. The runtime, the CNI, and kube-proxy are typically started by the kubelet (via static Pods) or by the node’s bootstrap automation.

The container runtime

The container runtime runs the containers. The runtime implements the CRI (Container Runtime Interface), a gRPC API the kubelet calls to create and manage Pod sandboxes and containers.

The standard runtime on Kubernetes 1.34 is containerd. The kubelet connects to the runtime over a Unix socket (/run/containerd/containerd.sock) and calls the CRI methods.

The runtime’s responsibilities:

• Image pulls. The runtime pulls images from the registry, stores them in the local image cache, and verifies the digest.
• Pod sandbox creation. The runtime creates the Pod sandbox (the network namespace, the cgroup, the filesystem mount) and then starts the containers.
• Container lifecycle. The runtime starts, stops, and restarts containers according to the kubelet’s CRI calls.
• Resource enforcement. The runtime configures the cgroups based on the container’s resource limits and applies the kernel’s cgroup controllers.
• Logging. The runtime writes the container’s stdout and stderr to a log file (typically /var/log/containers/<pod>_<ns>_<container>-<id>.log).

The runtime communicates with the kubelet via the CRI gRPC interface. The kubelet never directly manipulates containers; the runtime is the only process that creates and manages them.

The CNI plugin

The CNI plugin configures the Pod’s network. The kubelet calls the CNI plugin when the Pod’s sandbox is created; the plugin attaches the Pod’s network interface, sets the routes, and adds the IP address.

The CNI plugin’s responsibilities:

• Interface creation. The CNI creates a virtual interface (a veth pair) inside the Pod’s network namespace and connects it to the host’s bridge or routing table.
• IP assignment. The CNI assigns an IP from the Pod CIDR (spec.podCIDR on the node, or a cluster-wide pool). The IP is recorded in the Pod’s Status.PodIP.
• Routes. The CNI adds routes so the Pod can reach the cluster’s other Pods and the outside world.
• Network policy. CNI plugins that enforce NetworkPolicy (Calico, Cilium) inspect the Pod’s selectors and configure iptables / eBPF rules to enforce the policy.

The CNI plugin is a single binary (or a set of binaries) in /opt/cni/bin/. The kubelet calls the CNI through a JSON configuration file (/etc/cni/net.d/<name>.conflist) that lists the plugins to run.

The CNI plugin is started by the kubelet as a child process; it is not a long-running daemon. The CNI binary runs, configures the Pod, and exits. The CNI plugin’s agent (e.g., the Cilium agent, the Calico felix) is a long-running process that maintains the network state.

kube-proxy

kube-proxy is the cluster’s Service dataplane implementation. It is a single binary that runs on every node and watches the API server for Service and Endpoint changes. When a Service or Endpoint changes, kube-proxy updates the node’s iptables (or IPVS, or eBPF) rules to implement the Service IP load-balancing.

# Substitute your own value before running - the Service's ClusterIP,
# from `kubectl get svc`:
SERVICE_CLUSTER_IP=10.96.0.10

iptables -t nat -L -n -v | grep "$SERVICE_CLUSTER_IP"

The output shows the iptables rules kube-proxy has installed. Each Service has a chain of rules that distributes traffic to the Endpoints.

kube-proxy’s responsibilities:

• Watch Services and Endpoints. kube-proxy watches the API server for Service and Endpoint changes.
• Program the dataplane. kube-proxy translates the Service into iptables / IPVS / eBPF rules. The dataplane is the actual load-balancer.
• Maintain the rules. kube-proxy reconciles the dataplane on every change. The reconciliation is a full rebuild of the relevant rules; the existing rules are flushed and replaced.

kube-proxy is started by the kubelet as a static Pod (or by the node’s bootstrap). The binary is the same across deployments; the backend (iptables, IPVS, eBPF) is configurable.

The data flow during Pod creation

The kubelet’s startPod flow:

sequenceDiagram
    autonumber
    participant API as API server
    participant K as kubelet
    participant R as containerd
    participant CNI as CNI plugin
    participant KP as kube-proxy

    K->>API: watch Pods with spec.nodeName=node-1
    API-->>K: Pod billing-1, namespace=prod-app
    K->>R: RunPodSandbox (CRI)
    R->>R: create network namespace
    R->>R: create cgroup
    R-->>K: PodSandbox ID
    K->>R: CreateContainer (CRI)
    R->>R: pull image, set up mounts
    R-->>K: Container ID
    K->>R: StartContainer (CRI)
    R->>R: fork runc, exec container
    R-->>K: Container started
    K->>CNI: ADD (Pod's network namespace)
    CNI->>CNI: attach veth, assign IP, add routes
    CNI-->>K: interface ready
    K->>API: update Pod status (PodIP, Conditions)

The kubelet is the orchestrator. The runtime, the CNI, and the kube-proxy are the workers. The kubelet’s failure mode is the cluster’s failure mode for the node.

The failure modes by component

The diagnostic moves:

1. Run kubectl get pods -n kube-system -o wide | grep <node>. The kube-proxy-<hash>, cni-<daemonset>, and any per-node DaemonSet Pods show whether the node is running the components.
2. Run kubectl describe node <name>. The node’s conditions and the events on the node indicate which component is failing.
3. SSH to the node and check the processes. ps aux | grep -E 'kubelet|containerd|kube-proxy'.
4. Check the logs. Each component has a log file or journal. journalctl -u kubelet, journalctl -u containerd.

Quiz

Production discipline

• The kubelet is the node’s health signal. If the kubelet is failing, the node is failing. Monitor the kubelet’s process on every node.
• The container runtime is the most-failed component. Image pulls, container starts, and runtime errors are the most common node-level failures. The runtime’s metrics and logs are the first thing to check.
• The CNI plugin is the most-misconfigured component. A new CNI plugin that is not configured correctly silently breaks Pod networking. Validate the CNI’s installation at every cluster bootstrap.
• kube-proxy’s iptables backend has a scaling limit. Each Service adds rules; a cluster with thousands of Services can hit the kernel’s iptables rule limit (~65k). Switch to IPVS or eBPF for large clusters.
• Audit the four processes at every node replacement. A new node that joins the cluster without the right CNI, kubelet, or runtime is a node that is failing silently. The bootstrap automation should ensure the four processes are running before the kubelet registers.