The kubelet — gRPC client, Pod sync loop, status reporter

The kubelet is the cluster’s agent on the worker node. It is the only Kubernetes component that runs as a systemd service on the node (the others are static Pods or are managed by the cluster’s bootstrap). The kubelet’s job is to translate the cluster’s desired state (the Pods assigned to the node) into the local state (the containers running on the node) and to report the node’s status back to the cluster. This lesson walks the kubelet at the architectural level.

The kubelet’s role

The kubelet is a single binary (kubelet) that runs as a systemd service. Its primary responsibilities:

• Register the node with the API server.
• Sync Pods from the API server. The kubelet watches the API server for Pods with spec.nodeName == nodeName.
• Call the runtime to create and manage Pods.
• Run probes to determine the Pod’s health.
• Report status to the API server.

The kubelet is the only Kubernetes component that knows about the node’s local state. The API server, the scheduler, the controllers — they all see the cluster’s desired state, but only the kubelet knows whether the Pod is actually running.

The kubelet’s subsystems

The kubelet is a multi-subsystem binary. The primary subsystems:

flowchart LR
    A[kubelet main] --> B[Pod sync loop]
    A --> C[Status update loop]
    A --> D[Probe executor]
    A --> E[Volume manager]
    A --> F[Runtime manager]
    A --> G[Image manager]
    A --> H[GC manager]
    A --> I[Server<br/>10250/healthz/metrics/pods]
    A --> J[Static Pod manager]

• Pod sync loop: watches the API server for Pods assigned to the node and reconciles the local state.
• Status update loop: sends the node’s status and the Pods’ status to the API server.
• Probe executor: runs the Pods’ liveness, readiness, and startup probes.
• Volume manager: attaches and mounts the Pods’ volumes.
• Runtime manager: calls the CRI to start and stop Pods.
• Image manager: pulls and garbage-collects images.
• GC manager: cleans up unused containers and volumes.
• Server: the HTTP server on port 10250 (the kubelet API).
• Static Pod manager: watches a directory for static Pod manifests and creates mirror Pods.

The subsystems run concurrently. The kubelet is a goroutine-heavy binary; each subsystem runs in its own goroutine.

The Pod sync loop

The Pod sync loop is the kubelet’s primary work. The loop runs every 1 second (the default sync frequency):

flowchart TD
    A[Sync loop tick] --> B[List Pods from API server<br/>with spec.nodeName=node-1]
    B --> C[Diff with desired Pod set]
    C --> D{Pod in desired<br/>but not running?}
    D -->|Yes| E[Start Pod via CRI]
    D -->|No| F{Pod running<br/>but not desired?}
    F -->|Yes| G[Delete Pod via CRI]
    F -->|No| H{Pod running and desired<br/>but spec changed?}
    H -->|Yes| I[Update Pod via CRI]
    H -->|No| J[Pod in sync]

The loop is a reconciliation: the kubelet’s current state is the local Pods; the desired state is the API server’s Pods. The loop reconciles the two.

The loop’s frequency is the kubelet’s --sync-frequency flag (default 10s in older versions, 1s in newer versions). The lower the frequency, the faster the kubelet reacts to changes; the higher the cost in API server traffic.

The gRPC clients

The kubelet calls external services via gRPC:

• CRI (Container Runtime Interface): gRPC to the container runtime. The kubelet calls RunPodSandbox, CreateContainer, StartContainer, StopPodSandbox, RemovePodSandbox, ContainerStatus, PodSandboxStatus. The default socket is /run/containerd/containerd.sock for containerd.
• CNI (Container Network Interface): gRPC to the CNI plugin. The kubelet calls the CNI plugin as a child process with a JSON config; the CNI plugin configures the Pod’s network namespace.
• Device plugin: gRPC to the device plugin. The kubelet exposes the Pod’s resources to the device plugin and registers the device plugin’s resources.
• CSI (Container Storage Interface): gRPC to the CSI driver. The kubelet calls the CSI driver to mount and unmount volumes.

The CRI is the core. The other interfaces are auxiliary.

The status update loop

The status update loop runs every 10 seconds (the --node-status-update-frequency flag). The loop:

1. Reads the node’s local state (CPU, memory, disks).
2. Builds the NodeStatus object.
3. Compares with the previous status.
4. If different, sends a PATCH to the API server.
5. If the same, sends a heartbeat (a no-op PATCH).

The Pods’ status is updated every 1 second (the --sync-frequency flag). The Pods’ status is the container state, the probe state, and the conditions.

The kubelet’s status update is the cluster’s view of the node. A kubelet that has not updated for 40s is considered unreachable by the node controller.

The probe executor

The kubelet runs the Pods’ probes. The probe types:

• Liveness probe: determines if the container is alive. If the probe fails, the kubelet restarts the container.
• Readiness probe: determines if the container is ready to serve traffic. If the probe fails, the kubelet removes the Pod from the Service’s EndpointSlice.
• Startup probe: determines if the container has started. The probe runs only at startup; the liveness and readiness probes do not run until the startup probe succeeds.

The probe executors are HTTP, TCP, and gRPC. The kubelet calls the probe’s endpoint and checks the response. The probe result is recorded in the Pod’s Status.Conditions.

The static Pod mechanism

The kubelet can run static Pods. A static Pod is a Pod manifest that the kubelet reads from a directory (/etc/kubernetes/manifests by default) and runs on the node. The static Pod is not in the API server; it is managed by the kubelet.

The kubelet creates a mirror Pod for each static Pod. The mirror Pod is in the API server; it tracks the static Pod’s status. The mirror Pod is read-only; the operator cannot modify it.

Use cases:

• Cluster bootstrap. The control plane’s components (kube-apiserver, etcd, kube-controller-manager, kube-scheduler) are typically run as static Pods on the control plane nodes.
• Critical system Pods. The CNI agent, the log shipper, or the metrics exporter can be a static Pod on every node.
• Local development. A development cluster can run Pods without an API server.

The static Pod’s manifest can be a YAML file; the kubelet watches the directory. Adding a file creates a static Pod; removing a file deletes it.

The kubelet API

The kubelet exposes an HTTP server on port 10250. The endpoints:

• /healthz: the kubelet’s liveness.
• /pods: the Pods running on the node.
• /runningpods: the Pods currently running.
• /metrics: the kubelet’s Prometheus metrics.
• /exec: the kubelet’s exec endpoint (used by kubectl exec).
• /run: the kubelet’s run endpoint (used by kubectl run).
• /portForward: the kubelet’s port-forward endpoint.
• /logs: the kubelet’s log endpoint (used by kubectl logs).

The kubelet API is the primary way the operator interacts with the node. kubectl exec, kubectl logs, kubectl port-forward all go through the kubelet API.

The kubelet API is by default open to anyone who can reach the node’s port 10250. Production clusters use --anonymous-auth=false and --authorization-mode=Webhook to restrict access.

The kubelet’s failure modes

The kubelet’s primary failure modes:

The diagnostic moves:

1. Run kubectl get node <name>. The Ready condition shows the kubelet’s status.
2. Run kubectl describe node <name>. The events on the node indicate the kubelet’s failures.
3. SSH to the node and check the kubelet’s logs. journalctl -u kubelet.
4. Check the kubelet’s metrics. The kubelet_* metrics on the cluster’s Prometheus are the primary signals.

The kubelet’s flags

The kubelet’s flags control every aspect of its behaviour. The mandatory flags:

• --kubeconfig: the path to the kubelet’s kubeconfig.
• --config: the path to the kubelet’s configuration file (the newer recommended mechanism).
• --root-dir: the data directory.
• --cert-dir: the certificate directory.
• --node-ip: the node’s IP address.
• --register-node: whether to register the node.
• --hostname-override: the node’s hostname.

The optional flags tune the kubelet’s behaviour:

• --max-pods: the maximum number of Pods.
• --sync-frequency: the Pod sync loop frequency.
• --node-status-update-frequency: the status update frequency.
• --image-gc-high-threshold: the image GC threshold.
• --eviction-hard: the hard eviction threshold.

Quiz

Production discipline

• The kubelet is the node’s health signal. A kubelet that is failing is a node that is failing. Monitor the kubelet’s process on every node.
• The kubelet’s API is the operator’s primary tool. The kubelet’s endpoints (/pods, /metrics, /logs) are the primary sources of node-level diagnostics. Restrict access to the kubelet’s API in production.
• Static Pods are the cluster’s fallback. The cluster’s critical add-ons (CNI, log shipper, monitoring) should be static Pods or DaemonSets that the kubelet can start.
• The kubelet’s certificate must be rotated. A kubelet whose certificate has expired is a node that is failing. Rotate at 75% of the TTL.
• Audit the kubelet’s flags at every node repave. A new node that joins the cluster with the wrong kubelet configuration is a node that is failing silently. Validate the kubelet’s flags at bootstrap.