Pod lifecycle from the kubelet's view — admit, sync, run, terminate

The kubelet runs a per-Pod state machine. The state machine handles admission, container creation, network setup, volume mounting, probe execution, and termination. Each step has a failure mode; each failure mode has a diagnostic. This lesson walks the state machine and the operational patterns for stuck Pods.

The kubelet’s per-Pod state machine

The kubelet’s podWorker runs a per-Pod goroutine. The goroutine’s state machine:

stateDiagram-v2
    [*] --> Sync: Pod added to sync loop
    Sync --> Admit: kubelet admits Pod
    Admit --> CreateSandbox: CRI RunPodSandbox
    CreateSandbox --> SetupNetwork: CNI ADD
    SetupNetwork --> MountVolumes: CSI mount
    MountVolumes --> CreateContainers: CRI CreateContainer
    CreateContainers --> StartContainers: CRI StartContainer
    StartContainers --> Running: postStart hook
    Running --> Running: probes pass
    Running --> Terminating: API server delete
    Terminating --> PreStop: preStop hook
    PreStop --> StopContainers: SIGTERM
    StopContainers --> RemoveContainers: SIGKILL after grace
    RemoveContainers --> CleanupNetwork: CNI DEL
    CleanupNetwork --> CleanupVolumes: CSI unmount
    CleanupVolumes --> [*]

The state machine is the kubelet’s view of the Pod. The API server’s view is the Pod’s Status.Phase; the two are correlated but not identical.

The admit step

The kubelet’s admission step is the first decision. The kubelet checks:

• The Pod’s resource requests against the node’s available capacity.
• The Pod’s tolerations against the node’s taints.
• The Pod’s node selector against the node’s labels.
• The Pod’s affinity rules against the cluster’s state.
• The Pod’s owner references and lifecycle.

The admission step is where the kubelet decides whether the Pod can run on the node. The decision is recorded in the Pod’s status.

The admission failure modes:

The kubelet’s admission is the cluster’s filter plugin. The Pod is admitted once; the kubelet does not re-admit on every sync cycle.

The create-sandbox step

The kubelet calls the CRI to create the Pod’s sandbox. The sandbox is the network namespace, the cgroup, and the filesystem mount. The runtime creates these and returns the sandbox ID.

The failure modes:

A Pod stuck in ContainerCreating is most likely stuck in the create-sandbox step. The kubectl describe pod shows the kubelet’s error.

The setup-network step

The kubelet calls the CNI plugin to configure the Pod’s network. The CNI adds the veth pair, the IP address, and the routes.

The failure modes:

A Pod stuck in ContainerCreating with a network-related event is stuck in the setup-network step. The fix is to investigate the CNI plugin.

The mount-volumes step

The kubelet calls the CSI driver to mount the Pod’s volumes. The CSI driver attaches the volume to the node and mounts the filesystem.

The failure modes:

A Pod stuck in ContainerCreating with a volume-related event is stuck in the mount-volumes step.

The create-containers step

The kubelet calls the CRI to create the Pod’s containers. The runtime pulls the image, sets up the container’s filesystem, and creates the container.

The failure modes:

A Pod stuck in ContainerCreating with a container creation-related event is stuck in the create-containers step.

The start-containers step

The kubelet calls the CRI to start the containers. The runtime starts the container’s process; the process performs its initial work.

The failure modes:

A Pod that restarts repeatedly is in the start-containers step. The kubectl describe pod shows the container’s exit status and the reason.

The running state

The Pod is running. The kubelet runs the probes and updates the status.

The failure modes:

A Pod that is running but not serving traffic is in the running state with a failed readiness probe. The fix is to investigate the application.

The terminating state

The kubelet receives the API server’s delete request. The Pod’s containers are terminated gracefully.

sequenceDiagram
    autonumber
    participant API as API server
    participant K as kubelet
    participant C as Container
    participant CNI as CNI

    API->>K: delete Pod
    K->>C: preStop hook
    K->>C: SIGTERM
    C->>C: ... shutdown
    K->>C: SIGKILL (after grace period)
    K->>CNI: DEL
    K->>API: status update: Succeeded or Failed

The kubelet’s termination grace is the Pod’s terminationGracePeriodSeconds (default 30s). The kubelet sends SIGTERM, waits for the grace period, then sends SIGKILL.

The failure modes:

A Pod stuck in Terminating is a Pod whose cleanup is failing. The fix is to investigate the log path, the volume unmount, or the network namespace cleanup.

The “could not be evicted” state

A Pod that is being terminated but cannot be evicted is in a special state. The kubelet cannot remove the Pod’s sandbox because:

• Another process is using the network namespace.
• The volume unmount is hanging.
• The cgroup is busy.

The kubelet logs the error and retries. The fix is to investigate the kubelet’s logs.

The mirror Pod

The kubelet creates a mirror Pod for each static Pod. The mirror Pod is in the API server; the static Pod is on the node. The mirror Pod is read-only.

The kubelet syncs the mirror Pod’s status with the static Pod’s status. The operator can see the static Pod’s status via kubectl get pod -n kube-system | grep <name>.

Quiz

Production discipline

• A Pod stuck in ContainerCreating is the kubelet’s state machine. The kubectl describe pod shows the kubelet’s error. The fix is to read the error.
• A Pod stuck in Terminating is the kubelet’s cleanup. The kubelet’s logs show the cleanup error. The fix is to investigate the log path, the volume unmount, or the network namespace.
• The kubelet’s state machine is the cluster’s ground truth. The Pod’s Status.Phase is the cluster’s view; the kubelet’s state machine is the node’s view. The two can diverge during a long- running cleanup.
• Audit the kubelet’s logs at every release. A kubelet that is silently failing is a node that is silently failing. The audit catches the silence.
• Static Pods are the cluster’s fallback. The cluster’s critical add-ons should be static Pods or DaemonSets that the kubelet can start.