Static Pods and the mirror Pod — bootstrap without an API server

A static Pod is a Pod manifest that the kubelet reads from a directory and runs on the node, without the cluster’s API server. The kubelet creates a mirror Pod for each static Pod so the cluster can see the static Pod’s status. The static Pod mechanism is the cluster’s bootstrap primitive: the control plane’s components (API server, etcd, scheduler, controller-manager) are typically static Pods on the control plane nodes. This lesson walks the mechanism, the use cases, and the operational patterns.

The static Pod mechanism

The kubelet’s --pod-manifest-path (or --config.staticPodPath in the newer config) flag specifies a directory. The kubelet watches the directory for YAML or JSON files. Each file is a Pod manifest; the kubelet runs the Pod on the node.

kubelet --pod-manifest-path=/etc/kubernetes/manifests

# /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - name: kube-apiserver
      image: registry.k8s.io/kube-apiserver:v1.34.0
      command: [kube-apiserver, ...]

The kubelet runs the Pod. The Pod is not in the API server; the API server is not yet running. The Pod runs locally, on the node.

When the API server starts, the kubelet creates a mirror Pod for each static Pod. The mirror Pod is in the API server’s view; it is read-only. The operator can see the static Pod’s status via kubectl get pod -n kube-system, but the operator cannot modify the mirror Pod.

flowchart LR
    A[Static Pod manifest<br/>on disk] -->|kubelet reads| B[kubelet podWorker]
    B -->|runs locally| C[Container on node]
    B -->|creates| D[Mirror Pod<br/>in API server]
    D -->|read-only| E[kubectl get]

The mirror Pod’s name is the manifest’s Pod name with the node name appended: kube-apiserver-cp-1 for a kube-apiserver manifest on node cp-1. The node name is the suffix, not the prefix, which is why every control-plane node’s API server Pod sorts together under kubectl get pods -n kube-system.

The static Pod’s lifecycle

The static Pod’s lifecycle:

1. Bootstrap: the kubelet starts. It reads the pod-manifest-path directory and creates the static Pods.
2. Run: the kubelet runs the static Pods. The Pods are tied to the kubelet’s lifecycle; if the kubelet stops, the Pods stop.
3. Mirror: when the API server is available, the kubelet creates the mirror Pods. The mirror Pods expose the static Pods to the cluster.
4. Update: when the static Pod’s manifest is changed, the kubelet updates the local Pod and the mirror Pod.
5. Delete: when the static Pod’s manifest is removed, the kubelet deletes the local Pod and the mirror Pod.

The cluster’s controllers cannot modify the static Pod through the mirror Pod. The kubectl edit or patch commands fail with “the object is managed by the kubelet.” The only way to modify a static Pod is to edit the manifest file.

The control plane use case

The control plane’s components are static Pods on the control plane nodes. The kubeadm bootstrap creates a /etc/kubernetes/manifests directory with the manifests for:

• kube-apiserver.yaml
• etcd.yaml
• kube-controller-manager.yaml
• kube-scheduler.yaml

The kubelet on the control plane node reads the manifests and runs the components. The cluster’s bootstrap is robust to API server failures because the kubelet does not depend on the API server to run the static Pods.

flowchart LR
    A[kubelet] -->|reads manifests| B[/etc/kubernetes/manifests/]
    B --> C[kube-apiserver]
    B --> D[etcd]
    B --> E[kube-controller-manager]
    B --> F[kube-scheduler]
    C -->|creates| G[API server]
    D -->|creates| H[etcd cluster]
    G --> I[Cluster is operational]

The kubelet starts the API server, which then starts etcd, the controllers, and the scheduler. The cluster is operational.

The critical add-on use case

The cluster’s critical add-ons can be static Pods:

• CNI agent. The CNI plugin’s agent (Cilium, Calico) can be a static Pod on every node. The CNI agent is required for Pod networking; if the CNI agent is down, new Pods cannot get a network.
• Log shipper. A log shipper (Promtail, Fluent Bit) can be a static Pod on every node. The log shipper captures the container logs and forwards them to the log backend.
• Metrics exporter. A node-level metrics exporter (node_exporter) can be a static Pod on every node. The exporter reports the node’s resource usage.

A static Pod is restarted by the kubelet on failure. The add-on is robust to API server failures.

Static Pods versus DaemonSets

Static Pods and DaemonSets both run Pods on every node. The differences:

A DaemonSet is the production-grade mechanism for node-local workloads. The DaemonSet’s controller manages the rolling update, the Pod’s lifecycle, and the Pod’s status. The DaemonSet requires the API server to be running.

A static Pod is the bootstrap mechanism for the control plane and the critical add-ons. The static Pod is managed by the kubelet, not the API server. The static Pod is the only mechanism that survives API server failures.

The static Pod’s manifest

A static Pod’s manifest is a standard Pod YAML. The kubelet does not apply the cluster’s normal validation (no admission controllers, no validation plugins); the manifest is read directly.

The static Pod’s manifest must be valid YAML and must have the required fields:

apiVersion: v1
kind: Pod
metadata:
  name: critical-addon
  namespace: kube-system
spec:
  containers:
    - name: addon
      image: registry.example.com/addon:1.0.0
      resources:
        requests:
          cpu: 100m
          memory: 128Mi

A manifest that is missing the image or the containers fails the kubelet’s validation; the Pod is not started.

The static Pod’s update

A static Pod’s manifest is updated by editing the file. The kubelet watches the directory; when the file’s content changes, the kubelet updates the local Pod and the mirror Pod.

The update is not a rolling update. The kubelet deletes the local Pod and creates a new one. The mirror Pod is updated to reflect the new Pod.

The update is not a graceful rolling update. The critical add-on may be unavailable for the duration of the update. The production rule is to update the static Pod’s manifest atomically (write to a temporary file, rename) and to apply the update during a maintenance window.

The static Pod’s deletion

A static Pod is deleted by removing the manifest file. The kubelet deletes the local Pod and the mirror Pod.

The deletion is not graceful. The kubelet sends SIGTERM to the container, waits for the grace period, then sends SIGKILL. The Pod is gone.

Quiz

Production discipline

• Static Pods are for bootstrapping the cluster. The control plane’s components should be static Pods. The production rule is to version-control the manifests and apply them to every control plane node.
• Critical add-ons can be static Pods. The CNI agent, the log shipper, and the metrics exporter can be static Pods. The static Pod is robust to API server failures.
• General workloads should be DaemonSets. A workload that does not need to survive API server failures should be a DaemonSet. The DaemonSet’s controller manages the lifecycle.
• Audit the static Pod manifests at every release. A static Pod manifest that is missing the image, the resources, or the volumes is a Pod that fails. The audit catches the missing fields.
• Manage the static Pod manifests as code. A static Pod manifest that is edited by hand is a manifest that drifts. The production rule is to store the manifests in a Git repository and apply them with the cluster’s bootstrap automation.