Stopping the control plane — API server, controller manager, scheduler

The Kubernetes control plane is a network of components that read and write to etcd. Before the etcd restore, the operator must stop the writers: API server first, then the controller-manager and scheduler. The order matters because the API server is the only legitimate writer to etcd in a Kubernetes cluster; if it can write during a restore, the new cluster’s state will diverge from the snapshot. This lesson walks the safe shutdown sequence.

The control plane components

flowchart LR
    APIS[kube-apiserver] -->|writes| E[etcd]
    CM[kube-controller-manager] -->|writes via API server| APIS
    SCH[kube-scheduler] -->|writes via API server| APIS
    PXY[kube-proxy] -->|read via API server| APIS
    KL[kubelet] -->|read via API server, write Pod status| APIS
    E -->|watch| APIS
    APIS -->|watch| CM
    APIS -->|watch| SCH
    APIS -->|watch| KL

• kube-apiserver: the cluster’s gateway. The ONLY writer to etcd that the API contract supports.
• kube-controller-manager: a set of controllers (Node, ReplicaSet, Deployment, Endpoint, ServiceAccount, etc.) that watch the API and reconcile via API updates.
• kube-scheduler: watches unscheduled Pods and binds them to nodes.
• kube-proxy: deploys Service IP rules on every node.
• kubelet: per-node; runs Pods and reports status.

The restore only requires stopping the first three (API server, controller-manager, scheduler). The kubelet continues running on every node; its current Pods continue running; it cannot write Pod status updates against an unavailable API server, so it buffers them and retries.

The shutdown order

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as kube-apiserver
    participant CM as kube-controller-manager
    participant SCH as kube-scheduler
    participant E as etcd
    Op->>API: stop (on every control-plane host)
    Op->>CM: stop
    Op->>SCH: stop
    Op->>E: snapshot of broken state (if not done)
    Op->>E: stop (on every etcd host)
    Op->>Op: ready for restore

Stopping the API server first prevents new writes. The controller-manager and scheduler are then stopped because they write through the API server; once the API server is down, they cannot write anyway, but stopping them explicitly ensures no in-flight write is queued.

Stopping on a kubeadm cluster (static pods)

In a kubeadm cluster, the control plane components run as static pods. The kubelet watches the manifest file:

# The static pod manifests:
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml

To stop a static pod, the operator moves the manifest file aside. The kubelet observes the file’s absence, stops the pod, and does not restart it.

# On each control-plane host:
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml \
        /etc/kubernetes/manifests/kube-apiserver.yaml.stopped

sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml \
        /etc/kubernetes/manifests/kube-controller-manager.yaml.stopped

sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml \
        /etc/kubernetes/manifests/kube-scheduler.yaml.stopped

flowchart LR
    Y1[kube-apiserver.yaml] -->|mv .stopped| Z1[kubelet stops pod]
    Y2[kube-controller-manager.yaml] -->|mv .stopped| Z2[kubelet stops pod]
    Y3[kube-scheduler.yaml] -->|mv .stopped| Z3[kubelet stops pod]

Verify with crictl:

sudo crictl pods | grep -E 'apiserver|controller|scheduler'
# Expected: empty after the manifests are moved

Stopping via systemd (non-kubeadm)

For clusters where the components run as systemd services:

sudo systemctl stop kube-apiserver
sudo systemctl stop kube-controller-manager
sudo systemctl stop kube-scheduler

The systemctl stop issues SIGTERM and waits for the configured timeout before SIGKILL.

The kubelet during restore

The kubelet on every worker node continues running:

• Existing Pods continue to run.
• Probes continue to fire and read state.
• The kubelet cannot update Pod status to the API server (it is down); the kubelet buffers updates locally.

After the restore, the API server comes back up; the kubelet reconnects; buffered status updates flush. There is no need to stop the kubelet, but the operator should be aware that workers continue to run.

The danger of quick stops

A systemctl kill or kill -9 on the API server skips graceful shutdown. In-flight writes may be lost. For a restore, this is acceptable because the snapshot is the source of truth, but for an upgrade without snapshot restore, this is the kind of failure that loses state.

$ crictl pods

POD ID              CREATED             STATE    NAME                                          ...
<id-a>              5 minutes ago        Ready    kube-apiserver-cp-1                          ...
<id-b>              5 minutes ago        Ready    kube-controller-manager-cp-1                ...
<id-c>              5 minutes ago        Ready    kube-scheduler-cp-1                         ...

The default --shutdown-delay-duration for kube-apiserver in kubeadm 1.34 is sufficient for the restore preflight. If the API server is processing many requests at the stop moment, raise the delay temporarily.

Pre-restart verification

Before proceeding to the etcd restore, verify the API server is fully down:

# From any host that has kubectl access:
kubectl get nodes
# Expected: error (connection refused or timeout)

# From any host:
curl -k https://10.0.1.10:6443/livez
# Expected: connection refused (the API server is down)

If the API server is still answering, the manifest move has not yet propagated; wait a few seconds.

Worker behaviour during restore

On every worker:

# Worker pods continue running
sudo crictl pods | grep -E 'kube-system|prod' | head
# Expected: existing Pods (the kubelet is still running)

Workers do not need any intervention. Their pods will continue to serve traffic (assuming Service IPs are still configured via kube-proxy, which is largely a passive configuration).

Pre-restore state capture (last chance)

Just before stopping the API server, capture the cluster state one more time:

kubectl get all -A -o yaml > /tmp/final-pre-restore.yaml 2>&1

The capture may be partially incomplete (some controllers are still running) but provides a baseline.

The stop in sequence

Putting it together:

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as kube-apiserver
    participant CM as kube-controller-manager
    participant SCH as kube-scheduler
    participant E as etcd
    Op->>API: kubectl get all -A -o yaml (capture)
    Op->>API: move manifest aside on cp-1, cp-2, cp-3
    API-->>Op: kube-apiserver stopped (crictl pods)
    Op->>CM: move manifest aside on cp-1, cp-2, cp-3
    Op->>SCH: move manifest aside on cp-1, cp-2, cp-3
    Op->>E: snapshot save /backup/post-stop.db (final state)
    Op->>E: stop etcd (next lesson)

The next lesson covers stopping etcd itself and running the restore.

Quiz

Production discipline

• API server first. The only legitimate writer to etcd; stop it first to prevent in-flight writes.
• Stop controller-manager and scheduler after the API server. They write through the API server, so stopping them after is a confirmation.
• Workers do not need to stop. They buffer status updates; the cluster’s data plane continues.
• Capture state before stopping. The last chance to record the cluster’s current state is just before the API server stop.
• Verify the stop. curl healthz and crictl pods to confirm the API server is down.

Stopping the control plane is a destructive sequence; the order ensures the cluster state at the moment of restore is exactly what the snapshot captured.