Controller manager overview — the cluster's automation engine

The kube-controller-manager is the cluster’s automation engine. It runs dozens of controllers, each watching the API server for state changes and reconciling them toward the desired state. This lesson walks the manager’s role, the controllers it runs, the leader election pattern, and the metrics operators monitor.

The controller manager in one sentence

kube-controller-manager runs the cluster’s built-in controllers (Deployment, ReplicaSet, Node, Endpoint, ServiceAccount, etc.) in a single binary, with leader election so multiple instances coordinate.

flowchart LR
    AS[API server] -->|watch| CM[Controller manager]
    CM -->|node-controller| NC[Node reconciliation]
    CM -->|deployment-controller| DC[Deployment reconciliation]
    CM -->|endpoint-controller| EC[EndpointSlice reconciliation]
    CM -->|serviceaccount-controller| SC[ServiceAccount tokens]
    CM -->|token-controller| TC[Bootstrap tokens]
    CM -->|replicaset-controller| RC[ReplicaSet reconciliation]
    CM -->|statefulset-controller| SSC[StatefulSet reconciliation]
    CM -->|cronjob-controller| CJ[CronJob reconciliation]
    CM -->|clusterrole-aggregation| CRA[RBAC reconciliation]

Each controller is a long-running goroutine that watches state and reconciles.

What “reconcile” means

A controller’s loop:

flowchart LR
    W[Watch API server] -->|events| L[List current state]
    L --> D[Diff against desired state]
    D -->|drift detected| A[Apply correction]
    A --> W
    D -->|no drift| W

The controller:

1. Watches the API server for state changes.
2. Lists the current state (periodically as a backstop).
3. Computes the diff against desired state.
4. Applies corrections (writes back to the API server).
5. Loops to step 1.

The cycle repeats; the cluster converges to desired state.

The built-in controllers

The kube-controller-manager in 1.34 runs:

• Deployment controller. Reconciles Deployment objects to desired replicas; rolls out ReplicaSets.
• ReplicaSet controller. Reconciles ReplicaSets to desired replicas; creates/deletes Pods.
• StatefulSet controller. Reconciles StatefulSets; ordered Pod management.
• DaemonSet controller. Reconciles DaemonSets; one Pod per matching node.
• Job controller. Reconciles Jobs; runs to completion.
• CronJob controller. Reconciles CronJobs; creates Jobs on schedule.
• Node controller. Reconciles Node objects; updates status, evicts on NotReady.
• EndpointSlice controller. Reconciles EndpointSlices from Service selectors.
• Service controller. Reconciles Services on cloud providers (LoadBalancer).
• ServiceAccount controller. Reconciles SA objects; ensures each SA has a Secret with the SA token.
• Token controller. Reconciles bootstrap tokens; binds Roles to bootstrap tokens.
• ClusterRole / Role / ClusterRoleBinding controller. Reconciles RBAC objects; aggregator.
• PV controller. Reconciles PV / PVC binding; reclaims PVs.
• TTL controller. Cleans up finished Jobs and Pods.
• Garbage collector. Removes dependent objects when their owner is deleted.
• CSR (CertificateSigningRequest) controller. Signs CSR for node bootstrapping.
• HPAController (if HPA is enabled).
• Node lifecycle controller.
• Lease controller. Renews system leases.
• ResourceQuota controller. Updates ResourceQuota status.
• Namespace controller. Cleans up deleted namespaces.
• PersistentVolumeClaim protection controller.
• StatefulSetOrdinals controller.

A complete list spans 30+ controllers.

The leader election

kube-controller-manager supports running multiple instances. Only one is active at a time:

flowchart LR
    Inst1[Instance 1] -->|lease| LE[Lease object]
    Inst2[Instance 2] -->|lease| LE
    Inst1 -->|leader| ACTIVE[Active: reconcile]
    Inst2 -->|standby| STANDBY[Standby: watch leader]
    LE -.->|renewed| Inst1
    LE -.->|expires| Inst2

The standby instance watches the lease; if the leader fails to renew (e.g., crash), the standby takes over.

# Default lease object for controller manager
kube-system/kube-controller-manager

Production clusters typically run 2 instances: one active, one standby. The standby consumes minimal resources.

The controller’s API write

A controller that writes back to the API server follows the standard write path:

sequenceDiagram
    autonumber
    participant C as Controller
    participant AS as API server
    participant E as etcd
    C->>AS: PUT /api/v1/.../pods/...
    AS->>AS: authn, authz, admission
    AS->>E: txn
    E-->>AS: committed
    AS-->>C: 200 OK
    C->>C: continue watching

The controller makes the API call just like a user; admission policies apply; RBAC checks the controller’s ServiceAccount. Controllers that lack RBAC for an action fail with 403.

The metrics

The controller-manager exposes Prometheus metrics:

# Work-queue depth
workqueue_depth{controller="deployment", name="default"}

# Controller reconcile counts
controller_reconcile_total{controller="deployment"}

# Reconcile latency
controller_reconcile_duration_seconds{controller="deployment"}

# Active workers
controller_active_workers{controller="node"}

The depth of a work queue is a key health signal: a growing depth means the controller is falling behind.

$ kubectl get componentstatuses controller-manager

controller-manager Healthy   ok

The failure modes

The leader-on vs leader-off configurations

For control planes, leader election is enabled by default; multiple instances share the active role.

For edge or testing setups, leader election can be disabled:

kube-controller-manager --leader-elect=false

A single-instance cluster without leader election runs without HA. Production should use multiple instances.

The controller manager’s resource use

A typical kube-controller-manager on a small cluster:

• CPU: 100-200 mCPU at idle; 500+ mCPU during reconciler storms.
• Memory: 256-512 MiB baseline; can grow with cache.

A controller-manager on a busy cluster (1000+ Pods):

• CPU: 1-2 vCPU.
• Memory: 1-2 GiB.

Profiling specific controllers (kube-controller-manager

• flag --profiling=true) is the standard debugging.

Quiz

Production discipline

• Run 2 instances for HA. A single instance is a single point of failure.
• Monitor work-queue depth. A growing queue for a specific controller indicates that controller is falling behind.
• Watch leader election. Many leader changes correlate with instability.
• Profile controllers when their reconcile is slow. Most slow-controller cases are custom or third-party controllers.
• Reconcile latency p99 should be seconds, not minutes. A regression indicates a slow controller or a slow API.

kube-controller-manager is the cluster’s automation engine; operating it well is keeping the cluster’s desired-state convergence alive.