The controller manager and built-in controllers

The kube-controller-manager is one binary, but inside it runs dozens of controllers — each a small reconcile loop watching a specific resource type. This lesson covers what each major controller does, how they coordinate, and the operational patterns that arise when one of them fails.

The controller manager in one sentence

kube-controller-manager runs the core controllers. Each controller observes its resource type via the API server’s watch API, compares the observed state to the desired state, and acts to close the gap.

flowchart TB
    KCM[kube-controller-manager process]
    KCM --> D[Deployment controller]
    KCM --> R[ReplicaSet controller]
    KCM --> ST[StatefulSet controller]
    KCM --> DM[DaemonSet controller]
    KCM --> J[Job controller]
    KCM --> S[Service controller]
    KCM --> E[Endpoints controller]
    KCM --> N[Node controller]
    KCM --> NS[Namespace controller]
    KCM --> SA[ServiceAccount controller]
    KCM --> PV[PersistentVolume controller]
    KCM --> RQ[ResourceQuota controller]
    KCM --> NSR[Namespace termination]
    KCM --> GC[Garbage collector]
    KCM --> CS[ServiceAccount token controller]

Each controller is a goroutine inside the kube-controller-manager process. They share the API server client and the same leader election lease.

The major controllers

Deployment controller

Reconciles Deployment objects. Its job:

• Maintain the desired number of ReplicaSets (usually 1 active
  • previous versions during rollout)
• Scale the active ReplicaSet to the desired spec.replicas
• Roll over to a new ReplicaSet when the Pod template changes
• Track rollout status via status.conditions

sequenceDiagram
    autonumber
    participant D as Deployment controller
    participant API as API server
    participant R as ReplicaSet controller

    loop every N seconds
        D->>API: list Deployments
        D->>D: For each Deployment: reconcile
        alt Pod template changed
            D->>API: Create new ReplicaSet
            D->>API: Scale old ReplicaSet to 0
        else Pod template unchanged
            D->>API: Scale active ReplicaSet to spec.replicas
        end
    end
    R->>API: list ReplicaSets
    R->>API: Create/scale Pods

ReplicaSet controller

Reconciles ReplicaSet objects. Its job:

• Maintain the desired number of Pods (spec.replicas)
• Create new Pods when count is below desired
• Delete excess Pods when count is above desired
• Use owner references to clean up Pods when the ReplicaSet is deleted

StatefulSet controller

Reconciles StatefulSet objects. Its job:

• Maintain Pods with stable, ordered identity
• Create Pods in order (0, 1, 2, …) on initial deploy
• Delete Pods in reverse order
• Maintain PVCs for each Pod via volumeClaimTemplates
• Update Pods in-place or in rolling fashion

StatefulSet is more complex than Deployment because the order matters: Pod-0 must be Ready before Pod-1 starts.

DaemonSet controller

Reconciles DaemonSet objects. Its job:

• Ensure one Pod per node (subject to nodeSelector, taints, scheduling constraints)
• Add Pods to new nodes automatically
• Remove Pods from cordoned or removed nodes

Job controller

Reconciles Job objects. Its job:

• Create Pods to complete the work (spec.completions)
• Track completions and parallelism
• Clean up Pods after ttlSecondsAfterFinished

CronJob controller

Reconciles CronJob objects. Its job:

• Create Jobs on the cron schedule
• Suspend or resume per schedule
• Manage concurrency policy (Allow, Forbid, Replace)

Service controller

Reconciles Service objects. Its job:

• Watch Services of type LoadBalancer
• Trigger cloud controller manager to provision the cloud LB
• Update status.loadBalancer.ingress with the LB address

Endpoints / EndpointSlice controller

Reconciles EndpointSlice objects (Endpoints in older versions). Its job:

• Watch Pods matching each Service’s selector
• Create/update EndpointSlices with the matching Pod IPs
• Remove Pods from the slice when they are deleted or stop matching

Node controller

Reconciles Node objects. Its job:

• Monitor node lease renewals
• Mark nodes NotReady after lease expiry
• Evict Pods from NotReady nodes
• Garbage-collect unreachable nodes after a long grace period

Namespace controller

Reconciles Namespace objects. Its job:

• Maintain the Namespace’s status.phase (Active, Terminating)
• Clean up resources in a Namespace on deletion
• Manage the deletion timestamp and finalizers

ServiceAccount controller

Reconciles ServiceAccount objects. Its job:

• Create default ServiceAccount in every new Namespace
• Manage projected token signing (1.24+ uses the TokenRequest API; legacy long-lived tokens are deprecated)

ServiceAccount token controller (kube-controller-manager only)

A separate controller inside kube-controller-manager that issues tokens for legacy ServiceAccounts that do not yet use projected tokens. As of 1.24+, this is largely vestigial; modern ServiceAccounts use projected tokens.

PersistentVolume controller

Reconciles PersistentVolume and PersistentVolumeClaim objects. Its job:

• Bind PVs to PVCs (static provisioning)
• Trigger CSI provisioning for dynamic volumes
• Recycle/Retain/Delete PVs when their PVCs are deleted
• Update PV phase (Available, Bound, Released, Failed)

ResourceQuota controller

Reconciles ResourceQuota objects. Its job:

• Count resource usage in the Namespace
• Reject Pods that would exceed the quota
• Update status.used on the quota

Namespace termination controller

A separate sub-controller that handles Namespace deletion. Its job:

• Wait for all resources in the Namespace to be deleted
• Update Namespace status.phase to Terminating
• Finalize the deletion (remove the Namespace object itself)

Garbage collector

A controller that runs the owner reference and finalizer logic. Its job:

• Delete objects whose owner is gone (cascading delete)
• Wait for finalizers to be removed before deleting

Leader election

The controller manager uses leader election. Only the leader actively reconciles; the other replicas stand by. The leader holds a Lease object (kube-controller-manager Lease in kube-system namespace).

sequenceDiagram
    autonumber
    participant A as Instance A (leader)
    participant B as Instance B (standby)
    participant L as Lease (kube-system)

    A->>L: acquire (write identity, TTL=15s)
    loop every 5s
        A->>L: renew (extend TTL)
    end
    Note over A,B: A dies
    B->>L: after 15s, attempt acquire
    B->>B: become leader
    Note over B: reconcile loops run

If the leader fails:

1. The Lease TTL expires (15s default).
2. Standby replicas detect the expiry and compete to acquire.
3. One wins and becomes the new leader.
4. The other replicas continue to stand by.

Failover typically takes 15-30 seconds. During this time, reconciliation is stalled but the cluster’s state does not change — controllers are level-triggered, so the next reconcile cycle after failover re-evaluates the same objects.

How the controllers coordinate

Controllers do not call each other directly. They coordinate through the API server:

• Deployment controller creates a ReplicaSet (via API server).
• ReplicaSet controller watches ReplicaSets (via API server watch) and creates Pods.
• Pod’s owner reference points to the ReplicaSet.
• The garbage collector walks the owner reference graph on deletion.

This loose coupling means a controller failure does not break the controllers above or below it in the dependency chain — they will catch up on the next reconcile cycle.

How to inspect controller-manager

kubectl get pods -n kube-system -l component=kube-controller-manager

# From a control-plane node
journalctl -u kube-controller-manager --since "10 min ago"

I0815 12:01:01 deployment_controller.go:...] Successfully assigned 5 replicas to ReplicaSet web-7c8
I0815 12:01:01 garbagecollector.go:...] removing Pod web-7c8.abc (owner gone)

# Watch the leader lease
kubectl get lease kube-controller-manager -n kube-system -o yaml -w

The Lease object shows:

• spec.holderIdentity — which replica is leader
• spec.renewTime — when the lease was last renewed
• spec.leaseDurationSeconds — TTL

If the holderIdentity changes frequently, the cluster has a leader election problem (network blips, clock skew, or resource starvation).

Common controller-manager failure modes

A sustained “all controllers stalled” condition almost always means:

• Leader election Lease is missing (deleted by an operator script)
• Network partition prevents Lease renewal
• API server is unreachable (controllers cannot talk to it)
• Controller-manager process is OOMKilled or crashed

The fix is to restore the Lease, the network, the API server, or restart the controller-manager.

Controller metrics

The controller-manager exposes metrics on its /metrics endpoint:

• workqueue_adds_total — work queue depth (rate of objects added)
• workqueue_depth — current work queue depth
• workqueue_queue_duration_seconds — how long objects wait in the queue
• controller_runtime_seconds — time per reconcile
• controller_max_queue_size — backpressure threshold

A growing work queue depth is the early signal that a controller is falling behind.

Cross-course references

• The Linux course part VI-Linux-Processes covers the process management primitives; the controller manager runs as multiple goroutines inside one process.
• The Observability course part CIX-Observability-InvestigationWorkflows covers investigation methodology that applies to “which controller is stuck” diagnosis.
• The Linux course part XXIV-Linux-Time covers chrony — leader election depends on clocks within tolerance.
• The Docker course part XXIX-Docker-Build covers the image-pull-then-run pattern that controllers indirectly rely on.

Quiz

Production discipline

• Treat each controller as a separate control loop with its own failure mode. A “the cluster is broken” report often traces to a specific controller stalled.
• Monitor work queue depth and controller reconcile duration. Growing queues predict stalled reconciliation.
• Run 2 controller-manager replicas for fast failover.
• Add alerts on system Lease objects in kube-system. Loss of a Lease stalls the corresponding component.
• Back up etcd (Part LXVI) before any maintenance that touches kube-system namespaces; the system objects are cluster state.