The control loop — observe, diff, act

The control loop is the universal pattern that runs every controller in Kubernetes: Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, EndpointSlice, ServiceAccount — and every custom Operator. This lesson dissects the loop, identifies each component, and explains the failure modes that arise when it stops working.

The loop in one sentence

A controller observes the cluster’s current state, diffs it against the desired state, and acts to close the gap; it does so continuously, level-triggered, and idempotently.

flowchart LR
    O[Observe<br/>informer cache] --> D[Diff<br/>compare spec to status]
    D --> A[Act<br/>PATCH/POST/DELETE]
    A --> O
    Note1[API server<br/>source of truth]
    O -.reads.-> Note1
    A -.writes.-> Note1

This loop runs in every controller. The mechanisms are identical; the policy is what each controller implements.

The four parts of a Kubernetes controller

A production controller has four logical parts:

flowchart TB
    API[API server] -->|watch| I[Informer<br/>local cache]
    I -->|events| Q[Workqueue]
    Q -->|dequeue| R[Reconciler]
    R -->|GET/PUT| API
    R -->|PATCH/POST/DELETE| API
    API -.updates.-> I

Informer (the cache)

The informer maintains a local in-memory cache of the controller’s resource type, kept up-to-date via the API server’s watch. The informer:

• Lists resources on startup (initial state)
• Watches for ADDED / MODIFIED / DELETED events
• Stores the latest version of each object in memory
• Forwards events to the workqueue

The informer decouples the reconciler from the API server. The reconciler reads from the informer cache, not directly from the API server. This makes the reconciler fast (in-memory reads) and resilient to API server hiccups (the cache survives brief outages).

Workqueue

The workqueue holds the keys (typically <namespace>/<name>) of objects that need reconciliation. The reconciler dequeues keys, fetches the latest state, and reconciles.

The workqueue provides:

• Deduplication — multiple events for the same key are coalesced into one work item
• Rate limiting — if a key fails to reconcile, retries are backed off exponentially
• Re-listing on retry — the reconciler re-fetches the latest state, not the cached event

Reconciler

The reconciler is the controller’s policy. It implements the diff and act:

func (r *ReconcileDeployment) Reconcile(ctx context.Context, key string) error {
    obj, err := r.client.Get(ctx, key)
    if err != nil {
        return err  // requeue with backoff
    }
    
    desiredReplicas := obj.Spec.Replicas
    observedReplicas := obj.Status.Replicas
    
    if observedReplicas == desiredReplicas {
        return nil  // no gap, done
    }
    
    // Act: scale the owned ReplicaSet
    err = r.scaleReplicaSet(ctx, obj, observedReplicas, desiredReplicas)
    return err
}

The reconciler is a pure function of the current state. It does not store state between calls. Re-running with the same input produces the same output (idempotency).

Client

The client writes back to the API server. Patches (PATCH) are preferred over full updates (PUT) — patches carry only the fields being changed, reducing the chance of 409 Conflict.

Level-triggered vs edge-triggered

Kubernetes uses level-triggered reconciliation: the controller acts on the current state, not on a specific event. This is a deliberate design choice.

sequenceDiagram
    autonumber
    participant API as API server
    participant C as Controller
    participant W as World

    loop every N seconds
        C->>API: GET object (latest)
        C->>W: observe
        C->>C: diff
        alt gap
            C->>API: PATCH (act)
        else no gap
            Note over C: no-op
        end
    end

A dropped watch event does not break a level-triggered controller — the next reconcile re-reads the state and re-evaluates. An out-of-order event is harmless — the controller does not assume “if event A, state is B”; it asks “what is the state now?”.

Edge-triggered reconciliation (act on every event) is faster but fragile. Kubernetes uses it sparingly (e.g., the scheduler reacting to a Pod being created).

What makes a controller robust

A well-designed controller has these properties:

Idempotent

Re-running the reconcile with the same input produces the same output. If the reconciler is interrupted, the next call catches up; no duplicate effects.

Convergence

Eventually, repeated reconciliations drive the observed state to match the desired state, regardless of starting state.

Bounded retries

If the reconciler fails, the workqueue retries with exponential backoff. A persistent failure does not loop hot; it waits longer between retries.

Owner references

Objects created by the controller set metadata.ownerReferences to point at the parent. The garbage collector uses these to cascade delete.

Status updates

The controller writes its progress to status (e.g., status.replicas, status.conditions). The API server preserves it during spec updates.

A walkthrough: Deployment controller’s loop

sequenceDiagram
    autonumber
    participant API as API server
    participant I as Informer
    participant Q as Workqueue
    participant R as Deployment Reconciler
    participant RR as ReplicaSet Reconciler

    I->>Q: Deployment web-7c8 ADDED
    Q->>R: Reconcile web-7c8
    R->>API: GET Deployment web-7c8
    R->>R: read spec.replicas (5)
    R->>API: list ReplicaSets with owner=web-7c8
    R->>API: GET Pods with owner=...
    R->>API: count ready Pods (3)
    R->>R: diff: 3 < 5, gap = 2
    R->>API: PATCH active ReplicaSet (replicas: 4)
    R->>API: PATCH active ReplicaSet (replicas: 5)
    R->>Q: requeue (next reconcile cycle)
    
    Note over RR: ReplicaSet controller also reconciles<br/>independently
    
    RR->>Q: ReplicaSet rs-xyz ADDED
    RR->>R: Reconcile rs-xyz (independent loop)
    RR->>API: GET ReplicaSet rs-xyz
    RR->>RR: read spec.replicas (5)
    RR->>API: list Pods owned by rs-xyz
    RR->>RR: count running Pods (5)
    RR->>RR: no gap, no-op

The Deployment controller and ReplicaSet controller are separate controllers running separate loops. They do not call each other directly. The Deployment controller scales the ReplicaSet; the ReplicaSet controller creates Pods. Coordination is via the API server.

Why controllers can run in HA

A controller can be deployed as a Deployment with multiple replicas. Each replica runs the same loop against the same API server. Why doesn’t this conflict?

• The workqueue is per-replica. Each replica processes its own keys.
• API server writes use optimistic concurrency. If two replicas try to PATCH the same object with the same resourceVersion, one succeeds and the other gets 409.
• The loser’s reconciler re-reads, re-diffs, re-applies.

In production, controller HA is rare (most controllers are single-leader via Lease). But the design supports HA, and Operator SDKs often run controllers in HA by default.

Failure modes of the control loop

How to inspect a controller

# Workqueue metrics (from controller-manager /metrics)
workqueue_adds_total{controller="deployment"}
workqueue_depth{controller="deployment"}
workqueue_queue_duration_seconds{controller="deployment"}
controller_runtime_seconds{controller="deployment"}

# From kube-state-metrics or Prometheus
kube_deployment_status_replicas
kube_deployment_spec_replicas

A growing workqueue_depth with no change in replicas is the signature of a stuck controller.

Cross-course references

• The Linux course part VI-Linux-Processes covers the process management primitives that controllers run on top of (process state, signals).
• The Observability course part IX-Observability-Exporters covers the metrics the operator monitors on controllers.
• The Linux course part XXIV-Linux-Time covers chrony — controllers depend on clocks within tolerance for leases and timestamps.
• The Docker course part XXX-Docker-Lifecycle covers the lifecycle primitives that controllers drive.

Quiz

Production discipline

• Treat every controller as observability-critical: every controller exposes work queue depth, reconcile duration, and error rate metrics. Production monitors all three.
• Design reconcilers to be idempotent and level-triggered; reject designs that depend on event ordering.
• Set exponential backoff on retry; never hot-loop on a failing key.
• Use owner references for every created object; cascading delete depends on them.
• Audit reconciler errors with structured logs and trace IDs. A controller that errors on a single key can quietly fall behind if no one notices.