Desired state and reconciliation — the model that runs the cluster

Kubernetes is fundamentally a desired-state system. The operator never tells Kubernetes how to do something; the operator tells Kubernetes what should be true. A set of components called controllers continuously observes the actual state, compares it to the desired state, and acts to close the gap. This lesson introduces the model that runs every other part of the cluster.

The model in one sentence

flowchart LR
    S[Spec: what should be true] --> R[Reconciler]
    O[Observed: what is true] --> R
    R -->|act to close gap| A[Actuation]
    A --> O

A controller observes the cluster, compares the observed state to the spec, and acts to make the observed state match the spec.

That is the entire Kubernetes control plane. Every other concept in this course — Deployments, Services, StatefulSets, kubelet, the scheduler, etcd — is an instance or a support component of this loop.

Spec, status, and the gap

Every Kubernetes object has a spec (the intent) and a status (the observed reality). The controller’s job is to close the gap between them:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 5           # intent: 5 Pods
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.1
status:
  observed replicas: 3    # reality: 3 Pods running
  ready replicas: 2
  available replicas: 2
  unavailable replicas: 1
  conditions:
  - type: Progressing
    status: "True"
    reason: NewReplicaSetAvailable
  - type: Available
    status: "False"
    reason: MinimumReplicasUnavailable

The Deployment controller’s reconcile loop reads spec.replicas and status.observed replicas. If they differ, it acts: it creates a new ReplicaSet with the right replicas, or scales an existing one. The Pods themselves are reconciled by the kubelet on each node.

sequenceDiagram
    autonumber
    participant U as User
    participant API as API server
    participant Ctrl as Deployment controller
    participant RS as ReplicaSet controller
    participant Sched as Scheduler
    participant Kub as kubelet

    U->>API: Apply Deployment (replicas: 5)
    API-->>Ctrl: Watch event: Deployment ADDED
    Ctrl->>API: Create ReplicaSet (replicas: 5)
    API-->>RS: Watch event: ReplicaSet ADDED
    RS->>API: Create 5 Pods
    API-->>Sched: Pod ADDED (unscheduled)
    Sched->>API: Bind Pod to node X
    API-->>Kub: Pod ADDED on node X
    Kub->>Kub: Pull image, start container
    Kub->>API: Update Pod status -> Running
    API-->>Ctrl: Deployment status updated

Why the model is desirable

Imperative systems ask the operator to describe what to do: “start container X on host Y, then update load balancer Z”. Desired-state systems ask the operator to describe what should be true: “there should be 5 replicas of web:v2 behind this Service”.

The trade-offs:

The Kubernetes model is declarative, idempotent, and continuously reconciling. These three words describe the design intent at every layer.

The observe-diff-act loop

The reconciliation loop has three steps, repeated continuously:

1. Observe — read the desired state from the API server and the observed state from the world (kubelet, the cluster, etc.).
2. Diff — compare the two. Where do they diverge?
3. Act — perform the actions that close the divergence.

flowchart LR
    O[Observe] --> D[Diff]
    D --> A[Act]
    A --> O
    A -.->|update observed| W[World]
    W -.->|feedback| O

This loop runs:

• In the Deployment controller — every few seconds, comparing spec.replicas to the count of ready Pods.
• In the kubelet — every few seconds, comparing the Pod spec to the containers actually running on the node.
• In the scheduler — on every unscheduled Pod, matching constraints to nodes and binding.
• In the controller manager’s node controller — every few seconds, comparing node heartbeat leases to the NotReady threshold.

The loop is the design primitive. Every component is a specialisation of “observe what is, diff against what should be, act.”

Level-triggered vs edge-triggered reconciliation

Kubernetes uses level-triggered reconciliation: the controller acts based on the current state, not on a specific event. This is a deliberate choice:

• A dropped watch event does not break the loop. The next reconcile re-reads the current state.
• An out-of-order event does not break the loop. The reconciler does not assume “if event A then state is B”; it asks “what is the current state?”
• Two reconcilers racing on the same object do not break the loop. Optimistic concurrency in the API server ensures the second writer re-reads and re-tries.

sequenceDiagram
    autonumber
    participant API as API server
    participant C as Controller
    participant W as World

    loop every N seconds
        C->>API: GET object (current spec + status)
        C->>W: GET observed state
        C->>C: Diff
        alt Gap exists
            C->>API: PATCH / status / apply
        else No gap
            Note over C: no-op
        end
    end

Edge-triggered reconciliation (act on every event) is faster but fragile. Most production Kubernetes controllers are level-triggered for resilience; edge-triggered is used for low-latency paths (e.g., the scheduler reacting to a Pod being created).

Convergence: when does the system stabilise?

A controller is convergent if repeated reconciliations eventually drive the observed state to match the desired state, regardless of starting state. Most Kubernetes controllers are convergent under normal conditions:

• Deployment converges to replicas count of ready Pods.
• kubelet converges to the container state described by the Pod spec.
• scheduler converges to one Pod bound per Pod that has unscheduled Pods.

Convergence can fail (or take very long) when:

• The desired state is unachievable — replicas: 100 on a 20-node cluster with requests.cpu: 16 per Pod.
• The desired state is inconsistent — replicas: 5 with a PodDisruptionBudget of minAvailable: 10.
• The world is non-idempotent — the action the controller takes has side effects that change the desired state (e.g., scaling a StatefulSet’s volume).

The operational response to non-convergence is to read the controller’s events and the object’s status.conditions. The controller has logged what it is waiting for.

Reconciliation does not solve everything

The reconciliation model is powerful, but it does not solve:

• Stateful correctness — the controller can guarantee that 5 Pods are running; it cannot guarantee that they have consistent database state.
• Application-level invariants — the controller does not know that “web should not start until the database is migrated”. That requires init containers, readiness probes, or an admission policy.
• Cross-cluster consistency — a single Kubernetes cluster is the boundary of reconciliation. Multi-cluster requires an external system (GitOps, federation, application-level replication).
• Time-bound guarantees — convergence is eventually consistent. The cluster does not promise that “by T+5s, the new Deployment will be at full replica count”. Production must design for transient states.

The mental shift for new operators

Operators coming from imperative systems (Chef, Ansible, manual ssh) often make the same mistake: they try to tell Kubernetes what to do (imperative kubectl, custom scripts) instead of declaring what should be true. The shift is to:

1. Write the YAML for what should be true.
2. Apply it.
3. Watch the status converge.
4. When the status is wrong, fix the spec, not the world. kubectl edit on the world (the running Pod’s image, for example) will be reverted by the reconciler.

This last point is the most counterintuitive: direct edits to running objects are temporary. The reconciler will eventually re-create them to match the spec. This is not a bug; it is the design.

kubectl edit pod web-abc -n prod    # temporary; reconciler may overwrite
kubectl edit deployment web -n prod # permanent; spec is the source

Operators as a specialisation of the model

The Operator pattern is the same reconciliation model applied to application-specific resources. An Operator is a controller that:

1. Watches a custom resource (e.g., PostgresCluster).
2. Reconciles the world to match the spec (creates Pods, PVCs, Services, runs migrations).
3. Reports status back on the custom resource.

Operators extend Kubernetes with application knowledge while keeping the same fundamental loop. Production operators run in the cluster (as Deployments) and follow the same operational discipline as any other workload.

Cross-course references

• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives (namespaces, cgroups) the reconciler ultimately actuates.
• The Observability course part I-Observability-Foundations covers the metrics/logs/traces signals that the operator uses to verify reconciliation has converged to a healthy state, not just a running state.
• The Ansible course covers imperative configuration management; the reconciliation model is the opposite of how Ansible pushes state. Kubernetes does not push — it pulls.
• The Linux course part VI-Linux-Processes covers process states; Pod phases (Pending/Running/Failed/etc.) are a distributed-system analogue.

Quiz

Production discipline

• Treat the spec as authoritative and the running objects as ephemeral.
• Never edit a Pod spec directly to fix a problem — edit the Deployment, StatefulSet, or other controller that owns the Pod.
• Monitor reconciliation gap, not just object existence. A Pod that exists but is failing its readiness probe is not in the desired state for a healthy workload.
• Recognise that convergence is not health — a system that is technically reconciled can still be unhealthy. Probes and business KPIs are the health signal.
• When introducing a new resource type, design its controller to be level-triggered, idempotent, and convergent — even if the resource is small. The cost of a fragile controller is paid in incidents.