Reconciliation pitfalls — what breaks the loop in production

The reconciliation model is robust in theory but produces specific failure modes in production. This lesson walks the pitfalls that break clusters — cascading deletes, partial state, drift, lost updates — and the patterns that make reconciliation resilient.

Pitfall 1: cascading delete without intent

kubectl delete cascades by default. Deleting a Deployment deletes its ReplicaSets, which delete their Pods, which delete their PVCs (unless reclaimPolicy: Retain).

flowchart LR
    D[Deployment deleted] --> R[ReplicaSets deleted]
    R --> P[Pods deleted]
    P --> PVC[PVCs deleted]
    PVC --> PV[PVs deleted]

Production disasters:

• Deleting a Deployment by accident removes the entire workload. Recovery requires re-applying the manifest.
• Deleting a StatefulSet removes its Pods and PVCs (with default reclaimPolicy: Delete), which destroys persistent data.
• Deleting a Namespace cascades to every object in it.

Defensive patterns:

# Dry-run before delete
kubectl delete deployment web --dry-run=server -n prod

# Cascade delete but preserve PVCs
kubectl delete statefulset db --cascade=orphan -n prod

# Delete without grace period (immediate)
kubectl delete pod web-abc --grace-period=0 --force

# Background the cascade and confirm
kubectl delete namespace team-a-prod --wait=false
# ... review what would be deleted ...
kubectl get all -n team-a-prod
kubectl delete namespace team-a-prod --wait

Pitfall 2: partial state during convergence

Reconciliation is not atomic. Between the controller’s spec update and the world’s state update, there is a window of partial state:

sequenceDiagram
    autonumber
    participant API as API server
    participant Ctrl as Controller
    participant W as World

    Ctrl->>API: PATCH deployment.replicas: 5 -> 10
    API-->>Ctrl: 200 OK
    Note over W: Partial state: spec=10, world has 5 replicas
    Ctrl->>W: create 5 more Pods
    W-->>Ctrl: progress
    Ctrl->>API: PATCH status.replicas: 5 -> 10
    Note over W: Converged: spec=10, world has 10

During the partial-state window, spec.replicas=10 but the world has 5. A controller that depends on status.replicas sees 5; a controller that depends on spec.replicas sees 10.

Production consequences:

• HPA measures the old status.replicas and computes a scale-down that the controller will undo (race condition).
• PDB counts spec.replicas to enforce minAvailable; if the spec says 10 but only 5 are running, PDB thinks 5 can be unavailable, but only 5 actually are. Disruption budgets can be violated during partial state.
• CI/CD that gates on kubectl rollout status waits for the partial state to clear (the convergence).

The defence is steady-state-aware controllers: write controllers that read the current observed state plus the spec, not just the spec.

Pitfall 3: drift between spec and observed state

Drift is the difference between what the spec says and what the world has, not caused by the controller. Sources:

• Manual edits — kubectl edit of a running object
• Side effects — a CSI driver fails to delete a volume
• Network partitions — kubelet cannot reconcile its observed state with the spec
• Operator code bugs — a controller that updates the wrong field

# Detect drift
kubectl get deployment web -o yaml | \
  diff - <(kubectl get deployment web -o yaml | yq '.spec.replicas')

The controller’s reconcile loop is designed to close drift. A healthy cluster has the controller periodically re-asserting the spec. A drift that persists is a controller that is not running or not authoritative.

Defensive patterns:

• Git as source of truth — every spec change goes through Git, with CI validating the manifest
• Admit if reconciliation is authoritative — never kubectl edit a managed object; edit the source
• Drift detection as a separate concern — a periodic diff between Git and cluster state, alerting on divergence

Pitfall 4: lost updates via optimistic concurrency

Two actors editing the same object race. The API server uses optimistic concurrency: the second writer loses.

sequenceDiagram
    autonumber
    participant A as Actor 1 (controller)
    participant B as Actor 2 (operator)
    participant API as API server

    A->>API: GET object (resourceVersion=128034)
    B->>API: GET object (resourceVersion=128034)
    A->>API: PUT (spec=..., resourceVersion=128034) -> 200 OK (now 128035)
    B->>API: PUT (spec=..., resourceVersion=128034) -> 409 Conflict
    Note over B: B must re-read and try again

Production consequences:

• A controller and an operator writing to the same fields cause silent lost updates if the operator’s tooling swallows 409 Conflict
• A scale-up combined with a config change can drop the scale-up

Defensive patterns:

• Use kubectl apply --server-side which preserves managed fields
• Use field-level merging instead of full PUTs
• Audit the operator’s retry behaviour on 409 Conflict
• Separate concerns: different fields owned by different actors

Pitfall 5: non-idempotent controllers

A controller that has side effects on every reconcile. If re-running the controller produces duplicate effects, the system is not converging.

// Anti-pattern: not idempotent
func Reconcile(ctx context.Context, key string) error {
    return sendNotification("Reconciling " + key)  // duplicate notifications!
}

// Idempotent: uses the world's state to decide
func Reconcile(ctx context.Context, key string) error {
    obj, _ := r.client.Get(ctx, key)
    if obj.Status.LastNotified != obj.Generation {
        sendNotification("Reconciling " + key)
        obj.Status.LastNotified = obj.Generation
        r.client.Update(ctx, obj)
    }
    return nil
}

Production consequences of non-idempotent controllers:

• Duplicate notifications, duplicate metrics, duplicate resource creation
• The system appears to oscillate (controller alternates between reconcile and “re-reconcile” with new side effects)

The defensive pattern is: check the world state before acting; only act if the action is needed.

Pitfall 6: hot-looping on a failing dependency

A controller whose reconcile fails because a downstream is unavailable. Without exponential backoff, the controller hot-loops:

sequenceDiagram
    autonumber
    participant Ctrl as Controller
    participant Dep as Downstream (broken)

    loop retry
        Ctrl->>Dep: call
        Dep-->>Ctrl: error
        Ctrl->>Ctrl: log + retry
    end

Production consequences:

• Controller fills logs and metrics with retries
• Other work items in the queue starve
• API server rate limits may engage (429)

Defensive patterns:

• Exponential backoff in the workqueue (built into client-go)
• Circuit breaker: after N failures, stop calling the downstream for M seconds
• Distinct error types: distinguish “transient” from “permanent” failures; only retry transient

Pitfall 7: leader-election failures

A controller that loses its leader election Lease stops reconciling. Other replicas stand by. If none acquires, the controller is stalled cluster-wide.

Production causes:

• API server unreachable for Lease renewal
• Clock skew preventing Lease validation
• RBAC denies Lease creation
• Operator accidentally deletes the Lease

Defensive patterns:

• Monitor the Lease: alert on holderIdentity changes
• HA controller-manager (2+ replicas) for fast failover
• Externalised Lease: store leader election in a database outside the cluster

Pitfall 8: garbage collector races

When the GC is mid-walk and an owner reference changes, the GC may either:

• Delete the children of the old owner (orphan) — correct
• Delete children of the new owner (loss) — incorrect if the new owner was set incorrectly

Production causes:

• Operator code that changes ownerReferences mid-reconcile
• Manual kubectl edit of ownerReferences
• Third-party tools that re-parent objects

Defensive patterns:

• Never change ownerReferences outside the original controller
• Audit ownerReferences changes via admission or audit policy
• Treat ownerReferences as immutable in normal operation

Pitfall 9: status updates that don’t match reality

A controller that writes status but doesn’t actually close the gap. The system looks converged but isn’t.

status:
  replicas: 5
  readyReplicas: 5
  # ... but the Pods are crashlooping and the controller
  # hasn't actually fixed them

This is harder to detect than non-convergence. The defence is:

• Independent monitoring of the world’s state
• External smoke tests (synthetic probes from outside the cluster)
• Alerts on health, not just convergence

Pitfall 10: too many owners, no clear ownership

An object with multiple ownerReferences (the cluster allows this) is owned by all of them. When any is deleted, the GC deletes the object. When the controllers race, the object flaps.

Defensive pattern: one owner per object. Multi-ownership solves problems that belong in a controller’s design, not in the API.

Defensive patterns: a checklist

For every controller you write or operate:

• Reconciler is idempotent
• Reconciler uses exponential backoff on retry
• Controller uses owner references for created objects
• Reconciler handles optimistic concurrency (409 on write)
• Reconciler reads the world’s current state, not the watch event
• Status updates reflect reality, not intent
• Work queue is bounded (rate-limited, depth-monitored)
• Leader election Lease is monitored and HA
• Health (probes, business KPIs) is independent of convergence

Cross-course references

• The Linux course part VI-Linux-Processes covers process state transitions; reconciliation is the cluster’s analogue.
• The Observability course part CIX-Observability-InvestigationWorkflows covers investigation methodology that applies to reconciliation failures.
• The Linux course part XXIV-Linux-Time covers chrony — leader election depends on clock accuracy.
• The Docker course part XXX-Docker-Lifecycle covers the container lifecycle that drives much of the convergence.

Quiz

Production discipline

• Treat kubectl delete namespace as a multi-step operation with mandatory human review.
• Make the Git repo the source of truth; never kubectl edit a managed object directly.
• Test controllers for idempotency, lost-update tolerance, and hot-loop behaviour.
• Use server-side apply for human edits of managed objects (preserves field ownership).
• Monitor drift, work queue depth, and reconciliation errors independently of cluster state.