Convergence — when does the system stabilise?

Convergence is the property that distinguishes a working Kubernetes system from a broken one. A controller is convergent if repeated reconciliation eventually drives observed state to match desired state. This lesson covers what convergence looks like, what blocks it, and how to recognise when a system is partially or fully converged.

Convergence in one sentence

A Kubernetes system converges when the controllers’ repeated reconciliation drives the observed state of every object to match its desired state, and the system is at steady state.

stateDiagram-v2
    [*] --> Transient
    Transient --> Transient: diff != 0, retrying
    Transient --> Steady: diff == 0
    Steady --> Transient: spec changes
    Steady --> [*]

The system is rarely at steady state — every spec change re-enters transient state until reconciliation closes the gap.

Steady state vs transient state

Steady state: every object’s status matches its spec (as expressed by the controller’s satisfaction criteria). No reconcile loop has a pending action.

Transient state: a spec change has occurred (or a node has been lost, or a Pod has been rescheduled); reconcile loops are running; some objects have a gap between spec and status.

The transient-to-steady transition is what Kubernetes calls convergence. The controller’s job is to make this transition as fast as possible (usually seconds) and as predictable as possible.

# Check a Deployment's convergence
kubectl rollout status deployment/web

deployment "web" successfully rolled out

The rollout status command polls until the controller’s status.conditions reach the steady-state criteria.

What blocks convergence

A system fails to converge when:

• The desired state is unachievable — replicas: 100 on a 5-node cluster with requests.cpu: 16 per Pod
• The desired state is inconsistent — replicas: 5 with a PodDisruptionBudget of minAvailable: 10
• A dependency is broken — PVC Pending, ImagePullBackOff, Service has no Endpoints
• The controller is stalled — leader election lost, controller-manager OOMKilled
• The actions are non-idempotent — the controller’s acts have side effects that prevent convergence

A non-convergent system produces a gap that does not close. The Pod events, the controller’s status.conditions, and the workqueue metrics all show the same pattern.

Partial convergence

A system can be partially converged — some objects at steady state, others in transient. Production examples:

flowchart LR
    A[Deployment A: 5/5 ready, converged]
    B[Deployment B: 3/5 ready, gap=2]
    C[StatefulSet C: 1/3 ready, PVC Pending]
    D[DaemonSet D: 5/5 nodes, converged]

    A --> OK[Steady]
    B --> T[Transient<br/>2 missing]
    C --> T2[Transient<br/>2 missing + PVC]
    D --> OK

In the diagram, Deployment A and DaemonSet D are converged; Deployment B has a partial gap; StatefulSet C has a deeper gap (Pods Pending because PVC is Pending).

Partial convergence is normal during rollouts or after a node loss. It is abnormal when it persists beyond the expected transient window (a few minutes for typical workloads; longer for StatefulSets and rollouts with maxUnavailable=0).

Grace periods

Many Kubernetes controllers have grace periods to avoid flapping. These are bounded waits before declaring failure or taking action:

Grace periods dampen transient noise at the cost of delaying failure detection. Production tuning is workload-specific:

• Latency-sensitive services: short minReadySeconds (catch failures fast)
• Slow-start services (databases, JVM warm-up): longer minReadySeconds (avoid flapping)
• Stateful workloads: longer progressDeadlineSeconds (allow for ordered startup)

How to read convergence

For a Deployment

kubectl get deployment web -o jsonpath='{.status}'

{
  "observedGeneration": 3,
  "replicas": 5,
  "updatedReplicas": 5,
  "readyReplicas": 5,
  "availableReplicas": 5,
  "conditions": [
    {"type": "Progressing", "status": "True", "reason": "NewReplicaSetAvailable"},
    {"type": "Available", "status": "True"}
  ]
}

Convergence: readyReplicas == replicas and both conditions True.

For a Node

kubectl get node worker-04 -o jsonpath='{.status.conditions}'

[
  {"type": "Ready", "status": "True"},
  {"type": "MemoryPressure", "status": "False"},
  {"type": "DiskPressure", "status": "False"},
  {"type": "PIDPressure", "status": "False"}
]

Convergence: Ready == True and no pressure conditions.

For a StatefulSet

kubectl rollout status statefulset/db

rollout status waits for status.readyReplicas == status.replicas and status.currentRevision == status.updateRevision.

Convergence vs health

Convergence is necessary but not sufficient for health. A system can be technically converged and unhealthy:

flowchart LR
    C[Converged<br/>replicas == readyReplicas] --> H{Healthy?}
    H -->|probes pass| OK[Healthy]
    H -->|probes fail| UH[Unhealthy but converged]
    UH --> OBS[Observable<br/>via metrics + logs]

A Pod that passes liveness but has a broken database connection is “ready” by the orchestrator’s standards but unhealthy by the application’s. Production observability must cover both:

• Convergence signal: spec.replicas == status.readyReplicas
• Health signal: probes (liveness, readiness, startup) pass; business KPIs are met

The role of kubectl rollout status

kubectl rollout status is a polling helper. It waits for the controller to converge according to the controller’s status criteria.

kubectl rollout status deployment web --timeout=300s

Returns:

• 0 if the rollout converged within the timeout
• non-zero if it did not

Useful in CI/CD pipelines as a gate: “is this rollout successful?”. But it does not check health, only convergence.

Patterns of non-convergence

Pattern 1: rollout stuck at progressDeadline

deployment "web" exceeded its progress deadlineSeconds: 600

The Deployment controller has tried for 10 minutes and the new ReplicaSet is not converging. Causes:

• New image fails its readiness probe (the new version is broken)
• New Pods cannot be scheduled (capacity, affinity)
• New Pods are failing their startup (config error)

Pattern 2: StatefulSet waiting for Pod N

statefulset "db" rollout: waiting for statefulset rolling update to complete 1 out of 3 pods...

The StatefulSet is at 1/3. Pod 1 is Pending because Pod 0 is not Ready. Common cause: Pod 0 is failing its readiness probe (database is not accepting connections yet).

Pattern 3: EndpointSlice empty

A Service with selector: app=web but no Pods match. The Service has no Endpoints; traffic to the Service ClusterIP is dropped. The Deployment controller may show the Deployment as converged (5/5 replicas), but the selector mismatch means the Service does not see the Pods.

Convergence in custom controllers (Operators)

Custom controllers follow the same convergence rules. The Operator pattern:

1. Define a CRD (e.g., PostgresCluster)
2. The Operator controller watches the CRD
3. Reconcile: observe CR, diff against spec, create/update resources

Convergence in Operators can be more complex:

• Multiple resources must converge together (PVC, Service, Pod, etc.)
• Application-level state (database replication) is not in Kubernetes
• The Operator must report status.conditions accurately to reflect partial convergence

A well-designed Operator reports status.conditions[].status: False for resources not yet converged, so the operator’s state reflects reality.

Cross-course references

• The Linux course part VI-Linux-Processes covers process state transitions; convergence is the cluster’s analogue of process state stability.
• The Observability course part CIX-Observability-InvestigationWorkflows covers investigation methodology for “is the cluster converged?” questions.
• The Docker course part XXX-Docker-Lifecycle covers the container lifecycle primitives that drive convergence.
• The Linux course part XXIV-Linux-Time covers chrony — Node lease convergence depends on clock accuracy.

Quiz

Production discipline

• Distinguish convergence from health: a converged cluster can still be unhealthy.
• Monitor status.conditions of major objects, not just replicas. Conditions carry the controller’s assessment.
• Set alert thresholds shorter than grace periods so failures are caught before the controller’s deadline expires.
• Use kubectl rollout status in CI/CD as a convergence gate; pair it with a health check (Prometheus, business KPI) for full validation.
• Recognise partial convergence during rollouts; don’t page on it; do page on persistent gaps.