Topology spread HA patterns — balanced distribution in production

Topology spread is the modern HA primitive. This lesson walks the canonical patterns: layered constraints for node + zone distribution, per-version distribution during rollouts, per-tenant isolation, and the combinations that give production-grade availability.

Pattern 1: layered constraints

spec:
  replicas: 6
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web

Two constraints:

• Hostname: one per node (skew 1).
• Zone: balanced across zones (skew 1).

flowchart TB
    subgraph "us-east-1a"
      N1[node-01] --> P1[web-1]
      N2[node-02] --> P2[web-2]
    end
    subgraph "us-east-1b"
      N3[node-03] --> P3[web-3]
      N4[node-04] --> P4[web-4]
    end
    subgraph "us-east-1c"
      N5[node-05] --> P5[web-5]
      N6[node-06] --> P6[web-6]
    end

A node failure loses 1 Pod; a zone failure loses 2.

Pattern 2: per-version distribution

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web
    matchLabelKeys:
    - pod-template-hash

Each version (ReplicaSet) distributes independently across nodes. During a rollout, the new ReplicaSet’s Pods spread to all nodes; the old ReplicaSet’s Pods stay until evicted.

flowchart TB
    subgraph "Old RS: pod-template-hash=abc"
      N1[node-01] --> P1[web-1]
      N2[node-02] --> P2[web-2]
    end
    subgraph "New RS: pod-template-hash=def"
      N1 --> P3[web-3]
      N2 --> P4[web-4]
    end

Without matchLabelKeys, the constraint counts both versions together; with it, each version is independent.

Pattern 3: per-tenant isolation

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: workload
    matchLabelKeys:
    - tenant

Each tenant’s Pods distribute across nodes. Two tenants may co-locate on the same node, but each tenant’s distribution is balanced.

flowchart TB
    N1[node-01] --> P1A["Tenant A: pod-1"]
    N1 --> P1B["Tenant B: pod-1"]
    N2[node-02] --> P2A["Tenant A: pod-2"]
    N2 --> P2B["Tenant B: pod-2"]

Pattern 4: HA with PodDisruptionBudget

spec:
  replicas: 6
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: web

Topology spread distributes; PDB bounds voluntary disruption. Together: replicas survive failure (spread) and survive maintenance (PDB).

flowchart TB
    A[Failure scenario] --> B{Hardware failure}
    A --> C{Voluntary disruption}
    B -->|node fails| D[1 Pod lost]
    B -->|zone fails| E[2 Pods lost]
    C -->|drain| F["PDB blocks<br/>if minAvailable violated"]
    D --> G["Deployment recreates<br/>Pod; topology re-spreads"]
    E --> G
    F --> G

Pattern 5: HA database with quorum

spec:
  replicas: 3
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: db

Three replicas, one per zone. A zone failure loses 1 replica; the database quorum (2 of 3) holds.

flowchart TB
    subgraph "Zone a"
      N1 --> P1[db-1]
    end
    subgraph "Zone b"
      N2 --> P2[db-2]
    end
    subgraph "Zone c"
      N3 --> P3[db-3]
    end
    P1 -.->|replication| P2
    P2 -.->|replication| P3

Pattern 6: graceful skew during node failure

A ScheduleAnyway constraint tolerates skew during node failure:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: web

flowchart TB
    A["6 replicas on 6 nodes<br/>skew 0"] --> B[node-03 fails]
    B --> C[web-3 lost]
    C --> D["Deployment recreates<br/>web-7"]
    D --> E["web-7 schedules on<br/>least-loaded node"]
    E --> F["Distribution: 2, 1, 1, 1, 1<br/>skew 1"]

The skew temporarily exceeds the ideal but stays within the budget. The workload continues.

The trade-offs

Distribution vs flexibility

flowchart LR
    A[Strict skew] -->|no imbalance| B[Predictable]
    A -->|may produce Pending Pods| C[Inflexible]
    D[Loose skew] -->|tolerates imbalance| F[Flexible]
    D -->|fewer Pending Pods| G[Some imbalance]

Distribution vs cost

Strict skew requires more nodes (one per Pod). Loose skew fits more Pods per node. The cost trade-off: cluster size vs availability.

flowchart TB
    A[6 replicas, maxSkew 1] -->|6 nodes| B["Cost: 6 nodes"]
    C[6 replicas, maxSkew 3] -->|3 nodes| D["Cost: 3 nodes"]

Failure mode: cluster too small

A cluster with insufficient nodes for the desired distribution produces Pending Pods (with DoNotSchedule) or imbalanced distribution (with ScheduleAnyway).

flowchart TB
    A[Cluster too small] --> B{DoNotSchedule}
    A --> C{ScheduleAnyway}
    B --> D[Pending Pods]
    C --> E[Imbalanced distribution]
    D --> F["Scale cluster<br/>or relax skew"]
    E --> G["Accept imbalance<br/>or scale cluster"]

Anti-patterns

Anti-pattern 1: DoNotSchedule with insufficient nodes

replicas: 10
topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule

A 10-replica Deployment on a 5-node cluster: 5 Pods Pending. The fix: ScheduleAnyway or scale the cluster.

Anti-pattern 2: too many constraints

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
- maxSkew: 1
  topologyKey: topology.kubernetes.io/region
- maxSkew: 1
  topologyKey: rack.example.com/rack-id

Four constraints. Each adds evaluation cost. Most workloads need only hostname + zone.

Anti-pattern 3: ignore PDB

A Deployment with topology spread but no PDB. Voluntary disruption (drain) can take down multiple Pods at once.

flowchart LR
    A[Without PDB] -->|drain| B[Multiple Pods lost]
    B --> C[Topology re-spreads]
    C --> D[Workload degraded]
    A2[With PDB] -->|drain blocked| E[Drain waits or fails]
    E --> F[Workload protected]

Quiz

Production discipline

• Layer hostname + zone. Two constraints give both node-level and zone-level distribution.
• Use matchLabelKeys for rollouts. Each version distributes independently.
• Combine with PDB. Topology spread distributes; PDB bounds voluntary disruption.
• Choose ScheduleAnyway for flexibility. A workload that prefers balance but accepts some imbalance uses ScheduleAnyway.
• Verify the cluster has the labels. A topology key with missing labels has no effect.
• Audit the resulting distribution. A dashboard that surfaces per-domain Pod counts catches imbalance.

Topology spread is the modern HA primitive. Operators who use it deliberately with the right constraints, weights, and PDBs have workloads that distribute predictably and survive failure.