Failure domain and blast radius — partitioning risk across clusters

Failure domain and blast radius are the most fundamental reasons for cluster boundaries. A regional outage, control-plane failure, or CNI bug in one cluster takes down every workload in that cluster. This lesson walks the blast-radius partitioning, the trade-offs, the relationship to RTO/RPO, and the operational discipline.

Blast radius in single-cluster vs multi-cluster

flowchart LR
    A[Single cluster] --> B[All workloads]
    C[Outage] --> B
    D[Multi-cluster] --> E["Cluster 1: subset A"]
    D --> F["Cluster 2: subset B"]
    D --> G["Cluster 3: subset C"]
    H[Outage in cluster 1] --> E

In a single-cluster deployment, a regional outage or control-plane failure takes down every workload. The blast radius is 100%.

In a multi-cluster deployment, the workloads are partitioned across clusters. A failure in cluster 1 takes down only the workloads in cluster 1. The blast radius is 33% (for 3 clusters).

Failure modes that drive blast radius

flowchart TD
    A[Failure modes] --> B[Regional outage]
    A --> C[Control plane failure]
    A --> D[CNI bug]
    A --> E[etcd corruption]
    A --> F[Node kernel panic]
    A --> G[Cloud account compromise]

The failure modes that drive cluster boundaries:

• Regional outage. A cloud region becomes unavailable. Multi-region multi-cluster ensures workloads in other regions continue.
• Control plane failure. A bug or misconfiguration in the API server takes down the cluster. Multi- cluster with separate control planes ensures other clusters continue.
• CNI bug. A misconfiguration in the CNI affects every Pod. Multi-cluster with different CNI versions or configurations reduces correlated risk.
• etcd corruption. A bad write corrupts etcd. Multi-cluster with separate etcd clusters limits the damage.
• Node kernel panic. A kernel bug takes down nodes. Multi-cluster with different kernel versions reduces correlated risk.
• Cloud account compromise. An attacker gains access to the cloud account. Multi-cluster in different accounts limits the damage.

Cluster-per-workload-type

flowchart LR
    A["Cluster: stateful"] --> B[Database workloads]
    C["Cluster: stateless"] --> D[API workloads]
    E["Cluster: batch"] --> F[Analytics workloads]

A common pattern is cluster-per-workload-type:

• Stateful cluster for databases (Postgres, MongoDB, Elasticsearch). Different storage characteristics, different backup cadence.
• Stateless cluster for API workloads. Different scaling characteristics, different resource profile.
• Batch cluster for analytics, ML training. Different scheduling characteristics, different resource profile.

The pattern partitions blast radius by workload type: a bug in the batch cluster’s scheduling does not affect the API cluster.

RTO/RPO trade-offs

The RTO/RPO is influenced by the topology:

• Single cluster. Recovery depends on backups (etcd snapshot, Velero) and the recovery time.
• Multi-cluster, same region. Recovery is faster because workloads can fail over to the other cluster.
• Multi-cluster, multi-region. Recovery is fastest because a regional outage takes down only one cluster.

Designing for failure domains before scale

The discipline:

1. Identify the failure modes (regional outage, control plane, CNI bug, etc.).
2. For each failure mode, determine the impact (which workloads are affected, what is the RTO/RPO).
3. For each high-impact failure mode, design a cluster boundary that limits the impact.
4. Model the cost of each cluster boundary.
5. Choose the boundaries that limit the highest- impact failure modes at acceptable cost.

A design that focuses on scale (single huge cluster) without considering failure modes produces a fragile deployment. A design that considers failure modes first produces a resilient deployment.

The operational failure modes

Blast radius reduction fails for predictable reasons:

• Correlated failure. All clusters share the same CNI version, the same kernel version, or the same cloud account. A bug in any of these affects every cluster.
• Same region. Multi-cluster in the same region does not protect against regional outage.
• Shared credentials. All clusters use the same cloud credentials. A compromise of one set compromises all.
• Single Argo CD instance. All clusters are managed by one Argo CD; a compromise affects all.

Quiz

Production discipline

Blast radius reduction in production rests on five non-negotiable elements:

• Identify failure modes first. Regional outage, control plane, CNI bug, etcd corruption, account compromise — each drives a different boundary.
• Multi-region for regional resilience. Same- region multi-cluster does not protect against regional outage.
• Multi-account for credential compromise. A single cloud account is a single point of failure.
• Diverse CNI / kernel versions. A bug in one version should not affect every cluster.
• Test failure modes quarterly. Rehearse regional failover, control-plane rebuild, CNI swap.

Failure domain and blast radius are the foundation of multi-cluster design. The discipline is to identify the failure modes, design the boundaries to mitigate them, and test the mitigation regularly.