Cluster boundaries — why separate clusters

Cluster boundaries partition workloads into separate clusters. The decision to draw a boundary is one of the most consequential in Kubernetes architecture. This lesson walks the reasons for boundaries, hard vs soft boundaries, the trade-offs, and the operational discipline.

Why separate clusters

flowchart TD
    A[Cluster boundaries] --> B[Environment]
    A --> C[Security]
    A --> D[Geography]
    A --> E[Failure domain]
    A --> F[Compliance]
    A --> G[Scale]

The reasons:

• Environment. Production workloads should not share a control plane with dev workloads. A misbehaving dev Pod cannot starve production; a dev cluster failure does not affect production.
• Security. High-security workloads (PCI, HIPAA, classified) often warrant separate clusters with separate credentials, separate access logs, and separate monitoring.
• Geography. Latency to a single region may be unacceptable for global users. Regional clusters place workloads closer to users.
• Failure domain. A regional outage or a control- plane failure takes down every workload in the cluster. Separating critical workloads into different clusters partitions the blast radius.
• Compliance. Some regulations require data residency (data in the EU stays in the EU). Regional cluster boundaries satisfy this.
• Scale. A single cluster handles ~5,000 nodes. Beyond that, multi-cluster scales horizontally.

Hard vs soft boundaries

flowchart LR
    A[Hard boundary] --> B["Separate cluster, separate control plane"]
    C[Soft boundary] --> D[Namespace within a cluster]

The two boundary types:

• Hard boundary. Separate clusters with separate control planes. Workloads in cluster A cannot reach workloads in cluster B without explicit cross-cluster networking. Hard boundaries provide the strongest isolation at the highest operational cost.
• Soft boundary. Namespaces within a single cluster, with NetworkPolicy, ResourceQuota, RBAC, and PSS enforcing isolation. Soft boundaries provide moderate isolation at low operational cost.

The choice depends on the driver:

The trade-offs

flowchart LR
    A[Hard boundary] --> B[Strong isolation]
    A --> C[High operational cost]
    A --> D[Cross-cluster networking required]
    E[Soft boundary] --> F[Moderate isolation]
    E --> G[Low operational cost]
    E --> H[Cross-cluster networking not required]

The trade-offs:

• Hard boundary — strongest isolation, highest operational cost (separate control planes, separate upgrades, separate monitoring).
• Soft boundary — moderate isolation, lowest operational cost.

Hard boundaries are justified only when the driver requires them. A boundary for “team isolation” is usually a soft boundary (namespaces with RBAC). A boundary for “PCI compliance” is a hard boundary (separate cluster with dedicated credentials).

The decision framework

flowchart TD
    A[Need a boundary?] --> B{Driver}
    B -->|Environment, team| C["Soft boundary: namespace"]
    B -->|PCI, HIPAA| D["Hard boundary: separate cluster"]
    B -->|Geography| E["Hard boundary: regional cluster"]
    B -->|Compliance| F["Hard boundary: data residency"]
    B -->|Scale| G["Hard boundary: beyond single cluster"]
    C --> H["Cost: low"]
    D --> I["Cost: high"]
    E --> I
    F --> I
    G --> I

The framework:

1. Identify the driver (environment, security, geography, compliance, scale, failure domain).
2. Determine whether the driver requires hard isolation (PCI, geography, compliance) or soft isolation (environment, team).
3. Choose the boundary type (cluster or namespace) that satisfies the driver at the lowest cost.
4. Document the boundary and the driver in the architecture.

The operational failure modes

Boundaries fail for predictable reasons:

• Boundary without driver. A cluster exists because someone decided “we need multi-cluster,” not because a driver justifies the cost.
• Inconsistent boundaries. Cluster A has restricted PSS; cluster B has baseline. The inconsistency is a compliance gap.
• Missing cross-cluster networking. Hard boundaries without Submariner or service mesh produce islands.
• Boundary drift. A namespace drifts into another team’s resources because RBAC is not enforced.
• Cost underestimation. Each cluster adds control-plane cost; 10 clusters triples the infrastructure bill.

Quiz

Production discipline

Cluster boundaries in production rest on five non-negotiable elements:

• Define boundaries before topology. Identify the driver before choosing the cluster count.
• Match boundary type to driver. Soft for environment and team; hard for compliance, geography, and scale.
• Document the boundaries. The architecture diagram must show every boundary, the driver, and the type (hard or soft).
• Enforce consistent policies across boundaries. PSS, NetworkPolicy, RBAC must be consistent across clusters of the same type.
• Review boundaries quarterly. Boundary decisions drift; the review catches the drift.

Cluster boundaries are architecture decisions. The discipline is to make the decisions deliberately, based on drivers, and to review them regularly.