HA topology — stacked vs external etcd, design choices

The choice between stacked etcd (each etcd member on the same host as a control plane component) and external etcd (separate dedicated hosts) is a foundational decision in cluster design. Each topology has trade-offs in failure isolation, cost, and operational complexity. This lesson walks the differences, the decision drivers, and the production discipline of each.

The two topologies

flowchart LR
    subgraph stacked[Stacked topology]
        direction TB
        CP1[Control plane + etcd member on cp-1]
        CP2[Control plane + etcd member on cp-2]
        CP3[Control plane + etcd member on cp-3]
    end
    subgraph external[External topology]
        direction TB
        CP4[API server on cp-1]
        CP5[API server on cp-2]
        CP6[API server on cp-3]
        E1[etcd on etcd-1]
        E2[etcd on etcd-2]
        E3[etcd on etcd-3]
        CP4 --- E1
        CP5 --- E2
        CP6 --- E3
    end

Stacked topology

In stacked topology, each control-plane node runs:

• API server (static pod).
• Controller manager (static pod).
• Scheduler (static pod).
• etcd member (static pod).

The kubeadm default.

# kubeadm init on cp-1
kubeadm init --control-plane-endpoint "loadbalancer.example:6443"

# kubeadm join on subsequent control-plane nodes
kubeadm join ... --control-plane

Each subsequent kubeadm join --control-plane adds both a new control-plane host and a new etcd member.

Pros of stacked

• Lower host count. 3 hosts vs 6 (3+3).
• Simpler kubeadm setup. Init then join.
• Single cluster from the operator’s perspective. No separate etcd-cluster to manage.

Cons of stacked

• Co-location of failure domains. API server unavailability correlates with etcd member loss.
• 3 hosts of capacity. A 5-member etcd requires 5 stacked hosts (which is overkill for most).
• No dedicated etcd operations team. The single cluster team manages API servers and etcd together.

External topology

In external topology, etcd runs on dedicated hosts. The control plane runs on its own hosts.

# Set up etcd cluster on dedicated hosts first
# Then point kubeadm at the external etcd:
kubeadm init \
  --etcd-servers="https://etcd-1:2379,https://etcd-2:2379,https://etcd-3:2379" \
  --etcd-cafile=... --etcd-certfile=... --etcd-keyfile=... \
  --control-plane-endpoint "loadbalancer.example:6443"

The kubeadm-managed API server connects to the external etcd cluster.

sequenceDiagram
    autonumber
    participant API as API server cp-1
    participant E as etcd cluster
    API->>E: gRPC: write
    Note over E: writes are committed by majority
    E-->>API: committed

Pros of external

• Failure isolation. Control plane host failure does not affect etcd; etcd host failure does not affect API servers.
• Specialised operations. The etcd hosts can be managed by a database team with etcd-specific skills.
• Independent scaling. The control plane can scale separately from the etcd cluster.
• 5-member etcd feasible. A 5-member etcd cluster has dedicated hosts; reasonable to operate.

Cons of external

• Double the host count. 6 hosts (3 control plane + 3 etcd) vs 3 stacked.
• More complex bootstrap. The etcd cluster must be bootstrapped first; kubeadm does not manage it.
• Network is a factor. API server to etcd is over the network; latency must be acceptable.

Failure scenarios

A control-plane host fails

External topology recovers faster.

The etcd cluster loses quorum

External topology is robust against API plane failures.

The etcd cluster corrupts

Both topologies rely on snapshot restore. The cluster loses API state on restore. Recovery is approximately equal in difficulty.

Scaling the topologies

For very large clusters, external topology is the production answer.

The “neither” choice

Some clusters don’t run their own etcd at all:

• Managed Kubernetes (EKS, GKE, AKS). The cloud provider runs etcd; the cluster sees only the API server.
• k0s, k3s, k3d. Lightweight distributions that may embed etcd differently.

The “neither” choice is common for production deployments that want to avoid etcd operations.

The decision

The decision drivers:

Most production clusters: stacked for < 500 nodes; external for > 500 nodes.

The migration path

A cluster can be migrated from stacked to external (though complex):

sequenceDiagram
    participant Stack as Stacked cluster
    participant Extern as External cluster
    Stack->>Stack: snapshot stacked etcd
    Stack->>Extern: install dedicated etcd
    Stack->>Extern: restore snapshot to dedicated etcd
    Extern->>Extern: update API server to use external etcd
    Extern->>Stack: shut down stacked etcd

The migration is a maintenance window with cluster unavailability. It is rarely done; build the right topology from the start.

The discipline

• Document the topology choice. The runbook should state “stacked” or “external” and the rationale.
• Match the topology to the cluster size. Stacked for small; external for large.
• Manage dedicated etcd hosts well. External topology shifts the operational burden to dedicated hosts; operate them as stateful.
• Plan for the migration. If the cluster is growing, plan the move from stacked to external before the stacked host resources are exhausted.

Quiz

Production discipline

• Document the topology in the runbook. Stacked or external; the rationale.
• Match topology to size. Stacked for small to medium; external for large.
• Plan the migration. When a stacked cluster grows, plan external before resources exhaust.
• Treat etcd hosts as stateful. External or stacked, the etcd hosts deserve dedicated operations.
• Rehearse the topology setup. A team’s first kubeadm init --external-etcd is rarely smooth; rehearse on staging.

Topology is the foundation of cluster operations. Operating it well is knowing the trade-offs and choosing deliberately.