Member lifecycle — add, remove, replace, retire

Members in an etcd cluster come and go: hosts get replaced, AZs are added for fault tolerance, the cluster grows from 3 to 5, a single member is retired. Each operation has a safe sequence; each unsafe shortcut risks quorum loss or accidental cluster re-bootstrap. This lesson walks the four operations — add, remove, replace, retire — at the kubeadm and manual levels.

The four operations

flowchart LR
    ADD[Add member] --> HEALTH[Cluster at N, target N+1]
    RM[Remove member] --> SHRUNK[Cluster at N-1]
    REPLACE[Replace member] --> SAME[Cluster at N]
    RETIRE[Retire old member] --> DONE[Member off cluster and host]

Each has a kubeadm-managed and a manual form. The kubeadm-managed form handles certificates and the static pod manifest; the manual form is etcdctl member add / etcdctl member remove.

Adding a member — manual sequence

A new member is added in two phases: register with the cluster, then bootstrap the new member.

# On a healthy existing member (the leader is preferred):
etcdctl member add cp-4 \
  --peer-urls=https://10.0.1.13:2380

This outputs a JSON document or table row describing the new member:

Member added to cluster e9b3a2c:

ID:               f4a8e1c2...
NAME:             cp-4
PEER ADDRS:       https://10.0.1.13:2380
CLIENT ADDRS:     https://10.0.1.13:2379

# Now bootstrap the new member with this environment:
ETCD_NAME="cp-4"
ETCD_INITIAL_CLUSTER="cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380,cp-4=https://10.0.1.13:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

On the new host (cp-4):

etcd \
  --name=cp-4 \
  --data-dir=/var/lib/etcd \
  --listen-client-urls=https://10.0.1.13:2379 \
  --advertise-client-urls=https://10.0.1.13:2379 \
  --listen-peer-urls=https://10.0.1.13:2380 \
  --initial-advertise-peer-urls=https://10.0.1.13:2380 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,... \
  --initial-cluster-state=existing

When the new member starts, it contacts a peer, receives a snapshot of the log, and joins as a follower.

sequenceDiagram
    autonumber
    participant New as New member (cp-4)
    participant Leader as Leader (cp-1)
    New->>Leader: dial peer URL
    Leader->>New: send snapshot if log is not enough
    New->>New: install snapshot in bbolt
    New->>Leader: AppendEntries from leader applied
    New->>New: raftAppliedIndex climbs to leader's commitIndex

Adding a member — kubeadm-managed

For a kubeadm cluster, member addition is supported by kubeadm join on a control-plane node:

# On the new control-plane host, with the join token and CA cert hash:
kubeadm join 10.0.1.10:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:... \
  --control-plane \
  --certificate-key ...

kubeadm join --control-plane runs the etcd bootstrap under the hood: member add, copy certificates, write the static pod manifest, run a kubelet, and let the new member start.

$ kubectl get pods -n kube-system -l component=etcd -o wide

NAME          READY   STATUS    RESTARTS   AGE     IP          NODE
etcd-cp-1     1/1     Running   0          30d     10.0.1.10   cp-1
etcd-cp-2     1/1     Running   0          30d     10.0.1.11   cp-2
etcd-cp-3     1/1     Running   0          30d     10.0.1.12   cp-3
etcd-cp-4     1/1     Running   0          5m      10.0.1.13   cp-4

Removing a member

# Member ID from `etcdctl member list` (16 hex digits):
MEMBER_ID=8e9e05c52164694d

# On a healthy existing member (the leader is fine to issue this):
etcdctl member remove "$MEMBER_ID"

After removal, the cluster continues at N-1. Quorum recomputes on the next configuration change.

flowchart LR
    C[N=5 cluster] -->|member remove| C2[N=4 cluster, quorum=3]
    C2 -->|next config change| C3[quorum adjusted in C-space]

Replacing a member

The full replacement sequence combines remove + add:

1. Add the new member with etcdctl member add cp-new. Cluster is at N+1.
2. Wait for cp-new to be started.
3. Remove the old member with etcdctl member remove <old-id>. Cluster is at N.
4. Decommission the old host (delete data dir, stop the process).

sequenceDiagram
    autonumber
    participant Op as Operator
    participant Cluster
    participant NewHost as New host
    participant OldHost as Old host
    Op->>Cluster: member add cp-new
    Cluster->>NewHost: bootstrap join
    NewHost->>Cluster: raft applied
    Op->>Cluster: member remove old-id
    Op->>OldHost: stop etcd, decommission

A kubeadm-managed replacement uses a similar sequence under the hood; kubeadm does not currently support direct member-by-member replacement, so manual sequence on kubeadm is acceptable.

Retiring a member and decommissioning a host

The decommissioning sequence after a member has been removed from the cluster:

# On the old host, stop the kubelet and remove the static pod manifest
sudo crictl stop $(crictl ps -a --name etcd -q)
sudo mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.retired

# Verify the member is removed:
etcdctl member list --endpoints=https://10.0.1.10:2379,... \
  --cacert=... --cert=... --key=...
# Output should not include the retired member

The host can then be repurposed or decommissioned.

Member add timing and resource requirements

The bootstrap of a new member is heaviest in I/O:

• The new member downloads a snapshot from a peer.
• Snapshots can be hundreds of MB to multi-GB.
• The disk on the new host sustains this read; the leader’s disk sustains the read from its bbolt file.

A realistic timing:

Plan a maintenance window proportional to the DB size at bootstrap time. A weekend window for a 2 GiB DB is overkill; a 30-minute night window is right.

Members that won’t go started

The state where a member is registered but unstarted suggests:

• The new member’s data dir is empty (it was cleaned and the bootstrap flag was changed).
• The new member is failing to reach its peers.
• The new member’s clock is out of sync (--election- timeout works off clock-like timers).

Member retire checklist

Before retiring an etcd host:

• Remove the member from the cluster (member remove).
• Wait for the cluster’s quorum to readjust (next configuration change is the boundary).
• Stop the etcd process on the host.
• Optionally: wipe /var/lib/etcd and the certificates (the host will be decommissioned).
• Update monitoring to remove the host’s target.
• Update the kubeadm-config ConfigMap if the host was in the cluster’s inventory.

After retiring:

• Validate the cluster is at the expected member count (member list).
• Validate cluster is healthy (endpoint status, consistent raft indices).
• Update the runbook for the new member list.

Quiz

Production discipline

• Member changes are not high-frequency. A 3-to-5 growth is a quarterly-or-rarer event; member replacement is per-host, which is rarely.
• Member add is quorum-safe; member remove is quorum- safe. Both as long as you don’t end up with a cluster smaller than quorum in the moment.
• Always snapshot before a member change. Whether you add, remove, or replace, the snapshot is the rollback primitive.
• For kubeadm, use kubeadm join; for raw etcd, use etcdctl member add. The two paths differ in how certificates and manifests are managed; mixing them is the source of half the bugs.
• Document the member list in the runbook. The cluster’s member list is the foundation of how a new team member knows where the data is.

Member lifecycle is the operational surface most etcd incidents trace back to. The discipline is “one change at a time, snapshot before, observe after”.