Restarting the cluster — bringing etcd back up safely

After the snapshot is restored on each host, the next step is to start the static pods and watch the cluster form. The first member to start forms a single-member cluster; subsequent members join via the --initial-cluster peer list. Within seconds the cluster should be at three members with a leader. This lesson walks the per-host start and the cluster formation validation.

The start sequence

sequenceDiagram
    autonumber
    participant h1 as cp1
    participant h2 as cp2
    participant h3 as cp3
    Note over h1,h3: data dirs restored
    h1->>h1: start static pod
    h1->>h1: forms single-member cluster
    h1->>h1: leader elected
    h2->>h2: start static pod
    h2->>h1: contact via peer URL
    h2->>h1: snap install replay WAL
    h1->>h2: leader for current term
    h3->>h3: start static pod
    h3->>h1: contact via peer URL
    h3->>h1: snap install replay WAL

Each host’s etcd process is started by the kubelet from the static pod manifest. The kubelet observes the manifest file at /etc/kubernetes/manifests/etcd.yaml and starts the pod if the data dir is ready.

The per-host start

For a kubeadm-managed etcd, the start is automatic once the data dir is populated. The kubelet sees the static pod manifest and starts the pod.

# On each host, verify the manifest is intact:
sudo ls -la /etc/kubernetes/manifests/etcd.yaml
# Expected: file exists

# Verify the data dir is populated:
sudo ls -la /var/lib/etcd/
# Expected: snap/ wal/ member/ directories

For non-kubeadm etcd (systemd-managed), start the etcd service:

sudo systemctl start etcd
sudo systemctl status etcd
# Expected: active (running)

Member-by-member timing

The cluster forms in this order:

gantt
    title Cluster formation timeline
    dateFormat HH:mm
    axisFormat %H:%M
    section cp-1
    Start cp-1 :a1, 00:00, 10s
    Election :a2, after a1, 5s
    section cp-2
    Start cp-2 :b1, 00:30, 10s
    Snap install :b2, after b1, 30s
    section cp-3
    Start cp-3 :c1, 01:30, 10s
    Snap install :c2, after c1, 30s

In practice:

• cp-1 starts in seconds; it elects itself leader for term 1.
• cp-2 starts, contacts cp-1, installs the snapshot (WAL replay), and joins as follower.
• cp-3 starts similarly, joins as follower.

The total time: ~30-60 seconds for the cluster to reach 3 members; ~1-2 minutes for all members to apply the WAL to the leader’s commitIndex.

$ etcdctl --endpoints=https://10.0.1.10:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table

+---------------------------+------------------+---------+---------+
|         ENDPOINT          |        ID        | IS LEADER | ERRORS |
+---------------------------+------------------+---------+---------+
| https://10.0.1.10:2379    | <new-id-1>      |   true   |         |
| https://10.0.1.11:2379    | <new-id-2>      |  false   |         |
| https://10.0.1.12:2379    | <new-id-3>      |  false   |         |
+---------------------------+------------------+---------+---------+

The IDs are new (not the original IDs); the cluster has a new identity but the same data.

Member discovery via —initial-cluster

Each member starts with a flag listing the entire cluster membership:

--initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380

The mechanism:

sequenceDiagram
    autonumber
    participant M as cp-2 (just started)
    participant L as cp-1 (leader)
    Note over M: --initial-cluster names<br/>cp-1, cp-2, cp-3
    M->>L: dial https://10.0.1.10:2380
    L->>M: AppendEntries, possibly snapshot
    M->>M: install snapshot in bbolt
    L->>M: AppendEntries (continue log)
    M->>L: ack index N

The member knows where to find peers; the leader knows it is the leader; the cluster converges.

Verify cluster state before restarting API server

Before restarting the API server:

# 1. The cluster has 3 members
etcdctl member list --write-out=table

+------------------+---------+-------+-------------------------+-------------------+
|        ID        | STATUS  | NAME |       PEER ADDRS        |      CLIENT ADDRS |
+------------------+---------+-------+-------------------------+-------------------+
| <new-id-1>      | started | cp-1  | https://10.0.1.10:2380 | https://10.0.1.10:2379 |
| <new-id-2>      | started | cp-2  | https://10.0.1.11:2380 | https://10.0.1.11:2379 |
| <new-id-3>      | started | cp-3  | https://10.0.1.12:2380 | https://10.0.1.12:2379 |
+------------------+---------+-------+-------------------------+-------------------+

# 2. The leader is elected
etcdctl endpoint status --write-out=table

A consistent RAFT INDEX across members indicates the log has been replicated.

# 3. The cluster serves a read of expected objects
etcdctl get /registry/namespaces/prod-app --keys-only

If prod-app was a Namespace in the cluster, this returns the key. If not, an empty response is correct.

The cluster is in --initial-cluster-state=existing mode

This is the flag that matters at startup, distinct from the --initial-cluster-state=new flag used at first bootstrap of a brand-new cluster. The kubeadm-managed static pod manifest sets --initial-cluster-state=existing because the cluster already exists; the etcd process is just adding members to it.

After the first start, the flag is irrelevant — it is used once at bootstrap to decide whether to form a new cluster or join an existing one.

What to watch for

Stopping etcd before the API server starts

The etcd processes must be running before the API server restarts. The API server talks to etcd on startup; if etcd is down, the API server cannot read cluster state.

# On each host:
sudo crictl pods | grep etcd
# Expected: each etcd pod is Running

If a host’s etcd is not running, debug that host before restarting the API server.

Restarting the API server after etcd is ready

Once the etcd cluster is healthy, the API server manifests are restored:

# On each control-plane host:
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml.stopped \
        /etc/kubernetes/manifests/kube-apiserver.yaml

sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml.stopped \
        /etc/kubernetes/manifests/kube-controller-manager.yaml

sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml.stopped \
        /etc/kubernetes/manifests/kube-scheduler.yaml

The kubelet picks up the manifest and starts the static pod. Within seconds, the API server is back.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant A as kube-apiserver pod
    participant E as etcd
    K->>A: start (manifest returned)
    A->>E: connect to https://10.0.1.10:2379, etc.
    E->>A: serve objects
    A->>K: ready

The first API server may take 10-30 seconds to load the cluster’s state from etcd (it materialises the watch cache). Subsequent API servers are faster.

The cluster’s new member IDs

A subtle property: the restored cluster has new member IDs, even though the names and peer URLs match. Any external system that pinned to member IDs must update.

# Compare old vs new member IDs
# Old: c5e9a1b2..., d7a1c8f3..., e8b4d2a5...
# New: <new-id-1>, <new-id-2>, <new-id-3>

For most clusters, member IDs are not pinned anywhere outside of etcd. For clusters with external monitoring that includes member IDs, update the pinned IDs after the restore.

Quiz

Production discipline

• Bring etcd up before the API server. The API server reads from etcd on startup.
• Validate the cluster before proceeding. Member count, leader election, consistent raft indices.
• Confirm member discovery is correct. The new member IDs may differ from the original; update any external monitors that pinned them.
• First member forms alone briefly. The single-member state is expected during the sequential bootstrap.
• Don’t skip validation. A cluster that “looks up” may have a member that cannot replicate; check raft indices.

The cluster’s formation is the first validation that the restore succeeded. The next lesson is the production readiness checks.