Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-etcd-restore~55 min

The etcd snapshot restored and the cluster would not form

Reported symptoms

  • ●A power event at 02:14 on 2026-08-27 destroyed local storage on all three stacked control-plane nodes, rbdr-cp-1, rbdr-cp-2 and rbdr-cp-3, which sat in one rack
  • ●The nightly etcd snapshot is present in object storage, downloads without error onto all three rebuilt nodes, and is byte-identical on each of them
  • ●The first restore command in the control-plane-loss runbook prints help text instead of restoring anything, on all three nodes
  • ●After the team works around that, all three restores complete, all three etcd processes start, and all three stay up with no crash loop
  • ●No member ever reports a leader. Every kubectl call times out and the API server restarts continuously because it has no reachable etcd endpoint
  • ●Queried directly on loopback, each member answers, holds the full keyspace, and describes a cluster that does not include the other two
  • ●Restarting the members in different orders and waiting through several election timeouts changes nothing, and the peer logs fill with rejected connections rather than with election traffic

Evidence

  • · All three rebuilt nodes run the same build: etcd Version 3.7.1, Git SHA 5e7fd0d, etcdctl version 3.7.1, API version 3.7
  • · The runbook step fails identically on each node: the command prints "Manages etcd node snapshots" and a "Usage:" line, because on this version status and restore are etcdutl subcommands rather than etcdctl subcommands
  • · etcdutl snapshot status on the downloaded file reports HASH 8c66d749, REVISION 54, TOTAL KEYS 53, TOTAL SIZE 45 kB and VERSION 3.7.0, matching the four values the backup job recorded next to the object at 01:00
  • · rbdr-cp-3 on its own is healthy: one member, IS LEADER true, ERRORS empty, all 53 keys readable, and /registry/secrets/production/db-password returning S3cretValue in the clear
  • · Each of the three restore logs contains exactly one "added member" line, and the cluster-id field in those three lines holds three values that are different from one another
  • · The three --initial-cluster strings recovered from shell history differ: cp-1 names rbdr-cp-3 at the peer address it had before it was rebuilt in March, cp-2 names it at its current address, and cp-3 names only itself
  • · A reproduction on a scratch host restored the same snapshot twice with an identical --initial-cluster and derived 1c45a069f3a1d796 both times, then restored it once more with a different --initial-cluster and derived 80d54574493dd420
  • · The runbook rbdr-runbook-control-plane-loss says to restore the snapshot on each control-plane node and shows one example command. It never says that every member must be restored from the same snapshot file, and it does not mention --initial-cluster at all
  • · The runbook carries a last-reviewed date of 2024-11-05, and no job in the estate compares it against the installed etcd version or exercises it on disposable nodes
Diagnosis and resolutionclick to reveal

Root cause

Two defects. Three members were given three different membership configurations at restore time, and the document that drove the restore had never been written to prevent that. The immediate cause is that cluster identity in etcd is computed from the `--initial-cluster` string supplied to the restore, not read out of the snapshot. The capture taken for this course demonstrates both halves of that statement on one file: restoring it twice with an identical `--initial-cluster` produced `1c45a069f3a1d796` on both runs, and restoring the same file with a different `--initial-cluster` produced `80d54574493dd420`. Three engineers worked three nodes in parallel to save wall-clock time, and each took the membership list from a different place. One copied it from the runbook, which was written when the cluster was built and still names rbdr-cp-3 at the peer address it lost in March. One read it from the current Ansible group variables. One, on a console with no repository checkout, typed the member being restored and nothing else, on the reasonable-sounding grounds that this member was the cluster being created. Three strings, three cluster ids, and no two members that will accept each other as peers. Every restore exited zero and every etcd process started, because nothing in a restore validates a membership list against anyone else. The second defect is the control that should have made this impossible. The runbook says to restore the snapshot on each control-plane node and gives one example command. That sentence is true and useless: it does not say that all three restores must consume the same snapshot file, it does not say that `--initial-cluster` must be byte-identical on all three, and it does not say which flags are the per-member ones. It also still names a command that this etcd version does not have, so it fails at its first line and pushes the operator into improvising exactly the part that had to be uniform. A runbook that has not been exercised since 2024 against the version now installed is not a control; it is a document.

Remediation

Stop restoring. Three divergent members are three separate one-member clusters, and any further restart ordering, election waiting or peer-list editing only adds states to unpick. Nothing here is lost, so nothing needs to be rushed. Choose one snapshot file, on one node, and make it the only file used. Record its `etcdutl snapshot status` output and its checksum, then copy that exact file to the other two nodes and confirm the checksum matches on arrival. The three downloads were already byte-identical, but from this point the estate depends on it, so it is asserted rather than assumed. Write the single `--initial-cluster` string down before running anything. It is derived from the current peer addresses of the three rebuilt nodes, checked against the Ansible group variables and against what each node actually has on its interfaces, and it is the same string on all three commands. The only flags that differ per node are `--name` and `--initial-advertise-peer-urls`. Then discard all three restored data directories and restore again, from that one file, with that one membership string, into a fresh directory on each node. Read the cluster-id out of all three restore logs before starting a single etcd process. If the three ids are not identical, the restore is already wrong and starting etcd will only hide it behind a peer error. When they match, start the members and let them elect. Only after a leader exists and the API server is serving does the estate move on to the second question, which the restore cannot answer: what changed between the snapshot and 02:14 and is therefore no longer in the cluster.

Verification

The restore is verified before etcd starts, not after. Each restore log holds one line reporting the member it added and the cluster-id it derived; extract that field from all three logs and assert that the set has exactly one distinct value. This is the check that would have ended the incident at 03:00, and it costs one grep per node. The snapshot is verified against what was recorded at backup time. HASH 8c66d749, REVISION 54 and TOTAL KEYS 53 were written next to the object when the job ran, and etcdutl reports the same four values on the downloaded copy. A status output that differs from the recorded one is a different file, not a different opinion. The cluster is verified by membership and leadership together. One member list containing three ids, the same list from all three endpoints, exactly one endpoint reporting IS LEADER true, and an empty ERRORS column. A member that answers on loopback while describing a cluster of one is the failure this scenario is about, and endpoint status from a single node cannot distinguish the two. The service is verified last: the API server serving, nodes registering, and a known object read back by name. And the recovery is only complete when the gap between the snapshot and the incident has been enumerated and reported, because a control plane that comes back at yesterday-at-01:00 is a working cluster with the wrong contents.

Prevention

**Make the membership string an artefact, not a keystroke.** Generate the `--initial-cluster` value from the inventory into one file, distribute that file, and have every restore read it. Three people typing the same idea from three sources is the whole incident. **Restore every member from the same file, and prove it.** One snapshot, checksummed on arrival at each node. It costs one command and removes an entire class of divergence that no error message will report. **Assert on the cluster id before starting anything.** The restore prints it. Comparing three strings is the cheapest gate in this runbook and the only one that catches this failure while it is still free to fix. **Exercise the runbook against the version that is installed.** This one named a command that had moved to a different binary, and nobody found out until the control plane was down. A restore drill on disposable nodes, on the current version, converts that into a failed drill. **Write runbooks that name the invariants, not just the steps.** "Restore the snapshot on each node" describes three actions and no constraint. "The same file, the same membership string, and only --name and the peer URL differing" is the sentence that had to be there. **Rebuild control-plane nodes serially unless the parallel path has been rehearsed.** Parallelism saved perhaps twenty minutes and cost six hours, because the coordination it depends on existed only in the heads of three people working separate consoles.

Reported symptoms

At 02:14 on 2026-08-27 a rack lost power and took the local storage of all three stacked control-plane nodes with it. The nightly etcd snapshot is in object storage, downloads cleanly, and is byte-identical on the three rebuilt nodes.

The runbook’s restore command prints help text on every node. The team works around that, and by 03:40 all three restores have completed and all three etcd processes are running.

Nothing else works. There is no leader, kubectl times out, and the API server restarts in a loop. Queried on loopback each member answers, holds the whole keyspace, and describes a cluster that does not contain the other two.

Evidence provided

Start with what is installed, because the first failure was a version fact.

Read-only / Safethe build on all three rebuilt nodes
$ etcd --version; etcdctl version
etcd Version: 3.7.1
Git SHA: 5e7fd0d
etcdctl version: 3.7.1
API version: 3.7

The runbook’s first restore line produces this, on each node in turn.

Read-only / Safethe runbook command, on the version that is installed
$ etcdctl snapshot status /work/backup.db
Manages etcd node snapshots

Usage:
etcdctl snapshot [command]

That is help text. On 3.7.1 both status and restore have moved out of etcdctl and into the etcdutl binary in the same tarball.

Read-only / Safethe four values the backup job also recorded at 01:00
$ etcdutl snapshot status /work/backup.db --write-out=table
┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 8c66d749 │       54 │         53 │      45 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

Every restore log holds one line about membership, and it is the line that matters.

Configuration changewhat a restore decides about cluster identity
$ etcdutl snapshot restore /work/backup.db --data-dir /work/d2
2026-08-28T13:52:51Z	info	schema/membership.go:125	Trimming membership information from the backend...
2026-08-28T13:52:51Z	info	membership/cluster.go:408	added member	{"cluster-id": "1c45a069f3a1d796", "local-member-id": "0", "added-peer-id": "b71f75320dc06a6c", "added-peer-peer-urls": ["http://127.0.0.1:2380"], "added-peer-is-learner": false}

And the reproduction on a scratch host: the same file, restored twice with the same membership flags and once with different ones.

Read-only / Safethe same snapshot, two membership configurations, two identities
$ etcdutl snapshot restore /work/backup.db --data-dir /work/d4 --initial-cluster rbdr-b=http://127.0.0.1:2580
--- cluster id, original vs restored-with-IDENTICAL-initial-cluster ---
original : 1c45a069f3a1d796
restored : 1c45a069f3a1d796
SAME

--- now restore the SAME snapshot with a DIFFERENT --initial-cluster ---
restored with a different initial-cluster : 80d54574493dd420
DIFFERENT from the original

Finally, what one member alone contains.

Read-only / Safea single-member restore from the same file, complete
$ etcdctl get / --prefix --keys-only | grep -c .
keys restored : 53
the secret    : S3cretValue

Work the evidence before reading on

  1. The status table matches the hash, revision and key count recorded at 01:00. Which hypothesis does that retire, and which does it leave?
  2. rbdr-cp-3 alone is healthy and serves all 53 keys. What does that prove about the file, and what about the procedure?
  3. Three restore logs, three different values in one field, one set of bytes. Where did those values come from?
  4. Which flags must be identical across three members, and which must differ?
  5. Write the sentence the runbook needed, and say why “restore the snapshot on each control-plane node” reads as though it already says it.

Root cause

The alternative worth eliminating first

The natural first suspicion is the backup: a truncated download, a partial upload, a file that is not what the job thought it saved. Two readings retire it.

etcdutl snapshot status reports HASH 8c66d749, REVISION 54, TOTAL KEYS 53 and TOTAL SIZE 45 kB — the four values recorded beside the object when the job ran. The second is decisive because it is a restore rather than an inspection: rbdr-cp-3 came up on its own, healthy, with all 53 keys and the secret readable in the clear. A file that restores into a serving cluster once will do it three times.

The cluster id is derived at restore time

The measurement is on one file. Restored twice with an identical --initial-cluster it produced 1c45a069f3a1d796 both times; restored once more with a different --initial-cluster the same bytes produced 80d54574493dd420. Identity is computed from the membership configuration handed to the restore, not carried inside the snapshot, so it is not something a backup can get right for you.

Three engineers worked the three nodes in parallel, and each sourced the membership list separately. One copied it from the runbook, which still names rbdr-cp-3 at the peer address it lost when it was rebuilt in March. One read it from the current Ansible group variables. The third, on a console with no checkout, typed the member in front of him and nothing else, reasoning that this member was the cluster being created.

Three strings, three cluster ids, and members with different cluster ids do not accept each other as peers. Nothing in a restore compares its membership list with anyone else’s, because at restore time there is no one else to ask.

The runbook described actions and no invariants

rbdr-runbook-control-plane-loss says to restore the snapshot on each control-plane node and shows one example command. Every word is true. It does not say the three restores must consume the same file, that --initial-cluster must be byte-identical, or which flags are the per-member ones. A reader who follows it exactly can still produce this.

It also names a command this version does not have, so it fails at its first line and hands the operator an improvisation task at the worst moment. Its last-reviewed date is 2024-11-05.

Resolution

Stop restoring. Three divergent members are three one-member clusters, and more restart ordering only adds state to unpick. Fix the file first, then the string.

SNAP=/srv/rbdr-restore/rbdr-etcd-2026-08-27.db
etcdutl snapshot status "$SNAP" --write-out=table
sha256sum "$SNAP"

That output is compared with what the backup job recorded, and the same checksum is confirmed on each node after the file is copied there.

Then write the membership string down once, derived from the current peer addresses and checked against both the inventory and the interfaces the nodes have. It is identical on all three commands; only --name and the peer URL differ.

SNAP=/srv/rbdr-restore/rbdr-etcd-2026-08-27.db
CLUSTER=$(cat /srv/rbdr-restore/rbdr-initial-cluster.txt)
NAME=$(hostname -s)
etcdutl snapshot restore "$SNAP" \
  --name "$NAME" \
  --data-dir /var/lib/rbdr-etcd-restored \
  --initial-advertise-peer-urls "https://$NAME.rbdr.internal:2380" \
  --initial-cluster "$CLUSTER"

Discard the three earlier data directories, restore into fresh ones, and read the cluster id out of the three logs before starting any etcd process.

grep -ho 'cluster-id": "[0-9a-f]*' /srv/rbdr-restore/rbdr-restore-cp-*.log \
  | sort -u \
  | tee /srv/rbdr-restore/rbdr-cluster-id-check.txt \
  | wc -l

One line means the three restores agree. Anything else means the restore is already wrong, and starting etcd will only bury it under a peer error.

Verification

Verify before etcd starts: the check above is one grep per node, and it is the gate that would have ended this at 03:00.

Verify the file against what was recorded at backup time, not against itself — hash 8c66d749, revision 54, key count 53. A status output that differs from the recorded one is a different file.

Verify the cluster by membership and leadership together — one member list of three ids, the same list from all three endpoints, one endpoint reporting IS LEADER true, an empty ERRORS column. Endpoint status from a single node cannot tell a healthy member of three from a healthy member of one, which is the confusion this incident lived in for six hours.

Verify the service last, then enumerate the gap between the snapshot and 02:14 and report it: a control plane restored to yesterday at 01:00 is a working cluster with the wrong contents.

Prevention

Make the membership string an artefact. Generate --initial-cluster from the inventory into one file, distribute it, and have every restore read it. Three people typing the same idea from three sources is the entire incident.

One snapshot file, checksummed on arrival at every node. One command, and a whole class of divergence that no error message reports disappears.

Assert on the cluster id before starting anything. The restore prints it, and comparing three strings catches this while it is still free to fix.

Exercise the runbook on the version that is installed. This one named a command that had moved to another binary, and nobody found out until the control plane was down. A drill on disposable nodes turns that into a failed drill.

State invariants, not only steps. “Restore the snapshot on each control-plane node” describes three actions and no constraint. “The same file, the same membership string, only the name and peer URL differing” is the sentence that had to be there.