Snapshot and restore etcd, and measure what the RPO cost

Objective

An etcd snapshot is the cheapest recovery point in a Kubernetes estate and the easiest one to misread. It restores cleanly, exits 0, and returns a cluster that looks correct — while quietly reverting everything that changed after the file was written.

You will run a real etcd 3.7.1, fill it with registry-shaped keys, snapshot it, change two things, destroy the data directory, and restore. Then you will count what returned. The measurement you leave with is not “the restore worked”: it is a named list of the keys that did not.

Architecture

The snapshot is a point on a revision timeline. Everything to the right of that point is the RPO, and nothing in the restore reports it.

flowchart TD
    W["53 keys written into rbdr-d1\nrevision 1 -> revision 54\norders replicas=6\ndb-password = S3cretValue, in the clear"] --> S["etcdctl snapshot save\nHASH 8c66d749 REVISION 54\nTOTAL KEYS 53  TOTAL SIZE 45 kB"]
    S --> G["the gap: work continues\nreplicas 6 -> 12\nconfigmaps/production/feature-flags created"]
    G --> D["data directory rbdr-d1 destroyed"]
    D --> R["etcdutl snapshot restore --data-dir rbdr-d2\ncluster id 1c45a069f3a1d796"]
    R --> V["53 keys back  secret readable\nreplicas=6 again\nfeature-flags ABSENT\nevery command exit 0"]
    G -.->|"reverted, silently"| V

The dotted edge is the only lossy step, and the only one with no output.

Requirements

Mode B-nested. The capture ran in a container: one etcd member on loopback, no TLS, no peers, no Kubernetes API server in front of it.
etcd, etcdctl and etcdutl from the upstream release tarball. All three ship in the same archive, which matters for the version fact in Task 5. Everything quoted below came from this build:

Read-only / Safethe etcd build every capture in this lab came from— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from the header of docs/courses/backup-dr/execution-evidence/etcd-snapshot-and-restore.txt, which recorded these four lines only.

$ etcd --version; etcdctl version

etcd Version: 3.7.1
Git SHA: 5e7fd0d
etcdctl version: 3.7.1
API version: 3.7

Roughly 200 MB of free space and TCP 2379/2380 free on loopback.
Everything this lab creates on disk is prefixed rbdr- and lives under one directory, so Cleanup can be scoped and asserted. The keys use the /registry/... layout a Kubernetes API server writes, because that is the shape whose restore behaviour is being measured.
The capture ran under /work as root, so its paths, member id, cluster ids and snapshot hash are its own. The counts, the reverted value and the missing key are the point, and those reproduce.

What that capture does and does not cover. The transcript records the running member, the keyspace, the snapshot and its status output, the two post-snapshot changes, the destroyed data directory, the restore, the restored member and the two cluster ids — the substance of Tasks 3 through 9. It does not record the tarball download in Task 2, the baseline in Task 1, the exit code etcdctl itself returned in Task 5, the flag string that produced the second cluster id, or the Cleanup diff. Those older embedded blocks retain their original provenance. A later complete run from Task 1 through Cleanup is captured in docs/courses/backup-dr/execution-evidence/backup-dr-lab-19-etcd-snapshot-and-restore-2026-08-29.txt and supports the last_executed date.

Scenario

A control plane is backed up by a nightly etcdctl snapshot save into object storage. The job has exited 0 every night for a year, and the runbook for control-plane loss was written when the cluster was built.

Today the node holding etcd lost its disk. You have the snapshot. This lab answers the three questions nobody asks until the incident: does the runbook’s restore command still exist, what does the restored cluster contain, and how would anyone know what it is missing.

Tasks

Task 1 — Record the pre-lab state

Cleanup is diffed against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-19"
mkdir -p "$LAB"

{
  command -v etcd || echo "no etcd on PATH"
  command -v etcdutl || echo "no etcdutl on PATH"
  pgrep -a etcd || echo "no etcd process running"
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/state.pre-lab"

On a clean host all four lines report absence — no etcd, no etcdutl, no running process, and an ls error for the glob. That absence is the baseline: at the end the same four lines have to report it identically.

Task 2 — Install etcd 3.7.1 and start one member

LAB="$HOME/rbdr-lab-19"
VER=v3.7.1
curl -fsSL -o "$LAB/rbdr-etcd.tar.gz" \
  "https://github.com/etcd-io/etcd/releases/download/$VER/etcd-$VER-linux-amd64.tar.gz"
mkdir -p "$LAB/rbdr-etcd"
tar xzf "$LAB/rbdr-etcd.tar.gz" -C "$LAB/rbdr-etcd" --strip-components=1
export PATH="$LAB/rbdr-etcd:$PATH"
etcd --version
etcdctl version

LAB="$HOME/rbdr-lab-19"
EP=127.0.0.1:2379
etcd --name default --data-dir "$LAB/rbdr-d1" \
  --listen-client-urls http://127.0.0.1:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster default=http://127.0.0.1:2380 \
  > "$LAB/rbdr-d1.log" 2>&1 &
sleep 3
etcdctl --endpoints="$EP" endpoint status --write-out=table

Read-only / Safeone member, empty, before anything is written to it— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from the '--- cluster up ---' section of the course capture etcd-snapshot-and-restore.txt; the capture's two-space indentation of command output has been removed.

$ etcdctl endpoint status --write-out=table

┌────────────────┬──────────────────┬─────────┬─────────────────┬─────────┬────────┬───────────────────────┬────────┬───────────┬────────────┬───────────┬────────────┬────────────────────┬────────┬──────────────────────────┬───────────────────┐
│    ENDPOINT    │        ID        │ VERSION │ STORAGE VERSION │ DB SIZE │ IN USE │ PERCENTAGE NOT IN USE │ QUOTA  │ IS LEADER │ IS LEARNER │ RAFT TERM │ RAFT INDEX │ RAFT APPLIED INDEX │ ERRORS │ DOWNGRADE TARGET VERSION │ DOWNGRADE ENABLED │
├────────────────┼──────────────────┼─────────┼─────────────────┼─────────┼────────┼───────────────────────┼────────┼───────────┼────────────┼───────────┼────────────┼────────────────────┼────────┼──────────────────────────┼───────────────────┤
│ 127.0.0.1:2379 │ b71f75320dc06a6c │   3.7.1 │           3.7.0 │   29 kB │  25 kB │                   15% │ 2.1 GB │      true │      false │         2 │          4 │                  4 │        │                          │             false │
└────────────────┴──────────────────┴─────────┴─────────────────┴─────────┴────────┴───────────────────────┴────────┴───────────┴────────────┴───────────┴────────────┴────────────────────┴────────┴──────────────────────────┴───────────────────┘

Note the member id b71f75320dc06a6c and the raft index. Both reappear in Task 8, and one of them comes back changed.

Task 3 — Write 53 registry-shaped keys, including a secret

EP=127.0.0.1:2379
etcdctl --endpoints="$EP" put /registry/deployments/production/orders 'replicas=6'
etcdctl --endpoints="$EP" put /registry/secrets/production/db-password 'S3cretValue'
for i in $(seq 1 51); do
  etcdctl --endpoints="$EP" put "/registry/pods/production/rbdr-app-$i" 'phase=Running'
done
etcdctl --endpoints="$EP" get / --prefix --keys-only | grep -c .
etcdctl --endpoints="$EP" get /registry/secrets/production/db-password

Read-only / Safewhat is in the keyspace, and what a snapshot of it will therefore contain— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 1 of the course capture etcd-snapshot-and-restore.txt, which printed the two reads as labelled summary lines rather than raw etcdctl output; the capture's two-space indentation has been removed.

$ etcdctl get --prefix --keys-only, then read the secret key back

keys written: 53
a secret is in there, in the clear:
/registry/secrets/production/db-password
S3cretValue

Fifty-three writes into a fresh member leave the revision counter at 54, the number Task 5 reports. The secret reads back as plaintext because etcd stores what it is given; encryption at rest is an API-server feature, and a snapshot inherits whatever was written.

Task 4 — Take the snapshot

LAB="$HOME/rbdr-lab-19"
EP=127.0.0.1:2379
etcdctl --endpoints="$EP" snapshot save "$LAB/rbdr-backup.db"
echo "snapshot save exit code: $?"

Read-only / Safea snapshot is streamed from the member and written as one file— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 2 of the course capture etcd-snapshot-and-restore.txt; the capture's two-space indentation of command output has been removed.

$ etcdctl snapshot save /work/backup.db

{"level":"info","ts":"2026-08-28T13:52:49.940818Z","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/work/backup.db.part"}
{"level":"info","ts":"2026-08-28T13:52:49.942078Z","logger":"client","caller":"v3/maintenance.go:236","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-28T13:52:49.942716Z","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"127.0.0.1:2379"}
{"level":"info","ts":"2026-08-28T13:52:49.942856Z","logger":"client","caller":"v3/maintenance.go:302","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-28T13:52:49.942907Z","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"127.0.0.1:2379","size":"45 kB","took":"1.568657ms","etcd-version":"3.7.0"}
{"level":"info","ts":"2026-08-28T13:52:49.942934Z","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/work/backup.db"}
Snapshot saved at /work/backup.db
Server version 3.7.0
>>> exit code: 0

The saved line is stamped 13:52:49.942934Z — one end of the interval you will measure in the Expected Outcome.

Task 5 — The failing case: the command the runbook says

LAB="$HOME/rbdr-lab-19"
OUT=$(etcdctl snapshot status "$LAB/rbdr-backup.db" 2>&1)
printf '%s\n' "$OUT"

printf '%s\n' "$OUT" | grep -q 'Usage:'
echo "help text was printed (0 = yes): $?"
printf '%s\n' "$OUT" | grep -q 'HASH'
echo "a status table was printed (1 = no): $?"

etcdutl snapshot status "$LAB/rbdr-backup.db" --write-out=table \
  | tee "$LAB/snapshot-status.txt"

Read-only / Safea runbook command that no longer exists on this version— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 2 of the course capture etcd-snapshot-and-restore.txt, truncated at the usage line; the capture's two-space indentation and its own commentary below that line are not included.

$ etcdctl snapshot status /work/backup.db

Manages etcd node snapshots

Usage:
etcdctl snapshot [command]

That is help text, not a status. On etcd 3.7.1 status and restore are no longer subcommands of etcdctl; they moved to the etcdutl binary shipped in the same tarball. A runbook written against an older release therefore fails at its first line, during a control-plane outage. This is the failing case, and the cheapest possible place to meet it.

The capture recorded that text but not the exit code the binary returned, which is why the two greps are the assertion: help text present, no HASH column. It is also the better test for a runbook, because a command that prints usage and still exits 0 is the version of this failure a set -e pipeline sails past.

Read-only / Safethe whole of the integrity evidence a snapshot file carries about itself— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 2 of the course capture etcd-snapshot-and-restore.txt; the capture's two-space indentation of command output has been removed.

$ etcdutl snapshot status /work/backup.db --write-out=table

┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 8c66d749 │       54 │         53 │      45 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

snapshot file size: 48K

TOTAL KEYS 53 and REVISION 54 reproduce exactly if you wrote the keys in Task 3. HASH 8c66d749 and TOTAL SIZE 45 kB are computed over the stored bytes, so they are the capture’s reference, not a target. These four values are the only self-description the file carries, and a hash compared against nothing proves nothing — which is why snapshot-status.txt has to leave the host.

Task 6 — Work continues after the snapshot

LAB="$HOME/rbdr-lab-19"
EP=127.0.0.1:2379
etcdctl --endpoints="$EP" put /registry/deployments/production/orders 'replicas=12'
etcdctl --endpoints="$EP" put /registry/configmaps/production/feature-flags 'checkout_v2=on'
{
  echo "two changes made AFTER the snapshot:"
  echo "  orders replicas -> replicas=12"
  echo "  new key         -> /registry/configmaps/production/feature-flags"
} | tee "$LAB/rpo-report.txt"

Configuration changetwo ordinary changes, made in the gap the snapshot cannot see— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 3 of the course capture etcd-snapshot-and-restore.txt, which summarised the two puts in labelled lines; the capture's two-space indentation has been removed.

$ etcdctl put the new replica count and a new ConfigMap key

two changes made AFTER the snapshot:
orders replicas -> replicas=12
new key         -> /registry/configmaps/production/feature-flags

A scale-up and a feature flag: the two most ordinary changes in a working day, which is exactly why their disappearance in Task 8 goes unremarked.

Task 7 — Destroy the data directory and restore

LAB="$HOME/rbdr-lab-19"
pkill -f "$LAB/rbdr-d1" || true
sleep 2
echo "about to remove: $LAB/rbdr-d1"
rm -rf "$LAB/rbdr-d1"
ls -d "$LAB/rbdr-d1" 2>&1 || echo "data directory removed: gone"

LAB="$HOME/rbdr-lab-19"
T0=$(date +%s)
etcdutl snapshot restore "$LAB/rbdr-backup.db" \
  --name default \
  --data-dir "$LAB/rbdr-d2" \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster default=http://127.0.0.1:2380
echo "restore exit code: $?"
etcd --name default --data-dir "$LAB/rbdr-d2" \
  --listen-client-urls http://127.0.0.1:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster default=http://127.0.0.1:2380 \
  > "$LAB/rbdr-d2.log" 2>&1 &
sleep 3
T1=$(date +%s)
echo "restore + start seconds: $((T1 - T0))" | tee -a "$LAB/rpo-report.txt"

Destructivethe restore writes a fresh data directory and a fresh raft history— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 5 of the course capture etcd-snapshot-and-restore.txt, abridged to four structural log lines plus the exit code; the capture's two-space indentation and its repeated bbolt open lines have been removed.

$ etcdutl snapshot restore /work/backup.db --data-dir /work/d2 ...

2026-08-28T13:52:51Z	info	snapshot/v3_snapshot.go:306	restoring snapshot	{"path": "/work/backup.db", "wal-dir": "/work/d2/member/wal", "data-dir": "/work/d2", "snap-dir": "/work/d2/member/snap", "initial-memory-map-size": 10737418240}
2026-08-28T13:52:51Z	info	schema/membership.go:125	Trimming membership information from the backend...
2026-08-28T13:52:51Z	info	membership/cluster.go:408	added member	{"cluster-id": "1c45a069f3a1d796", "local-member-id": "0", "added-peer-id": "b71f75320dc06a6c", "added-peer-peer-urls": ["http://127.0.0.1:2380"], "added-peer-is-learner": false}
2026-08-28T13:52:52Z	info	snapshot/v3_snapshot.go:334	restored snapshot	{"path": "/work/backup.db", "wal-dir": "/work/d2/member/wal", "data-dir": "/work/d2", "snap-dir": "/work/d2/member/snap", "initial-memory-map-size": 10737418240}
>>> exit code: 0

Trimming membership information from the backend and added member are the lines that matter: the restore discards the membership recorded in the snapshot, writes a new one from the flags you passed, and lays down a fresh write-ahead log. That is why the restored member’s raft index in the next table is lower than the original’s.

Task 8 — Count what came back, and what did not

LAB="$HOME/rbdr-lab-19"
EP=127.0.0.1:2379
etcdctl --endpoints="$EP" endpoint status --write-out=table
etcdctl --endpoints="$EP" get / --prefix --keys-only | grep -c .
etcdctl --endpoints="$EP" get /registry/secrets/production/db-password
etcdctl --endpoints="$EP" get /registry/deployments/production/orders
etcdctl --endpoints="$EP" get /registry/configmaps/production/feature-flags
echo "get on the post-snapshot key exit code: $?"

{
  echo "keys that did not come back:"
  echo "  /registry/configmaps/production/feature-flags (created after the snapshot)"
  echo "values the restore reverted:"
  echo "  /registry/deployments/production/orders  replicas=12 -> replicas=6"
} | tee -a "$LAB/rpo-report.txt"

Read-only / Safethe restored member: same id, smaller raft index, larger database— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from the '--- restored cluster ---' section of the course capture etcd-snapshot-and-restore.txt; the capture's two-space indentation of command output has been removed.

$ etcdctl endpoint status --write-out=table

┌────────────────┬──────────────────┬─────────┬─────────────────┬─────────┬────────┬───────────────────────┬────────┬───────────┬────────────┬───────────┬────────────┬────────────────────┬────────┬──────────────────────────┬───────────────────┐
│    ENDPOINT    │        ID        │ VERSION │ STORAGE VERSION │ DB SIZE │ IN USE │ PERCENTAGE NOT IN USE │ QUOTA  │ IS LEADER │ IS LEARNER │ RAFT TERM │ RAFT INDEX │ RAFT APPLIED INDEX │ ERRORS │ DOWNGRADE TARGET VERSION │ DOWNGRADE ENABLED │
├────────────────┼──────────────────┼─────────┼─────────────────┼─────────┼────────┼───────────────────────┼────────┼───────────┼────────────┼───────────┼────────────┼────────────────────┼────────┼──────────────────────────┼───────────────────┤
│ 127.0.0.1:2379 │ b71f75320dc06a6c │   3.7.1 │           3.7.0 │   45 kB │  41 kB │                   10% │ 2.1 GB │      true │      false │         2 │          3 │                  3 │        │                          │             false │
└────────────────┴──────────────────┴─────────┴─────────────────┴─────────┴────────┴───────────────────────┴────────┴───────────┴────────────┴───────────┴────────────┴────────────────────┴────────┴──────────────────────────┴───────────────────┘

Read-only / Safethe restore assessment: four reads, one of which returns nothing at exit 0— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 6 of the course capture etcd-snapshot-and-restore.txt, which printed the four reads as labelled summary lines rather than raw etcdctl output; the capture's two-space indentation has been removed.

$ etcdctl get the key count, the secret, the Deployment and the post-snapshot ConfigMap key

keys restored : 53
the secret    : S3cretValue
orders replicas: replicas=6
feature-flags key (written after the snapshot): ABSENT - this is the data loss the RPO describes

Read that block twice. Fifty-three keys is a success. The secret is a success. replicas=6 is also reported as a success by every command involved, and it is wrong: the Deployment was at 12 when the disk died. The ConfigMap key returns no rows at exit code 0, because an absent key is not an error in etcd. Nothing here will tell you what you lost; only the comparison against rpo-report.txt does.

Task 9 — Restore again, with a different membership

LAB="$HOME/rbdr-lab-19"
etcdutl snapshot restore "$LAB/rbdr-backup.db" \
  --name rbdr-a \
  --data-dir "$LAB/rbdr-d3" \
  --initial-advertise-peer-urls http://127.0.0.1:2480 \
  --initial-cluster rbdr-a=http://127.0.0.1:2480,rbdr-b=http://127.0.0.1:2580 \
  2>&1 | grep -o 'cluster-id": "[0-9a-f]*'

Read-only / Safecluster identity is derived from the flags, not carried in the file— EXECUTION VALIDATED on etcd 3.7.1. Byte-for-byte from STEP 7 of the course capture etcd-snapshot-and-restore.txt, including the capture's own indentation of these lines.

$ compare the cluster id from a restore with identical flags against one with a different --initial-cluster

--- cluster id, original vs restored-with-IDENTICAL-initial-cluster ---
original : 1c45a069f3a1d796
restored : 1c45a069f3a1d796
SAME

--- now restore the SAME snapshot with a DIFFERENT --initial-cluster ---
restored with a different initial-cluster : 80d54574493dd420
DIFFERENT from the original

The cluster id is computed from the --initial-cluster string, so restoring with identical parameters reproduces 1c45a069f3a1d796 and restoring with a different membership does not. The capture recorded 80d54574493dd420; the membership string it used is not in the transcript, so your value will be its own. What must hold is that it differs from the original — the assertion your runbook needs, because a mismatched cluster id is how a restored member gets rejected by peers that were not restored with it.

Validation

Every row names the command, the exact string, and the exit code expected.

Command	Expected output	Exit code
`etcd --version` (Task 2)	`etcd Version: 3.7.1` and `Git SHA: 5e7fd0d`	0
`etcdctl version` (Task 2)	`etcdctl version: 3.7.1` and `API version: 3.7`	0
`etcdctl endpoint status` (Task 2)	one row, `IS LEADER` `true`, `ERRORS` empty	0
`etcdctl get / --prefix --keys-only \| grep -c .` (Task 3)	`53`	0
`etcdctl get /registry/secrets/production/db-password` (Task 3)	`S3cretValue` on the value line	0
`etcdctl snapshot save` (Task 4)	`Snapshot saved at`, `Server version 3.7.0`	0
`etcdctl snapshot status ... \| grep -q 'Usage:'` (Task 5)	the captured text is `Manages etcd node snapshots` and `Usage:` — help text, because the subcommand moved to `etcdutl`	0 (the grep matched)
`etcdctl snapshot status ... \| grep -q 'HASH'` (Task 5)	nothing; there is no status table to match	1 (no match)
`etcdutl snapshot status --write-out=table` (Task 5)	`REVISION` `54`, `TOTAL KEYS` `53`, `VERSION` `3.7.0`; the capture also recorded `HASH` `8c66d749` and `TOTAL SIZE` `45 kB`	0
`etcdutl snapshot restore` (Task 7)	`Trimming membership information from the backend...`, `added member` with `"cluster-id": "1c45a069f3a1d796"`, then `restored snapshot`	0
`etcdctl endpoint status` after restore (Task 8)	same member id `b71f75320dc06a6c`, `DB SIZE` `45 kB`, `RAFT INDEX` `3` (lower than the pre-incident `4`)	0
`etcdctl get / --prefix --keys-only \| grep -c .` (Task 8)	`53`	0
`etcdctl get /registry/secrets/production/db-password` (Task 8)	`S3cretValue` — readable in the clear	0
`etcdctl get /registry/deployments/production/orders` (Task 8)	`replicas=6`, not `replicas=12`	0
`etcdctl get /registry/configmaps/production/feature-flags` (Task 8)	no output at all	0
`etcdutl snapshot restore` with a different `--initial-cluster`, piped to `grep -o` (Task 9)	one `cluster-id": "…` line, holding an id that is not `1c45a069f3a1d796`; the capture recorded `80d54574493dd420`	0 (the grep matched)

The two rows in bold are the lab: a wrong value and a missing key, both reported at exit code 0, by commands whose output carries no error string to alert on. Where a row asserts a grep exit code, that is deliberate: the capture holds the text those commands print, not the codes the binaries returned.

Expected Outcome

The restore succeeded at every step, and the cluster is wrong in two nameable ways.

Measure	Value
Keys before the snapshot	53, revision 54
Keys after the restore	53
Secret at rest	`S3cretValue`, readable without a key
Deployment after restore	`replicas=6`; it was `replicas=12` when the disk was lost
Post-snapshot ConfigMap key	absent; `etcdctl get` returns nothing at exit 0
Commands reporting an error	none
Member id	`b71f75320dc06a6c`, unchanged
Raft index	`4` before the incident, `3` after the restore — a fresh write-ahead log
Cluster id, identical flags	`1c45a069f3a1d796` (same as the original)
Cluster id, different `--initial-cluster`	different; the capture recorded `80d54574493dd420`
Actual restore time	record `restore + start seconds` from `rpo-report.txt`. In the capture the `etcdutl` restore alone spans `2026-08-28T13:52:51Z` to `2026-08-28T13:52:52Z` — about one second for a 45 kB snapshot — and starting etcd on the new directory is additional
Actual RPO observed	the capture’s snapshot was `saved` at `13:52:49.942934Z` and the restore began at `13:52:51Z`: roughly 1.1 seconds, and it still cost two changes. Record your own interval and the list of keys in `rpo-report.txt`. The duration is the schedule; the list is the loss

An RPO expressed only as “nightly” or “fifteen minutes” describes the interval and says nothing about what lives in it. Two puts in one second produced a scaled-down Deployment and a vanished feature flag; a nightly interval on a busy cluster contains thousands.

Troubleshooting

Symptom	Cause
`Manages etcd node snapshots` / `Usage:` from a status or restore attempt	The command was given to `etcdctl`. On 3.7.1 both subcommands live in `etcdutl`; confirm `command -v etcdutl` resolves inside the extracted tarball.
`etcdutl: command not found`	`PATH` was exported in a different shell. Re-run `export PATH="$LAB/rbdr-etcd:$PATH"`; all three binaries are in that one directory.
`context deadline exceeded` from every `etcdctl` call	The member is not listening. Read `$LAB/rbdr-d1.log` or `$LAB/rbdr-d2.log`; the commonest cause is port 2379 already bound by the previous etcd, which `pkill` in Task 7 did not stop.
`etcdutl snapshot restore` fails and the message names the `--data-dir` path	The target path is left over from an earlier attempt. Restore into a path that does not exist yet — this lab uses `rbdr-d2` and then `rbdr-d3` for exactly that reason — rather than deleting a directory under time pressure.
Key count comes back as 55, not 53	Task 6 ran before Task 4. The snapshot then contains the post-change state and there is no RPO to measure; restart from Task 2 with a fresh data directory.
`etcdctl get` on the ConfigMap key prints a value after the restore	etcd was restarted on `rbdr-d1` rather than `rbdr-d2`. Check the `--data-dir` in the running process with `pgrep -a etcd`.
Restored `HASH` or `TOTAL SIZE` differs from `8c66d749` / `45 kB`	Expected. Those are computed over the stored bytes and your key values are your own. `REVISION 54` and `TOTAL KEYS 53` are the two that must match.
The restored member is rejected by peers in a real cluster	The other members were not restored from this snapshot, so their cluster id and raft history disagree with the restored one. See the Production notes.

Cleanup

LAB="$HOME/rbdr-lab-19"
pkill -f "$LAB/rbdr-d" || true
sleep 2
rm -rf "$LAB"/rbdr-d1 "$LAB"/rbdr-d2 "$LAB"/rbdr-d3 "$LAB"/rbdr-etcd
rm -f "$LAB"/rbdr-etcd.tar.gz "$LAB"/rbdr-backup.db "$LAB"/rbdr-d1.log "$LAB"/rbdr-d2.log
hash -r

{
  command -v etcd || echo "no etcd on PATH"
  command -v etcdutl || echo "no etcdutl on PATH"
  pgrep -a etcd || echo "no etcd process running"
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/state.post-lab"

diff "$LAB/state.pre-lab" "$LAB/state.post-lab" \
  && echo "CLEAN: post-lab state matches the baseline recorded in Task 1"

The diff must print nothing and exit 0. Run it in the same shell that ran Task 1, so PATH is comparable. state.pre-lab, state.post-lab, snapshot-status.txt and rpo-report.txt are deliverables and are deliberately kept; none matches the rbdr-* glob, which is why the assertion holds. The snapshot itself is removed because it contains a plaintext secret.

Production notes

A restore resets raft history, so every member must be restored from the same snapshot. The restored member began from a freshly written write-ahead log, which is why its raft index dropped from 4 to 3. The procedure is: stop every member, restore each from the same snapshot file with its own --name and peer URL, then start them together.
Cluster identity comes from the restore flags. Identical --initial-cluster reproduced 1c45a069f3a1d796; a different membership produced a different id. Keep the exact flag string beside the snapshot: reconstructing it under pressure from a half-remembered topology is how a restore turns into a rebuild.
Record the four status values at backup time and put them somewhere else. A self-computed hash compared against nothing detects neither a substituted file nor a truncated overwrite.
The snapshot contains secrets in whatever form they were stored. This one handed back S3cretValue in plaintext, so snapshot files belong in encrypted, access-controlled storage on the same footing as the data directory — not in /tmp and not in a CI artefact.
Pin the tool name and version in the runbook, and rehearse it on the release you actually run. The help text in Task 5 is a correct-looking runbook silently invalidated by an upstream reorganisation; a rehearsal is the only cheap way to find that out.
Alert on the gap, not on the job. Every command here exited 0, including the read of a key that no longer existed. Control-plane recovery assurance has to compare the restored keyspace against an independent record of desired state — a GitOps repository, an inventory, a manifest export — because etcd has nothing to compare against.

What You Learned

A clean restore and a correct cluster are different claims. Fifty-three keys returned, the secret returned, every exit code was 0, and the Deployment was still wrong.
An absent key is not an error. etcdctl get on the ConfigMap written after the snapshot returned no rows at exit 0. Nothing in the tooling distinguishes “not there” from “never asked for”.
The RPO is a list, not a duration. Roughly one second of gap held two changes; the useful record was the enumeration of them, not the interval.
Tool names move. On etcd 3.7.1 snapshot status and snapshot restore answer only to etcdutl, and a runbook that says otherwise fails at line one of the recovery.
A snapshot carries its keyspace and its secrets, but not its identity. The cluster id came from the restore flags and the raft history was rebuilt from nothing.

Objectives

Prerequisites