etcd: snapshot and restore · Backup & DR

The previous lesson split a Kubernetes backup into pieces that fail independently and put etcd at the centre of exactly one of them: the cluster’s own record of what is supposed to exist. This lesson takes that piece apart on a real binary. etcd is not a database you protect the way you protect an application’s data. It is the store a consensus protocol writes into, so restoring it is an act of re-declaring the cluster’s membership as much as an act of returning keys — and almost every fact the capture behind this lesson established is one that a runbook written two releases ago now gets wrong.

The subcommand your runbook names may no longer exist

The capture ran on etcd 3.7.1 (Git SHA 5e7fd0d, etcdctl 3.7.1, API version 3.7) against a single member with id b71f75320dc06a6c on 127.0.0.1:2379, holding 53 keys written to look like a small production cluster.

Taking the snapshot worked exactly as every guide describes it. etcdctl snapshot save /work/backup.db streamed the keyspace out of the running member, logged "size":"45 kB" and "took":"1.568657ms", printed Snapshot saved at /work/backup.db and Server version 3.7.0, and exited 0. Nothing on the backup side of this has moved. Reading the file back is where the ground gave way.

Read-only / Safeasking the snapshot to describe itself, the way a 3.5-era runbook does— EXECUTION VALIDATED on etcd 3.7.1 (etcdctl 3.7.1, API version 3.7), against the snapshot taken in step 2 of the capture.

$ etcdctl snapshot status /work/backup.db

  Manages etcd node snapshots

Usage:
  etcdctl snapshot [command]

That is the usage text of the parent command. No status table, no hash, and no message naming an unknown subcommand — just help. On this build, status and restore are no longer etcdctl subcommands at all. They moved to a separate binary, etcdutl, and the split follows a real boundary: etcdctl speaks to a running cluster over the API, while etcdutl operates on a data directory or a snapshot file on disk. As design that is defensible. As an operational change it is invisible until the exact moment it is not.

The runbook that was correct on the day it was written

A disaster-recovery runbook is written once, reviewed by people who already agree with it, and then executed years later by whoever happens to be on call. If it still says etcdctl snapshot restore — a form that no longer exists on this version — the first command of the recovery prints usage text at three in the morning, and the person running it now has to decide whether they mistyped, whether the binary is missing, or whether the snapshot is damaged, with the control plane down and a clock running.

The remedy is not to memorise which binary owns which verb this year. It is to record in the runbook the version it was validated against, and to re-execute the runbook whenever either that version or the installed one changes. This course measured etcdutl on etcd 3.7.1; on an older cluster the etcdctl form is the correct one, which is precisely why the version has to be written down next to the command.

`8c66d749`, revision 54, 53 keys: everything the file says about itself

Pointed at the right binary, the same question produces the table the runbook wanted.

Read-only / Safethe integrity evidence an etcd snapshot carries— EXECUTION VALIDATED on etcd 3.7.1. Step 2 of the capture, run against /work/backup.db immediately after it was written; the trailing file-size line is the capture script's.

$ etcdutl snapshot status /work/backup.db --write-out=table

  ┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 8c66d749 │       54 │         53 │      45 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

snapshot file size: 48K

Five columns, each answering a different question. HASH is a checksum over the snapshot’s contents and is the only value in the file that later lets you say the bytes are the bytes you took. REVISION is etcd’s global revision counter at the instant of the snapshot — 54 here — which is a monotonic logical clock rather than a timestamp, so comparing two snapshots’ revisions orders them without depending on either machine’s wall clock. TOTAL KEYS is 53. TOTAL SIZE is 45 kB, the same figure snapshot save logged as "size" for the stream it fetched, while the file itself occupied 48K on disk. VERSION reads 3.7.0 — the value the running endpoint reported in its STORAGE VERSION column, not the 3.7.1 the binaries report as their own version.

Those five values are the entire self-description of an etcd snapshot. There is no manifest, no catalogue and no embedded statement of which cluster produced it. Whatever you fail to record at backup time is not recoverable from the file later, so record it at backup time:

SNAP=/var/backups/etcd/etcd-$(date -u +%Y%m%dT%H%M%SZ).db
etcdctl --endpoints=127.0.0.1:2379 snapshot save "$SNAP"
etcdutl snapshot status "$SNAP" --write-out=table > "$SNAP.status.txt"
sha256sum "$SNAP" > "$SNAP.sha256"

Reading the status at backup time and keeping it beside the file turns an opaque blob into something you can make a statement about: this file held 53 keys at revision 54 and hashed to 8c66d749 when it was written. Reading it only during the incident tells you what the file says about itself now, which is a different and much weaker claim.

A 48K file that holds every Secret in plaintext

The capture wrote one key specifically to make this visible: /registry/secrets/production/db-password, carrying the value S3cretValue, and recorded that it sat in the cluster in the clear. After the data directory was destroyed and the snapshot restored into a new one, the same value read back unchanged.

That is correct behaviour, and it is also the whole security story of a snapshot file. Unless encryption at rest is configured on the API server, an etcd snapshot contains every Secret in the cluster in readable form, which means the file inherits the sensitivity of the most sensitive Secret it contains. Whether to encrypt at rest is a separate decision from whether to take snapshots, and it has to be made deliberately: the snapshot mechanism will not make it for you, and nothing in the status output hints at the exposure.

What the restore actually did to the data directory

With the data directory removed, recovery is a single etcdutl invocation writing a new data directory rather than repairing the old one.

Cluster-wide riskthe tail of the restore log, where the membership is rebuilt— EXECUTION VALIDATED on etcd 3.7.1. Step 5 of the capture, restoring /work/backup.db into a fresh data directory after the original was destroyed; the block is the tail of the command's own log.

$ etcdutl snapshot restore /work/backup.db --data-dir /work/d2 ...

  2026-08-28T13:52:51Z	info	membership/cluster.go:408	added member	{"cluster-id": "1c45a069f3a1d796", "local-member-id": "0", "added-peer-id": "b71f75320dc06a6c", "added-peer-peer-urls": ["http://127.0.0.1:2380"], "added-peer-is-learner": false}
2026-08-28T13:52:52Z	info	bbolt	backend/backend.go:213	Opening db file (/work/d2/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, MaxSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0x202f26b8a068, NoStatistics: false}
2026-08-28T13:52:52Z	info	bbolt	bbolt@v1.5.0/db.go:327	Opening bbolt db (/work/d2/member/snap/db) successfully
2026-08-28T13:52:52Z	info	snapshot/v3_snapshot.go:334	restored snapshot	{"path": "/work/backup.db", "wal-dir": "/work/d2/member/wal", "data-dir": "/work/d2", "snap-dir": "/work/d2/member/snap", "initial-memory-map-size": 10737418240}
>>> exit code: 0

Just above that tail the log records Trimming membership information from the backend..., which is the restore discarding the membership held in the file it opened. What replaces it is the added member line, carrying "cluster-id": "1c45a069f3a1d796", "local-member-id": "0", "added-peer-id": "b71f75320dc06a6c" and the peer URL http://127.0.0.1:2380. Trim-then-add is the whole membership sequence: the restore does not read a membership out of the snapshot and keep it, it removes one and writes another. Where the replacement comes from is a question the capture answers later, by measurement, and the answer changes what a matching cluster id is worth as evidence. The command then reports restored snapshot and exits 0.

Started against /work/d2, the member served the restored keyspace: DB SIZE 45 kB where the original had reported 29 kB, the same member id, and RAFT TERM 2 in both. One column did not agree. The original endpoint reported RAFT INDEX 4 and RAFT APPLIED INDEX 4; the restored one reported 3 and 3. The keyspace came back and the log did not.

Why a restore resets the raft history

etcd keeps two things on disk. One is a bbolt database holding the keyspace as it currently stands — the capture’s log shows it being opened at /work/d2/member/snap/db. The other is a write-ahead log of raft entries, the ordered record of every decision the cluster’s members agreed on, which is what makes a member able to catch up with peers by replaying what it missed.

A snapshot is a copy of the first and none of the second. It is the keyspace at one revision, with no raft entries and no record of the agreement that produced them. So etcdutl snapshot restore cannot reconstruct a member; it has to manufacture one. It opens the bbolt file, trims the membership recorded inside it, writes a new membership derived from the flags it was given, and writes a brand-new WAL under /work/d2/member/wal containing exactly enough to let the member believe it starts here. That is why the restored endpoint reported RAFT INDEX 3 where the original reported 4: the index counts entries in a log that has just been rewritten, so the two numbers are measured against different histories and are not comparable.

The consequence is the operational rule the rest of this lesson depends on. A member whose log has been rewritten cannot be reconciled with a member whose log was not, because raft’s whole safety argument rests on members agreeing about what entry number n was. There is no merge, no catch-up and no reconciliation step available; there is only a set of members that all start from the same manufactured history, or a set that disagrees.

`replicas=6`, and nothing said a word

Between the snapshot and the disaster, the capture did what a production cluster does: it changed. The orders Deployment was scaled to replicas=12, and a new key /registry/configmaps/production/feature-flags was written. Then the data directory was destroyed and the snapshot restored.

Data-loss riskthe restored cluster, compared against the state that was lost— EXECUTION VALIDATED on etcd 3.7.1, against the member started from the restored data directory. Step 6 of the capture, which reads the key count, the secret, the orders Deployment and the ConfigMap key in turn; the four lines and their labels are the capture script's summary of those reads, not the stdout of any single command.

$ etcdctl get /registry/secrets/production/db-password; etcdctl get /registry/configmaps/production/feature-flags

  keys restored : 53
the secret    : S3cretValue
orders replicas: replicas=6
feature-flags key (written after the snapshot): ABSENT - this is the data loss the RPO describes

Read those four lines in the order an operator would. The key count is 53 — which matches TOTAL KEYS in the snapshot status exactly, and is therefore a statement about the snapshot rather than about the cluster. The secret came back intact. And then the two lines that matter: the Deployment came back at replicas=6, and the ConfigMap key written after the snapshot was simply ABSENT.

No command failed. No exit code was non-zero. Nothing in etcd, in the API server or on any dashboard is capable of reporting the difference, because from etcd’s point of view there is no difference: revision 54 was restored faithfully, and revision 54 is what the file contained. This is what an RPO looks like when you finally see one — not an error, but an absence, arriving in the units the business actually uses. A service that should be running twelve replicas is running six, and a feature flag that existed an hour ago does not.

The companion capture on k3s v1.36.3+k3s1 makes the boundary sharper still. A 1208352-byte cluster-state snapshot taken there was recorded as protecting “Kubernetes objects only”, with “every byte in every PersistentVolume” outside it: when the namespace was deleted and rebuilt, every object returned and the application’s orders.csv did not. An etcd snapshot is a backup of the cluster’s mind, and the mind is not where the orders live.

`1c45a069f3a1d796` and `80d54574493dd420`: where a cluster id comes from

The added member line above carried a cluster id, which raises an obvious question: is that id a property of the snapshot, or of the restore? The capture answered it by restoring the same file twice.

Cluster-wide riskthe same snapshot, two restores, two membership configurations— EXECUTION VALIDATED on etcd 3.7.1. Step 7 of the capture: the command line is the restore form the capture used, run twice with different --initial-cluster values, and the block below is the capture script's comparison of the two cluster ids that resulted; the labels are its own.

$ etcdutl snapshot restore /work/backup.db --data-dir /work/d2 ...

--- cluster id, original vs restored-with-IDENTICAL-initial-cluster ---
original : 1c45a069f3a1d796
restored : 1c45a069f3a1d796
SAME

--- now restore the SAME snapshot with a DIFFERENT --initial-cluster ---
restored with a different initial-cluster : 80d54574493dd420
DIFFERENT from the original

Restoring with an --initial-cluster identical to the original reproduced 1c45a069f3a1d796, the id the cluster had before it died. Restoring the same bytes with a different membership produced 80d54574493dd420. The id is therefore derived from the membership configuration supplied at restore time, not carried inside the file.

Two operational consequences follow, and they point in opposite directions. A matching cluster id after a restore is weaker evidence than it looks: it proves the membership flags matched, and says nothing at all about which snapshot was used or how old it was. Meanwhile a changed cluster id is a real event to plan for, because recovery onto replacement hosts with new peer URLs will produce one, and anything that recorded the old id — monitoring configuration, dashboards, saved queries, operational documentation — now refers to a cluster that no longer exists under that name.

One snapshot, every member, all stopped

Everything above converges on a single restore procedure, and it is not the one people reach for first. Because a restore rewrites the raft history, a restored member and a surviving member hold logs that disagree about what happened. The surviving members cannot bring the restored one up to date, and the restored one cannot be joined to them as if nothing had happened.

So the sequence for a three-member cluster is: stop every member, restore each one from the same snapshot file, then start them together. Not the failed member. Not the two that look healthy. All of them, from one file, verified to be one file:

SNAP=/var/backups/etcd/etcd-20260828T135249Z.db
for member in etcd-1 etcd-2 etcd-3; do
  ssh "$member" systemctl stop etcd
done
for member in etcd-1 etcd-2 etcd-3; do
  ssh "$member" sha256sum "$SNAP"
done

The three digests have to be identical before any restore begins, for the same reason the snapshot has to be one file: a member restored from a different snapshot is a member starting from a different history, and the cluster that forms is then serving a keyspace nobody selected deliberately. Each member is also restored into a new data directory rather than over the old one: the capture’s restore wrote /work/d2, and every path in its log — /work/d2/member/wal, /work/d2/member/snap/db — was created by the restore. Saying that explicitly in the runbook is worth the line, because it means the old directory survives the recovery for anyone who later wants to know what happened to it.

Rehearsing this end to end is the only way to find out what it costs. The capture’s restore completed in about a second on 53 keys; the number that matters for your cluster is the one you measure on your object count, your disk, and your procedure for stopping and starting every member of the control plane at once.

What to take from this

On etcd 3.7.1, etcdctl snapshot status returned the parent command’s usage text — Manages etcd node snapshots — because status and restore are now etcdutl subcommands. etcdctl snapshot save is unchanged.
The snapshot’s entire self-description is HASH 8c66d749, REVISION 54, TOTAL KEYS 53, TOTAL SIZE 45 kB, VERSION 3.7.0, in a file occupying 48K. Record those at backup time; nothing else in the file will tell you later.
The restore returned 53 keys and exit code 0 while a Deployment reverted from replicas=12 to replicas=6 and /registry/configmaps/production/feature-flags came back ABSENT. The RPO arrives as an absence, never as an error.
The value S3cretValue at /registry/secrets/production/db-password was in the snapshot in the clear and read back unchanged, so the file carries the sensitivity of every Secret in the cluster unless encryption at rest is configured separately.
Restoring with an identical --initial-cluster reproduced cluster id 1c45a069f3a1d796; the same file with a different one produced 80d54574493dd420. The id comes from the membership configuration, not from the snapshot.
The restored member reported RAFT INDEX 3 against the original’s 4, because a restore writes a fresh WAL. That is why every member must be stopped and restored from the same snapshot rather than one member repaired in place.

Cross-course references

Kubernetes for Production Sysadmins — Part LXVIII (etcd Backup) and Part LXIX (etcd Restore) cover the cluster-operator view of the same two operations, including the certificate and endpoint flags this lesson elides behind ...; read them for the mechanics of running the commands against a kubeadm control plane, and this lesson for what the resulting file does and does not let you claim.
Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) is the direct answer to the plaintext S3cretValue measured here: if the material a workload needs is issued by a platform outside the cluster rather than stored inside it, an etcd snapshot stops being a file that carries every credential you own.
Observability for Production Sysadmins — Part XX (Alert Quality) applies to the silent revert in this capture, where the restore succeeded and the cluster ran six replicas instead of twelve: no signal exists for that condition unless someone deliberately builds one from an independent record of the desired state.

The subcommand your runbook names may no longer exist

8c66d749, revision 54, 53 keys: everything the file says about itself