Lab 24: Take and verify an etcd snapshot

Objective

By the end of this lab you will have taken an etcd snapshot, proved it holds an object you can name, proved that your verification catches a file somebody damaged, and proved the snapshot is restorable — all without stopping the control plane once. Then you will put the sequence behind a timer and break the timer on purpose, so you have seen what a failed backup job looks like on a host where nobody was watching.

Taking the snapshot is one command and takes about a minute. The other eighty-nine minutes are the part that matters, because the failure this lab exists to prevent is not “we had no backup command”. It is “we had a backup command, it had been failing for five weeks, and nobody looked”.

Architecture

One disposable control-plane node running a single-member etcd. The live cluster is only ever read: the snapshot is a read, the verification runs on files, and the restore lands in a sandbox directory served by a second etcd process on ports nothing else uses. Nothing in this lab moves a static Pod manifest or touches /var/lib/etcd.

flowchart LR
    OP[Operator on cp-1] --> KB[kubectl]
    KB --> API[kube-apiserver static Pod]
    API --> E[etcd static Pod, port 2379]
    E --> DD["/var/lib/etcd on the host"]
    CTL[etcdctl snapshot save] --> E
    CTL --> FILE["/var/backups/etcd/*.db"]
    FILE --> UTL[etcdutl snapshot status]
    FILE --> SB[etcdutl snapshot restore]
    SB --> SBD["/var/tmp/etcd-sandbox"]
    SBD --> SBE["sandbox etcd, port 12379"]
    TIMER[etcd-snapshot.timer] --> CTL

Requirements

• One disposable virtual machine: 2 vCPU, 4 GiB RAM, 40 GiB disk. Nested virtualisation is fine; this lab never nests further.
• A single-control-plane kubeadm cluster at v1.34.x, already built and healthy, with a CNI installed and CoreDNS Ready. Lab 1 builds it.
• At least 5 GiB free on the filesystem holding /var. The lab keeps several snapshots and unpacks one of them into a sandbox data directory under /var/tmp.
• sudo on the node, and a systemd-managed host — Task 7 installs a unit and a timer.
• Outbound network access from the node to fetch the etcd release tarball from GitHub. Nothing else is downloaded; the lab deploys no workload.
• No out-of-band console is required. This lab does not reconfigure networking, the firewall or sshd, and does not stop the control plane at any point.

Scenario

Your team’s cluster has an hourly etcd snapshot. It has run for months. Nobody has restored one.

This week an auditor asked a question that turned out to be hard: “how do you know the backups work?” The honest answer was that a file appears in /var/backups/etcd every hour and the job exits zero. That is a claim about the job, not about the backup. Between that claim and a usable recovery point sit three unproven steps — that the file is intact, that it holds the objects you think it holds, and that etcd will accept it.

Your job in this lab is to close all three, and to leave behind a schedule whose failures are visible from the host.

Tasks

Task 1 — Put the right tools on the node

Three binaries, three different jobs, and mixing them up is the most common way this procedure stalls.

etcdctl is the network client. It speaks gRPC to a running member, and snapshot save is one of its calls — which is why taking a snapshot needs a live, healthy member and a TLS identity it will accept.

etcdutl is the offline utility. It operates on data files and never connects to anything, which is why snapshot status and snapshot restore live there. etcd v3.6.0 removed the etcdctl equivalents, so on a 1.34 cluster there is no alternative.

etcd itself is the server, and you need it in Task 6 to serve the sandbox copy.

All three live inside the container image kubeadm runs rather than on the host, and that image ships etcd and etcdctl but not etcdutl — so there is nothing to crictl exec into for the offline half of the work. Install all three on the host. Read the image tag first and match the tarball to it: a snapshot written by one minor version and inspected by another is a variable you do not want in your first restore.

$ sudo grep -- 'image:' /etc/kubernetes/manifests/etcd.yaml

    image: registry.k8s.io/etcd:3.6.5-0

# Match this to the tag printed above: add the leading v, drop the -0 suffix.
ETCD_VER=v3.6.5

curl -fsSL "https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz" \
  -o /tmp/etcd.tar.gz

sudo tar xzf /tmp/etcd.tar.gz -C /usr/local/bin --strip-components=1 --no-same-owner \
  "etcd-$ETCD_VER-linux-amd64/etcd" \
  "etcd-$ETCD_VER-linux-amd64/etcdctl" \
  "etcd-$ETCD_VER-linux-amd64/etcdutl"

/usr/local/bin/etcd --version
/usr/local/bin/etcdctl version
/usr/local/bin/etcdutl version

Now set the client environment once. Every etcdctl call against the live cluster reuses it, which is why the later commands look short.

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

sudo -E etcdctl endpoint health -w table
sudo -E etcdctl endpoint status -w table
sudo -E etcdctl alarm list

If etcdctl answers unauthenticated, swap ETCDCTL_CERT and ETCDCTL_KEY for the healthcheck-client pair in the same directory; kubeadm issues it from the same CA specifically for client calls, and it is the cleaner choice for a scheduled job in any case.

Record the DB SIZE column from endpoint status and check that alarm list is empty. Both matter before you snapshot: a NOSPACE alarm means the database is already at its quota, and a snapshot taken while an alarm is active captures a cluster you would not want to restore to.

Task 2 — Give yourself something to look for

A snapshot you cannot interrogate is a snapshot you can only trust. Put two named objects in the cluster now so that in Task 6 you can ask a concrete question — “is recovery-marker in this file?” — instead of a vague one.

mkdir -p "$HOME/etcd-backup-lab/evidence"
cd "$HOME/etcd-backup-lab"

# ~/etcd-backup-lab/marker.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: backup-proof
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: recovery-marker
  namespace: backup-proof
data:
  note: If this key is in the snapshot, the snapshot holds cluster state.

kubectl apply -f marker.yaml

# The nonce makes this snapshot distinguishable from every other one.
NONCE=$(date -u +%Y%m%dT%H%M%SZ)
kubectl -n backup-proof patch configmap recovery-marker \
  --type merge -p "{\"data\":{\"nonce\":\"$NONCE\"}}"
echo "$NONCE" | tee evidence/nonce.txt

kubectl -n backup-proof get configmap recovery-marker -o yaml

Now record what the cluster looks like at the etcd layer. These are the numbers you will compare the snapshot against.

cd "$HOME/etcd-backup-lab"

sudo -E etcdctl get /registry/ --prefix --keys-only | head -5
sudo -E etcdctl get /registry/ --prefix --keys-only | grep -c '^/registry/' | tee evidence/live-registry-keys.txt
sudo -E etcdctl get /registry/namespaces/ --prefix --keys-only | grep -c '^/registry/' | tee evidence/live-namespace-keys.txt
sudo -E etcdctl get /registry/configmaps/backup-proof/ --prefix --keys-only

/registry/apiregistration.k8s.io/apiservices/v1.

/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io

/registry/configmaps/backup-proof/recovery-marker

Two things to notice in that first head -5. Every Kubernetes object lives under /registry/, so that prefix is the whole cluster. And the keys come back separated by blank lines, because --keys-only leaves the value line empty — which is why the counts above use grep -c on the key pattern rather than wc -l, and why a count you took with wc -l on another day is not comparable to these.

The marker’s etcd key is /registry/configmaps/backup-proof/recovery-marker. Write it down. That exact string is the question you will put to the sandbox in Task 6.

Task 3 — Take the snapshot, and measure what it cost

snapshot save reads the member’s database from local disk and writes it out as a self-contained file. On a multi-member cluster you aim this at a follower, because on the leader the read competes with the WAL fsync path that every commit waits on. This lab cluster has one member, so there is no follower and no choice — note that the reason the cost is invisible here is the lab, not the command.

sudo install -d -m 700 /var/backups/etcd

SNAP=/var/backups/etcd/etcd-$(date -u +%Y%m%dT%H%M%SZ).db
echo "$SNAP" | tee "$HOME/etcd-backup-lab/evidence/snapshot-path.txt"

time sudo -E etcdctl snapshot save "$SNAP"

sudo ls -l "$SNAP"
sudo -E etcdctl endpoint status -w table

Compare three numbers before moving on: the wall-clock time from time, the file size from ls -l, and the DB SIZE from endpoint status. On an idle lab cluster the snapshot is tens of megabytes and the save takes a second or two. On a production cluster with a two-gigabyte database and a cold page cache it is a minute of sustained disk read, and that minute is the operational cost you are scheduling.

Note also that snapshot save takes exactly one endpoint. It is a request to a specific member for that member’s view of the database, not a cluster-wide operation you can point at a list and let it choose.

Task 4 — Verify at write time

Verification is not a separate chore that happens later. A snapshot that has not been verified is not a recovery point, and the window in which verification is cheap is the thirty seconds after the save, while you are still standing there.

cd "$HOME/etcd-backup-lab"
SNAP=$(cat evidence/snapshot-path.txt)

sudo etcdutl snapshot status "$SNAP" -w table | tee evidence/snapshot-status.txt
sudo etcdutl snapshot status "$SNAP" -w json | tee evidence/snapshot-status.json

sudo sha256sum "$SNAP" | sudo tee "$SNAP.sha256"
sudo sha256sum -c "$SNAP.sha256"

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| a1b2c3d4 |    41289 |       1204 |      6.6 MB |
+----------+----------+------------+------------+
/var/backups/etcd/etcd-20260819T101500Z.db: OK

Read the table properly. Each of its four columns answers a different question, and three of them are routinely over-read.

REVISION is the cluster’s revision at the moment of the save. It is the single number that says how far back a restore from this file would wind the cluster, and it is what makes a data-loss window concrete in Lab 25. Record it next to the timestamp.

TOTAL KEYS is a count taken from the snapshot’s underlying database. Compare it now against evidence/live-registry-keys.txt from Task 2 and record both. Do not expect them to agree: the database keeps bookkeeping of its own and retains superseded versions of keys until compaction reclaims them, so this is a storage-layer number rather than an inventory of your objects. That does not make it useless. Recorded every run, it is an excellent tripwire — a snapshot whose key count has collapsed towards zero is a broken snapshot, whatever its exit status said — but a backup policy that treats it as an object count will one day be surprised by a compaction.

TOTAL SIZE should track the DB SIZE you read in Task 3. A large divergence is worth understanding before you rely on the file.

HASH is computed from the file in front of it, right now. This is the part that is most often misunderstood, so state it plainly: snapshot status does not compare the file to anything. It reports what the file currently contains. The sidecar you just wrote with sha256sum is the only artefact that lets you ask the different and more important question — is this file still what it was when it was written?

Task 5 — Prove your verification catches something

Every backup document says “verify the snapshot”. Almost none of them check that the verification has any teeth. You are going to damage two copies, run both checks against each, and write down what happened — and the deliverable from this task is the table you fill in, not a result this page hands you.

cd "$HOME/etcd-backup-lab"
SNAP=$(cat evidence/snapshot-path.txt)

sudo cp "$SNAP" /var/tmp/snap-truncated.db
sudo cp "$SNAP" /var/tmp/snap-flipped.db

# A partial write: the last 64 KiB never landed.
sudo truncate -s -64K /var/tmp/snap-truncated.db

# Silent bit rot: one byte changed, file length identical.
printf 'X' | sudo dd of=/var/tmp/snap-flipped.db bs=1 seek=100000 conv=notrunc status=none

sudo ls -l "$SNAP" /var/tmp/snap-truncated.db /var/tmp/snap-flipped.db

Now run both checks against both files, and record the exit status as well as the message. The exit status is what a scheduled job sees.

cd "$HOME/etcd-backup-lab"
SNAP=$(cat evidence/snapshot-path.txt)
GOOD=$(sudo awk '{print $1}' "$SNAP.sha256")

for f in /var/tmp/snap-truncated.db /var/tmp/snap-flipped.db; do
echo "=== $f"
sudo etcdutl snapshot status "$f" -w table || echo "status exit=$?"
echo "recorded sha256: $GOOD"
sudo sha256sum "$f"
done 2>&1 | sudo tee evidence/damage-log.txt

Fill this in from your own output. The two right-hand columns are the answer you came for.

The conclusion is the same whichever way your results fall, and it is worth writing out in your own words before you read on: the sha256 taken at write time is the only check that compares the file against what it was. Every other check reads the file that is in front of it and tells you about that. A verification pipeline that computes the hash but never stores it has an audit trail, not an integrity check.

While you are here, find out what your own binary will do at restore time. This is a version probe, not decoration.

sudo etcdutl snapshot restore --help | grep -i -- 'hash\|revision\|compact'

Delete the damaged copies now, before they can be mistaken for snapshots. A damaged file in a backup directory is worse than no file.

sudo rm -f /var/tmp/snap-truncated.db /var/tmp/snap-flipped.db
sudo ls -l /var/backups/etcd/

Task 6 — Prove the snapshot is restorable, without an outage

This is the check that converts “we have backups” into “we can recover”, and it is the one teams skip because they assume it needs a maintenance window. It does not. etcdutl snapshot restore writes a new data directory and never contacts a running member; a second etcd process can serve that directory on ports nothing else is using, on this same host, while the cluster carries on.

cd "$HOME/etcd-backup-lab"
SNAP=$(cat evidence/snapshot-path.txt)

sudo etcdutl snapshot restore "$SNAP" --name sandbox --initial-cluster sandbox=http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-sandbox-verify --data-dir /var/tmp/etcd-sandbox

sudo ls -l /var/tmp/etcd-sandbox/member/

Serve it. systemd-run gives the sandbox a transient unit, which means its logs land in the journal and stopping it is one command instead of a PID hunt.

sudo systemd-run --unit=etcd-sandbox --collect /usr/local/bin/etcd   --name sandbox   --data-dir /var/tmp/etcd-sandbox   --listen-client-urls http://127.0.0.1:12379   --advertise-client-urls http://127.0.0.1:12379   --listen-peer-urls http://127.0.0.1:12380   --initial-advertise-peer-urls http://127.0.0.1:12380   --initial-cluster sandbox=http://127.0.0.1:12380

sleep 10
sudo systemctl is-active etcd-sandbox
sudo journalctl -u etcd-sandbox -n 20 --no-pager

The sandbox listens on loopback only and carries no TLS material, which is correct for a throwaway that lives for four minutes on a host you already have root on. Do not copy that choice to anything that outlives the check.

Now interrogate it — and clear the live cluster’s client environment first.

cd "$HOME/etcd-backup-lab"

( unset ETCDCTL_ENDPOINTS ETCDCTL_CACERT ETCDCTL_CERT ETCDCTL_KEY
SB=http://127.0.0.1:12379

etcdctl --endpoints="$SB" endpoint status -w table
etcdctl --endpoints="$SB" endpoint status -w json | tee evidence/sandbox-status.json
etcdctl --endpoints="$SB" get /registry/configmaps/backup-proof/ --prefix --keys-only   | tee evidence/sandbox-marker.txt
etcdctl --endpoints="$SB" get /registry/ --prefix --keys-only   | grep -c '^/registry/' | tee evidence/sandbox-registry-keys.txt
)

The subshell and the unset are not tidiness. Task 1 exported an endpoint and a certificate pair that point at production, and a sandbox command that silently inherits them is a command that answers a question about the wrong cluster — convincingly, and with no sign that anything went wrong. Clearing them means a mistake produces a connection error instead of a false pass, and the parentheses keep the clearing local so the rest of your session still talks to the live cluster.

Three results to compare, and only one of them is a confirmation:

• The JSON form of endpoint status carries the response header, and its revision field is what you compare against Task 4’s REVISION. Expect it to match or to sit a little above — what must not happen is a revision far below the snapshot’s, which would mean you are looking at a different file than you think.
• sandbox-marker.txt must contain /registry/configmaps/backup-proof/recovery-marker. This is the object you planted in Task 2, recovered from a database rebuilt entirely from the file. That is the whole point of the exercise.
• sandbox-registry-keys.txt against live-registry-keys.txt from Task 2: close, but not equal, because the cluster kept writing while you worked. Record the difference. On a real cluster that gap is the shape of your data-loss window.

Optionally confirm the nonce survived the round trip. strings comes from binutils; skip this if the node does not have it.

# Substitute the marker key you recorded in Task 2.
MARKER=/registry/configmaps/backup-proof/recovery-marker
NONCE=$(cat "$HOME/etcd-backup-lab/evidence/nonce.txt")

( unset ETCDCTL_ENDPOINTS ETCDCTL_CACERT ETCDCTL_CERT ETCDCTL_KEY
  etcdctl --endpoints=http://127.0.0.1:12379 \
    get "$MARKER" --print-value-only | strings | grep -F "$NONCE"
)

Then stop the sandbox. Leave the data directory for now — the Validation section reads it once more.

sudo systemctl stop etcd-sandbox
sudo systemctl is-active etcd-sandbox || echo 'sandbox stopped'
sudo -E etcdctl endpoint health -w table

That last line is deliberate. Confirm the live cluster is exactly as healthy as it was before you started, because the claim this task makes is not only “the snapshot restores” but “proving it cost the cluster nothing”.

Task 7 — Make it repeatable, and make it verify itself

A snapshot you take by hand is a snapshot that stops when you go on leave. Everything you have done so far goes into a script that saves, verifies, stamps its success and prunes — in that order, with the script aborting if any step fails.

#!/usr/bin/env bash
# Save as ~/etcd-backup-lab/etcd-snapshot.sh; installed below as
# /usr/local/sbin/etcd-snapshot.sh
set -euo pipefail

BACKUP_DIR=/var/backups/etcd
KEEP=3
ENDPOINT=https://127.0.0.1:2379
CACERT=/etc/kubernetes/pki/etcd/ca.crt
CERT=/etc/kubernetes/pki/etcd/server.crt
KEY=/etc/kubernetes/pki/etcd/server.key

install -d -m 700 "$BACKUP_DIR"
SNAP="$BACKUP_DIR/etcd-$(date -u +%Y%m%dT%H%M%SZ).db"

ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints="$ENDPOINT" --cacert="$CACERT" --cert="$CERT" --key="$KEY" \
  snapshot save "$SNAP"

/usr/local/bin/etcdutl snapshot status "$SNAP" -w json
sha256sum "$SNAP" > "$SNAP.sha256"
sha256sum -c "$SNAP.sha256"

date -u +%Y-%m-%dT%H:%M:%SZ > "$BACKUP_DIR/last-success"

ls -1t "$BACKUP_DIR"/*.db | tail -n +$((KEEP + 1)) | while read -r old; do
  rm -f -- "$old" "$old.sha256"
done

Four decisions in that script are worth naming.

set -euo pipefail means a failed verification fails the job. Without it, snapshot status can report a corrupt file and the script carries on to write a success stamp, which is precisely the silent-failure pattern the schedule exists to avoid.

The binaries are absolute paths. A systemd unit does not inherit your login shell’s environment, and a command not found at 03:00 is a much duller failure to diagnose than it deserves to be.

The success stamp is written last, only after the verification passed. It is the local equivalent of the snapshot-age metric the lessons alert on, and it is what Task 8 interrogates.

KEEP=3 is a lab value that makes retention observable inside ninety minutes. Hourly snapshots with twenty-four kept is the production starting point, and the number is set by your RPO, not by disk pricing. One consequence to expect: after three scheduled runs the prune will delete the snapshot you took by hand in Task 3. Copy it somewhere outside /var/backups/etcd first if you want to keep it for reference.

Save that as ~/etcd-backup-lab/etcd-snapshot.sh, then install it and the two units.

sudo install -m 750 -o root -g root "$HOME/etcd-backup-lab/etcd-snapshot.sh" /usr/local/sbin/etcd-snapshot.sh

sudo bash -n /usr/local/sbin/etcd-snapshot.sh && echo 'script parses'

bash -n parses the script without running it. A syntax error caught here costs you five seconds; caught by the timer at 03:00 it costs you a snapshot and gives you a unit failure with no obvious cause.

# /etc/systemd/system/etcd-snapshot.service
[Unit]
Description=Take and verify an etcd snapshot
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/etcd-snapshot.sh

# /etc/systemd/system/etcd-snapshot.timer
[Unit]
Description=Snapshot etcd every ten minutes (lab cadence; hourly in production)

[Timer]
OnCalendar=*:0/10
Persistent=true

[Install]
WantedBy=timers.target

Create both unit files at the paths named in their first lines, owned by root. Persistent=true makes systemd run a trigger it missed while the host was down, at the next boot. On a control plane that reboots for a kernel upgrade, that is the difference between a gap in the chain and a late snapshot.

systemd-analyze calendar '*:0/10'

sudo systemctl daemon-reload
sudo systemctl enable --now etcd-snapshot.timer
sudo systemctl list-timers etcd-snapshot.timer --all

sudo systemctl start etcd-snapshot.service
sudo systemctl status etcd-snapshot.service --no-pager
sudo cat /var/backups/etcd/last-success
sudo ls -l /var/backups/etcd/

systemd-analyze calendar parses the expression and prints the next time it fires. It is the cheapest possible check that you have not written a timer that never triggers, and it costs nothing to run before you enable anything.

Task 8 — Break it, and find out whether you would have noticed

Recall the failure the scenario opened with. A team took hourly snapshots for six months, a control-plane host was rebuilt for a kernel upgrade, the job was not migrated, and the gap ran for five weeks before an audit found it. Nothing about that story requires anyone to be careless. It requires only that a failing job be invisible.

Break yours in the same shape a certificate rotation would.

sudo sed -i 's#^CERT=.*#CERT=/etc/kubernetes/pki/etcd/rotated-away.crt#' /usr/local/sbin/etcd-snapshot.sh
sudo grep '^CERT=' /usr/local/sbin/etcd-snapshot.sh

sudo systemctl start etcd-snapshot.service || echo "unit exited non-zero"
sudo systemctl is-failed etcd-snapshot.service
sudo journalctl -u etcd-snapshot.service -n 30 --no-pager
sudo cat /var/backups/etcd/last-success

Three observations, and the third is the one to take away.

The unit is in failed state and systemctl is-failed says so. The journal holds the reason.

The success stamp still shows the time of the last successful run, not this one — because the script aborted before reaching it. That is set -euo pipefail earning its place, and it is why the stamp can be trusted as an age signal.

And nothing told you. No alert fired, no ticket opened, no dashboard changed colour. The failure is completely visible to anyone who logs into this host and completely invisible to everyone who does not. That gap is the whole of the five-week story, and closing it is a monitoring job, not a backup job.

Here is the check a monitoring agent would run, and you can run it now:

if [ -n "$(sudo find /var/backups/etcd/last-success -mmin +20)" ]; then
echo 'STALE: no verified snapshot in the last 20 minutes'
else
echo 'FRESH: last verified snapshot is inside the window'
fi

sudo systemctl is-failed etcd-snapshot.service && echo 'FAILED: the snapshot unit is in a failed state'

The find is wrapped in a test rather than chained with && for a reason worth stealing: find exits zero when it matches nothing, so a naive find ... && echo STALE reports every snapshot as stale. An alert that fires when everything is fine gets muted, and a muted alert is the same as no alert.

Twenty minutes is two lab cadences. In production it is two hours against an hourly schedule, which is the threshold the alerting lesson uses: “snapshot age exceeds two intervals” catches a job that has stopped without firing on a single late run.

Now repair it and clear the failure.

sudo sed -i 's#^CERT=.*#CERT=/etc/kubernetes/pki/etcd/server.crt#' /usr/local/sbin/etcd-snapshot.sh
sudo systemctl reset-failed etcd-snapshot.service
sudo systemctl start etcd-snapshot.service
sudo systemctl is-active etcd-snapshot.service || sudo systemctl status etcd-snapshot.service --no-pager
sudo cat /var/backups/etcd/last-success

The stamp should now carry the current time. Type=oneshot units report inactive once they finish successfully, so read the stamp and the exit status rather than expecting active.

Validation

Run these in order. Each one proves something the lab claimed rather than restating it.

The live cluster was never disturbed:

kubectl get --raw '/readyz?verbose'
kubectl get nodes
sudo -E etcdctl endpoint health -w table
sudo -E etcdctl alarm list

Every check ok, the node Ready, the member healthy, no alarms. If any of that is untrue, it is not the snapshot that broke it — but find out before you record a result.

The snapshot chain is intact and each file is verifiable:

cd "$HOME/etcd-backup-lab"

sudo ls -l /var/backups/etcd/
for s in /var/backups/etcd/*.db; do
  sudo sha256sum -c "$s.sha256"
  sudo etcdutl snapshot status "$s" -w table
done

Every sidecar reports OK and every status prints a non-zero revision.

Retention did what the script said it would:

$ sudo ls -1 /var/backups/etcd/*.db | wc -l

3

Three .db files, no more, because KEEP=3. If you see more, the prune loop did not run — check the tail of the journal for the last unit run.

The snapshot demonstrably contains cluster state:

grep -c 'recovery-marker' "$HOME/etcd-backup-lab/evidence/sandbox-marker.txt"

One or more. This is the file you captured in Task 6 from a database that was reconstructed entirely from the snapshot, so it is evidence of recoverability rather than of the cluster still working.

The schedule is armed and its next run is in the future:

sudo systemctl is-enabled etcd-snapshot.timer
sudo systemctl list-timers etcd-snapshot.timer --all
sudo systemctl is-failed etcd-snapshot.service || echo 'unit is not in a failed state'

And finally, write the claim down. This is the deliverable the auditor in the scenario was asking for, and it is four lines:

Cadence:   every 10 minutes (lab); hourly in production
Retention: 3 local snapshots (lab); 24 hourly + 30 daily off-host in production
Verified:  etcdutl snapshot status + recorded sha256 on every run, enforced by set -e
Proven:    snapshot <name> restored into a sandbox on <date>, marker key present

Anything you cannot fill in from your own evidence directory is a claim you have not earned yet.

Expected Outcome

• /var/backups/etcd holds exactly three .db files, each with a .sha256 sidecar that passes sha256sum -c.
• /var/backups/etcd/last-success carries a timestamp from the last successful run and not from the failed one.
• etcd-snapshot.timer is enabled with a next elapse in the future, and etcd-snapshot.service is not in a failed state.
• Your evidence directory shows the marker key recovered from a sandbox restore, plus the damage log from Task 5 with both checks recorded.
• The live cluster is healthy, alarm-free, and was never stopped.
• You can say, in one sentence and without hedging, what your verification catches and what it does not.

Troubleshooting

Rollback

The only change here that outlives the session is the scheduled job. Backing it out is one command, and it is worth rehearsing: a snapshot timer misfiring on a busy production member is a real reason to disable one in a hurry.

sudo systemctl disable --now etcd-snapshot.timer
sudo systemctl reset-failed etcd-snapshot.service || true
sudo systemctl list-timers --all | grep etcd-snapshot || echo 'timer no longer scheduled'
sudo ls -l /var/backups/etcd/

The snapshots stay where they are. Disabling a backup schedule and deleting the backups are separate decisions, and conflating them under time pressure is how a rollback turns into an incident.

Cleanup

Run this once you have finished comparing. It removes everything the lab created and leaves the cluster as it was.

sudo systemctl disable --now etcd-snapshot.timer
sudo systemctl stop etcd-sandbox || true
sudo rm -f /etc/systemd/system/etcd-snapshot.service /etc/systemd/system/etcd-snapshot.timer
sudo rm -f /usr/local/sbin/etcd-snapshot.sh
sudo systemctl reset-failed etcd-snapshot.service || true
sudo systemctl daemon-reload

sudo rm -rf /var/tmp/etcd-sandbox /var/backups/etcd
sudo rm -f /tmp/etcd.tar.gz

kubectl delete namespace backup-proof --ignore-not-found
rm -rf "$HOME/etcd-backup-lab"

kubectl get ns
sudo -E etcdctl endpoint health -w table
sudo ls -ld /var/lib/etcd

Leave etcd, etcdctl and etcdutl in /usr/local/bin. All three are worth having on a control-plane node, and the version checks in Task 1 are how you match them to the cluster next time.

Production notes

Aim the snapshot at a follower. On a three-member cluster the save competes with the leader’s WAL fsync path, and the leader is the member whose latency everything else waits on. Point the job at a follower, or rotate the target between members and let the leader be skipped.

One snapshot, not three. An etcd snapshot is cluster-wide, not per-member. Running the job on all three control-plane hosts triples the disk read for no additional coverage. Run it on one, and make the schedule’s failover explicit rather than accidental.

Tier 1 is not a backup. Everything this lab produced lives on the same host as the member it came from, so it dies with that host. The production shape is the same script with an upload step: local fast volume first because the save must finish quickly, then off-host within minutes, then object storage with versioning and a lifecycle policy. The hash you recorded is what makes the upload verifiable at the far end.

The snapshot is a copy of the Secret store. Whatever the API server wrote is what the file contains, so a cluster without encryption at rest produces snapshots holding plaintext Secrets and ServiceAccount tokens. Encrypt before the file leaves the host, keep the key somewhere the cluster’s own failure cannot take with it, and remember that a snapshot restored without its key is a cluster whose Secrets will not decrypt.

Age is the alert, not exit status. A job that fails loudly is the easy case. The dangerous case is a job that stopped being scheduled at all, which produces no failures because it produces nothing. Alert on the age of the last verified snapshot, and let a failed run be the secondary signal.

Rehearse the restore on a cadence. The sandbox restore in Task 6 costs four minutes and no outage, which means there is no defensible reason to run it less often than weekly. The full-cluster restore — the one with an outage and a data-loss window — is Lab 25, and it is the drill that turns this file into a recovery capability.

What you learned

• A snapshot is a file until somebody proves otherwise. The proof is three separate claims — intact, correct contents, accepted by etcd — and each needs its own check.
• snapshot status reports; it does not compare. It reads the file in front of it. Only the sha256 recorded at write time can tell you the file is still what it was.
• TOTAL KEYS is a storage-layer number. Do not read it as an object count. Recorded every run it is still a good tripwire, which is a different and more modest job.
• Restorability can be proved without an outage. A sandbox restore on alternate ports answers the only question that matters, on a live host, in minutes.
• The environment is part of the blast radius. Exported ETCDCTL_* variables will happily make a sandbox command answer for production; unsetting them inside a subshell turns that silent wrong answer into a loud connection error.
• The failure mode of a backup schedule is silence. You broke the job and watched nothing happen — which is exactly what five weeks without a usable snapshot looks like from the outside.