medium riskcluster affecting~30 min
Runbook: Back Up etcd
1 · Prerequisites
Confirm every item is in place before any state change.
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the etcdctl binary is available on a control-plane node:
etcdctl version - · Confirm the etcd TLS files exist:
ls -l /etc/kubernetes/pki/etcd/{ca.crt,server.crt,server.key} - · Confirm the etcd cluster is healthy:
etcdctl ... endpoint health --cluster -w table - · Confirm the off-host backup destination is reachable and authenticated:
restic -r sftp:backup@backup.internal:/srv/etcd snapshots - · Confirm a clean local directory exists for the snapshot:
mkdir -p /var/backups/etcd && chmod 700 /var/backups/etcd
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Set the etcd environment variables for the session:
export ETCDCTL_API=3and prepare the flags - 2Take the snapshot:
etcdctl --endpoints=https://127.0.0.1:2379 --cacert=<ca> --cert=<cert> --key=<key> snapshot save /var/backups/etcd/etcd-<timestamp>.db - 3Verify the snapshot integrity:
etcdutl snapshot status /var/backups/etcd/etcd-<timestamp>.db -w table - 4Compute the SHA256 of the snapshot for off-host verification:
sha256sum /var/backups/etcd/etcd-<timestamp>.db > /var/backups/etcd/etcd-<timestamp>.db.sha256 - 5Encrypt the snapshot before transmission (e.g. with age, GPG, or restic):
age -r age1backup... < /var/backups/etcd/etcd-<timestamp>.db > /var/backups/etcd/etcd-<timestamp>.db.age - 6Push to off-host:
restic -r sftp:backup@backup.internal:/srv/etcd backup /var/backups/etcd/etcd-<timestamp>.db.age - 7Verify the off-host copy matches:
restic -r sftp:backup@backup.internal:/srv/etcd snapshots - 8Validate the snapshot periodically by restoring it in a sandbox (see
kubernetes-rb-restore-etcd) - 9Apply retention: prune old snapshots according to the cadence policy (e.g. 24 hourly, 7 daily, 4 weekly)
- 10Capture the run in the change ticket: snapshot path, hash, retention policy applied
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓
etcdutl snapshot status <snap> -w tablereports the snapshot hash, revision, and total key count - ✓
sha256summatches between local and off-host copies - ✓The snapshot is restorable: in a sandbox,
etcdutl snapshot restoreproduces a valid DB and the test API server starts - ✓
restic snapshotslists the snapshot with the expected timestamp and size - ✓No active alarms on etcd after the snapshot:
etcdctl ... alarm list - ✓No errors in etcd logs at the time of snapshot
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶A snapshot cannot be "rolled back" - it is a point-in-time read of the cluster. The next snapshot will supersede it.
- ↶If the off-host push fails, the local copy is still valid; do not delete it until the off-host copy is verified
- ↶If the encryption key is lost, the snapshot is unrecoverable; this is by design but must be guarded by a tested key-rotation procedure
- ↶If the snapshot is invalid, take another immediately and investigate the etcd cluster health; an invalid snapshot indicates an etcd issue
- ↶Capture the snapshot, its hash, and the etcd logs at the time of snapshot to the change ticket
6 · Escalation
When the runbook isn't enough, contact:
- · Snapshot fails to save: etcd is unhealthy; see
kubernetes-rb-investigate-etcd-healthfirst - · Snapshot integrity check fails: re-take immediately; investigate etcd disk and WAL
- · Off-host push repeatedly fails: the backup destination is unreachable; escalate to backup ownership
- · Encryption key rotation required but no rotation procedure documented: pause the rotation and escalate to security
- · Snapshot cannot be restored in the sandbox: investigate before declaring the backup pipeline healthy; an unrestorable snapshot is no backup
A snapshot is the only recoverable representation of the cluster
state. It is taken at a point in time; restoring it returns the
cluster to that point. The runbook treats the snapshot as a
deliverable to the restore path, not a file in /var/backups.
1. Prepare the environment
Read-only / SafePrepare the environment
export ETCDCTL_API=3
# Verify etcd is healthy before snapshotting
etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health -w table
# Verify the off-host destination
restic -r sftp:backup@backup.internal:/srv/etcd snapshots 2>&1 | head -10
# Create the local backup directory
sudo mkdir -p /var/backups/etcd
sudo chmod 700 /var/backups/etcd
A snapshot of an unhealthy etcd is an unhealthy snapshot. Health first, snapshot second.
2. Take the snapshot
Read-only / SafeTake the snapshot
sudo ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save "$SNAP"
sudo etcdutl snapshot status "$SNAP" -w table
# Columns: hash, revision, totalKey, totalSize
A snapshot is taken via the etcd streaming snapshot protocol; it does not block the etcd cluster.
3. Verify the snapshot locally
Read-only / SafeVerify the snapshot locally
$SNAP" | sudo tee "$SNAP.sha256"
# Confirm the snapshot file is readable
sudo file "$SNAP"
# Inspect the snapshot metadata
sudo etcdutl snapshot status "$SNAP" -w json | jq
The hash is the off-host verification fingerprint. It must match the remote copy byte-for-byte.
4. Encrypt the snapshot
Read-only / SafeEncrypt the snapshot
AGE_KEY=$(cat /etc/backup/age-key.pub)
sudo age -r "$AGE_KEY" < "$SNAP" > "$SNAP.age"
sudo rm -f "$SNAP"
# Or GPG
sudo gpg --batch --yes --recipient backup@backup.internal \
--output "$SNAP.gpg" --encrypt "$SNAP"
Encryption is non-optional. The snapshot contains every Secret in the cluster; transmitting it unencrypted is a data exposure.
5. Push off-host
Read-only / SafePush off-host
$SNAP.age" --tag etcd,$(date -u +%Y%m%d)"
sudo rm -f "$SNAP.age"
# Verify the snapshot is present
sudo restic -r sftp:backup@backup.internal:/srv/etcd snapshots | head
6. Validate the snapshot in a sandbox
A snapshot is only as good as its restore. Validate it periodically
in a sandbox by following the kubernetes-rb-restore-etcd runbook
against a throwaway cluster.
Read-only / SafeValidate the snapshot in a sandbox
sudo restic -r sftp:backup@backup.internal:/srv/etcd restore latest --target /tmp/etcd-test
AGE_KEY=$(cat /etc/backup/age-key.pub)
sudo age -d -i /etc/backup/age-key < /tmp/etcd-test/.../snap.age > /tmp/snap.db
# Restore in a sandbox directory
sudo etcdutl snapshot restore /tmp/snap.db \
--data-dir=/tmp/etcd-sandbox \
--name=sandbox \
--initial-cluster=sandbox=http://localhost:2380 \
--initial-advertise-peer-urls=http://localhost:2380
# Inspect the sandbox DB
sudo ETCDCTL_API=3 etcdctl --data-dir=/tmp/etcd-sandbox member list
7. Apply retention
Read-only / SafeApply retention
sudo restic -r sftp:backup@backup.internal:/srv/etcd forget \
--keep-hourly 24 --keep-daily 7 --keep-weekly 4 --prune
8. Capture the run
Read-only / SafeCapture the run
.[-1].id")
echo "etcd snapshot id: $SNAP_ID"
echo "sha256: $(sudo cat /var/backups/etcd/etcd-<timestamp>.db.sha256)"
Common pitfalls
| Symptom | Cause | Action |
|---|---|---|
| Snapshot fails to save | etcd unhealthy or disk full | See kubernetes-rb-investigate-etcd-health |
| Snapshot status reports an error | File corruption during save | Re-take immediately |
| Encryption key lost | Backup pipeline failure | Test key restoration in the next drill |
| Off-host push fails repeatedly | Backup destination down | Escalate to backup ownership |
| Snapshot cannot be restored in sandbox | Snapshot is corrupt or incompatible with the etcd version | Investigate; do not rely on this snapshot for restore |
A backup that is not validated by restore is not a backup; it is a file. The runbook treats validation as part of the backup.