Backup before change — the snapshot as a change gate

Before any control-plane change — upgrade, flag update, cert renewal, member add/remove — an etcd snapshot is the production gate. The snapshot is what makes the change reversible. This lesson walks the discipline of taking, verifying, and storing the snapshot before the change.

The change gate

flowchart LR
    A[Plan change] --> B[Take snapshot]
    B -->|verify| C{Valid?}
    C -->|no| B
    C -->|yes| D[Off-cluster upload]
    D --> E[Execute change]
    E -->|succeed| F[Validate]
    E -->|fail| G[Restore from snapshot]
    F --> H[Document]
    G --> H

The gate is: snapshot is taken, verified, uploaded off-cluster. Only after the gate completes does the change proceed.

The change types

The line: changes that affect cluster state require a snapshot. Changes that affect only workload state do not.

Taking the snapshot

The snapshot command (covered in detail in Part LXVIII):

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-pre-$(date +%Y%m%d-%H%M).db

A pre-upgrade snapshot. The timestamp is in the filename so multiple snapshots are distinguishable.

Verifying the snapshot

The snapshot must be valid before the change proceeds:

etcdutl snapshot status \
  /backup/etcd-snapshot-pre-20260816-1200.db \
  --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

The output must show:

• A non-zero HASH.
• A non-zero REVISION (at or near the cluster’s current state).
• A reasonable TOTAL KEY count.
• A TOTAL SIZE consistent with the bbolt DB size.

# Verify the hash matches the file
sha256sum /backup/etcd-snapshot-pre-20260816-1200.db

A non-matching hash or zero-value fields indicate a corrupt snapshot; do not proceed.

$ etcdutl snapshot status /backup/etcd-snapshot-pre-20260816-1200.db --write-out=json | jq '.hash, .revision, .totalKey, .totalSize'

2735966328
41289312
4123
82419000

Off-cluster upload

The local snapshot is a copy, not a backup. The off-cluster copy:

aws s3 cp /backup/etcd-snapshot-pre-20260816-1200.db \
  s3://prod-etcd-backups/pre-upgrade/etcd-snapshot-pre-20260816-1200.db \
  --sse aws:kms

The upload verifies the snapshot is restorable from a different host (in case the cluster’s local hosts are lost).

The change proceeds

With the snapshot taken, verified, and uploaded, the change can begin:

# Example: kubeadm upgrade
sudo kubeadm upgrade apply v1.34.1

The change typically proceeds without the snapshot being needed. The snapshot is the rollback path; it is invoked only on failure.

The change fails — restoring

A change that goes wrong uses the snapshot:

# Verify the snapshot is intact (paranoid check)
etcdutl snapshot status /backup/etcd-snapshot-pre-20260816-1200.db

# Restore (Part LXIX)
sudo kubeadm init phase etcd local --config=...
# Or, with full cluster restore:
# 1. Stop the API server
# 2. Stop etcd
# 3. Restore snapshot to each etcd host
# 4. Start etcd
# 5. Verify etcd cluster health
# 6. Start the API server

The restore is the recovery path. The cluster reverts to the snapshot’s state.

The “small change” trap

A team thinks: “We’re just changing a single line in the manifest; no snapshot needed.” The change:

1. Causes the API server to crash on startup.
2. The cluster’s control plane is offline.
3. The fix is to revert; but the original manifest is needed.

A 30-second snapshot prevents the 30-minute incident.

The “fresh cluster” trap

A fresh cluster (just kubeadm init-ed) does not need a snapshot before the first change? It does. The fresh cluster has live state; an upgrade that goes wrong loses the fresh state.

A snapshot is the production gate, regardless of cluster age.

The snapshot policy

A documented policy:

CONTROL PLANE CHANGE POLICY
=========================

Before any of the following changes, take an etcd snapshot:
  - Kubernetes upgrade
  - kubeadm-managed component upgrade
  - Static pod manifest edit
  - cert-manager / kubelet upgrade
  - CRD install or change
  - RBAC restructure
  - Member add / remove

Procedure:
  1. Run etcdctl snapshot save /backup/etcd-snapshot-pre-<timestamp>.db
  2. Verify: etcdutl snapshot status
  3. Upload to off-cluster storage
  4. Document the snapshot's location in the change ticket
  5. Execute the change
  6. On failure: restore from the snapshot (Part LXIX)

The change ticket

Each change has a ticket (or change record). The ticket captures:

Change: Upgrade cp-2 from kubeadm 1.34.0 to 1.34.1
Timestamp: 2026-08-16 12:00 UTC
Operator: <name>
Snapshot: /backup/etcd-snapshot-pre-20260816-1200.db
  md5sum: abc123
  revision: 41289312
  keys: 4123
  size: 82419000
  S3 URI: s3://prod-etcd-backups/pre-upgrade/etcd-snapshot-pre-20260816-1200.db
  SHA-256: def456
Approved: <change-advisor>
Result: success / partial / failed

The snapshot’s hashes / revision are the post-change verification anchor.

The “before-snapshot” and “after-snapshot”

A disciplined change has TWO snapshots:

1. Before-snapshot. Captured just before the change. This is the rollback path.
2. After-snapshot. Captured just after the change is validated. This is the “current known good” reference.

The operator compares the after-snapshot to the before-snapshot and the latest scheduled snapshot.

The cluster snapshot

# After successful change
ETCDCTL_API=3 etcdctl snapshot save \
  /backup/etcd-snapshot-post-20260816-1230.db

The post-change snapshot is captured within minutes of the change completion.

The production discipline

• Snapshot before change. Always.
• Verify the snapshot. Status + hash.
• Upload off-cluster. The local snapshot is one host loss away from being lost.
• Document the snapshot in the change ticket. Hash, revision, location.
• Take an after-snapshot. Capture the new known-good.
• Rehearse the restore. The snapshot’s value is only proven by the drill.

Quiz

Production discipline

• Snapshot before every control-plane change. Always.
• Verify the snapshot. Status + hash.
• Upload off-cluster. Local is one host loss away.
• Document in the change ticket. Hash, revision, location.
• After-snapshot for the new known-good state.
• Rehearse quarterly. A snapshot that has never been exercised is a snapshot that doesn’t restore.

The snapshot is the change gate. Operating without it is operating with no rollback path.