Pre-upgrade backup — the snapshot that catches the rollback

The pre-upgrade etcd snapshot is the rollback path for any Kubernetes upgrade. If the upgrade fails or breaks the cluster, the snapshot is the recovery. This lesson walks the snapshot procedure, the verification, and the storage strategy.

When to take the snapshot

The snapshot is taken immediately before the upgrade begins. The sequence:

flowchart LR
    A[Read release notes] --> B[Take snapshot]
    B --> C[Verify snapshot]
    C --> D[Copy off-cluster]
    D --> E[Begin upgrade]

The snapshot is the pre-condition for the upgrade. No upgrade proceeds without a verified snapshot.

The snapshot command

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd-pre-upgrade-$(date +%Y%m%d-%H%M%S).db

The arguments:

• --cacert: the etcd CA cert.
• --cert, --key: the healthcheck-client cert pair.
• --endpoints: the etcd member address. For stacked etcd, this is https://127.0.0.1:2379.
• snapshot save: the subcommand.
• The output path is timestamped for identification.

The output:

{"level":"info","ts":"2026-08-16T10:00:00.000Z","logger":"snapshot","caller":"snapshot.go:110","msg":"saving snapshot","path":"/var/backups/etcd-pre-upgrade-20260816-100000.db"}
{"level":"info","ts":"2026-08-16T10:00:01.500Z","logger":"snapshot","caller":"snapshot.go:130","msg":"saved","path":"/var/backups/etcd-pre-upgrade-20260816-100000.db","wal-apply-temp-dir":"/tmp/etcd-snapshot-...","total-keys":12453,"total-size":67108864}

The snapshot is saved; the total keys and total size are reported.

The verification

sudo etcdutl snapshot status /var/backups/etcd-pre-upgrade-*.db --write-out=table

+----------+----------+------------+------------+
|   HASH   | REVISION |   KEYS     |   SIZE     |
+----------+----------+------------+------------+
| 1a2b3c4d | 12345    | 12453      | 67108864   |
+----------+----------+------------+------------+

The output confirms:

• The snapshot is a valid etcd snapshot (HASH is non-zero).
• The revision is the etcd’s last applied revision.
• The keys count is the total number of keys in the snapshot.
• The size is the on-disk size.

A snapshot with HASH: 0 is invalid. The verification fails; the upgrade does not proceed.

The off-cluster copy

The snapshot is uploaded off-cluster:

# Upload to S3
sudo aws s3 cp /var/backups/etcd-pre-upgrade-*.db s3://k8s-backups/etcd/

# Or copy to a network file system
sudo cp /var/backups/etcd-pre-upgrade-*.db /mnt/external/etcd/

# Or copy to a backup server
sudo scp /var/backups/etcd-pre-upgrade-*.db backup@backup-server:/backups/etcd/

The off-cluster copy is the disaster recovery path. If the control plane host is unrecoverable, the snapshot is available from the off-cluster store.

$ sudo etcdutl snapshot status /var/backups/etcd-pre-upgrade-*.db --write-out=table

+----------+----------+------------+------------+
|   HASH   | REVISION |   KEYS     |   SIZE     |
+----------+----------+------------+------------+
| 1a2b3c4d | 12345    | 12453      | 67108864   |
+----------+----------+------------+------------+

The snapshot retention

The snapshot retention strategy:

• Daily snapshots for 7 days.
• Weekly snapshots for 4 weeks.
• Monthly snapshots for 12 months.
• Pre-upgrade snapshots for the duration of the upgrade
  • 30 days.

The retention is implemented via a cron job or a backup tool (Restic, Velero, Bacula).

The snapshot during the upgrade

The snapshot is taken on the leader. The leader’s etcd member is the snapshot’s source. On a stacked etcd cluster, each member has its own copy of the data; the snapshot from the leader is representative of the cluster state.

On an external etcd cluster, the snapshot is taken on one member. The snapshot is consistent across the cluster because etcd’s raft log is the same on all members.

The snapshot during etcd upgrade

If the upgrade includes an etcd version change (--etcd-upgrade=true), the snapshot is taken before the etcd upgrade. The etcd upgrade replaces the etcd binary; the snapshot is the previous data. If the new etcd cannot read the previous data, the snapshot is the recovery.

sudo kubeadm upgrade apply v1.34.1 --etcd-upgrade=true

The command:

1. Takes an etcd snapshot (if --etcd-upgrade=true).
2. Upgrades the etcd binary.
3. Migrates the data format if needed.
4. Restarts the etcd static pod.

The pre-upgrade snapshot is the safety net for the etcd upgrade.

The snapshot integrity

The snapshot is verified by:

1. etcdutl snapshot status: confirms the internal metadata.
2. etcdutl snapshot restore (in a test environment): confirms the snapshot can be restored.
3. Off-cluster copy verification: the off-cluster copy is a checksum-verified copy.

The verification is mandatory.

Cross-course references

• The Linux course covers filesystem snapshots (LVM, ZFS).
• The Observability course covers backup monitoring (e.g., the existence of today’s snapshot).
• The Ansible course covers backup automation.

Quiz

Production discipline

• Never upgrade without a snapshot. The snapshot is the rollback path.
• Verify the snapshot. etcdutl snapshot status confirms the integrity.
• Store off-cluster. S3, NFS, backup server. The on-cluster copy is not enough.
• Timestamp the snapshot. The filename is the optimization target.
• Track the snapshot in the runbook. The snapshot ID, the timestamp, the off-cluster location.
• Test the snapshot. Periodic restore-drill on a staging cluster.

The snapshot is the safety net. Operating it well is taking it before every upgrade, verifying it, and copying it off-cluster.