etcd backup — the cluster state protection

The etcd backup is the cluster state protection. The snapshot, the storage, the encryption, the schedule, the verification, the recovery are the components. This lesson walks the etcd backup, the verification, the recovery, and the production patterns.

The etcd backup

The etcd backup:

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd-$(date +%Y%m%d-%H%M%S).db

The etcd backup is the snapshot.

The etcd backup verification

The etcd backup verification:

sudo etcdutl snapshot status /var/backups/etcd-20260816-100000.db --write-out=table

+----------+----------+------------+------------+
|   HASH   | REVISION |   KEYS     |   SIZE     |
+----------+----------+------------+------------+
| 1a2b3c4d | 12345    | 12453      | 67108864   |
+----------+----------+------------+------------+

The verification confirms the snapshot is valid.

The etcd backup storage

The etcd backup storage:

# Substitute your own KMS key before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab

# Upload to S3
aws s3 cp /var/backups/etcd-20260816-100000.db s3://k8s-backups/etcd/

# Encrypt with KMS
aws s3 cp /var/backups/etcd-20260816-100000.db s3://k8s-backups/etcd/ \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

The storage is the 3-2-1 rule.

The etcd backup encryption

The etcd backup encryption:

# Substitute your own KMS key before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab

# Encrypt the etcd backup
gpg --symmetric --cipher-algo AES256 /var/backups/etcd-20260816-100000.db

# Or use KMS
aws s3 cp /var/backups/etcd-20260816-100000.db s3://k8s-backups/etcd/ \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

The encryption is the data protection.

The etcd backup schedule

The etcd backup schedule:

# /etc/cron.d/etcd-backup
0 2 * * * root /usr/local/bin/etcd-backup.sh

The schedule is the daily backup.

The etcd backup script

The etcd backup script:

#!/bin/bash
# /usr/local/bin/etcd-backup.sh
set -e

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="/var/backups/etcd"
BACKUP_FILE="${BACKUP_DIR}/etcd-${TIMESTAMP}.db"
# Substitute your own KMS key before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab

mkdir -p ${BACKUP_DIR}

# Take the snapshot
sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save ${BACKUP_FILE}

# Verify the snapshot
sudo etcdutl snapshot status ${BACKUP_FILE}

# Upload to S3
aws s3 cp ${BACKUP_FILE} s3://k8s-backups/etcd/ \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

# Cleanup local backups older than 7 days
find ${BACKUP_DIR} -name "etcd-*.db" -mtime +7 -delete

The script is the operational discipline.

The etcd backup for stacked etcd

The etcd backup for stacked etcd:

# Stacked etcd: each control plane has the etcd
# Backup on each control plane host
for host in cp-1 cp-2 cp-3; do
  ssh ${host} sudo /usr/local/bin/etcd-backup.sh
done

The stacked etcd backup is per host.

The etcd backup for external etcd

The etcd backup for external etcd:

# External etcd: backup on the etcd hosts
# The etcd members are on separate hosts
for host in etcd-1 etcd-2 etcd-3; do
  ssh ${host} sudo /usr/local/bin/etcd-backup.sh
done

The external etcd backup is per host.

The production patterns

The production patterns:

flowchart LR
    A[etcd] --> B[Daily snapshot]
    B --> C[Verify]
    C --> D[Encrypt]
    D --> E[Upload to S3]
    E --> F[3-2-1 rule]
    F --> G[Offsite]

The pattern is the production discipline.

The cross-course references

• The etcd course covers the snapshot.
• The S3 course covers the storage.
• The KMS course covers the encryption.

Quiz

Production discipline

• Take the daily snapshot. The etcd backup.
• Verify the snapshot. etcdutl snapshot status.
• Encrypt the backup. The KMS encryption.
• Upload to S3. The 3-2-1 rule.
• Test the backup. The verification.
• Document the etcd backup. The script, the cron.

The etcd backup is the cluster state protection. Operating it well is the snapshot, the verification, the encryption, the storage, and the production patterns.