Why back up etcd — the failure modes only snapshots cover

etcd snapshots are the cluster’s source of truth, taken off-cluster for safe storage and used to recover from the class of failures that affect the cluster’s state itself. They are not application backups; they are control-plane backups. This lesson walks what they cover, what they do not, and the failure modes that make them a non-negotiable operational discipline.

What etcd snapshots cover

An etcd snapshot is a consistent point-in-time copy of every API object the cluster records:

flowchart LR
    E[etcd db file] -->|snapshot| S[snapshot file]
    S -->|restore on new cluster| R[new cluster's state]
    S -->|store off-cluster| O[off-cluster storage]
    O -->|recovery target| R

The objects that are covered:

• Workload definitions. Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, and their revision histories.
• Configuration. ConfigMaps, ResourceQuotas, LimitRanges, HPAs, PodDisruptionBudgets.
• Service identity. Services, Ingresses, NetworkPolicies, EndpointSlices.
• Access control. Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, ServiceAccounts (the SA objects and the projected tokens’ stored halves).
• Stateful definitions. PersistentVolumeClaims, StorageClasses (cluster-scoped), CustomResourceDefinitions and their instances.
• Cluster state. Nodes (their objects in etcd), leases (rebuilt), events (subject to GC), audit log (if configured to write through etcd).

What is not in an etcd snapshot:

• Application data in PVCs. The PVC object is in etcd; the bytes written by the application are in the storage backend (CSI driver, NFS, Ceph, etc.).
• Container logs. Container logs are in /var/log/pods/... on the nodes; etcd has only the Event object that triggered them.
• Container runtime cache. The running state of containers is in the kubelet’s local cache.
• Real-time metrics. Cluster metrics are in Prometheus; the snapshot has only the metric exporter objects.

Failure modes an etcd snapshot recovers

Accidental deletion of cluster-scoped objects

A kubectl delete of a Namespace, a CRD, or an entire RBAC structure can tear down dozens of objects in seconds. The git history of the manifests helps re-create, but the runtime state (ReplicaSets’ replicas, status fields, controller-revision hashes) is lost. An etcd snapshot that pre-dates the deletion restores the cluster’s state exactly as it was.

Corruption of etcd data

A bbolt file can become corrupt (LXVI-03 walked the mechanism). The corrupt member can be wiped and re-added; but if multiple members’ data is corrupt, the cluster cannot recover without a snapshot restore.

Failed upgrade or RBAC restructure

A Kubernetes upgrade that goes wrong can leave the cluster in a state where nodes cannot register, RBAC bindings are missing, or controllers crash. A snapshot pre-upgrade is the rollback primitive.

Disaster recovery across a region or site

A regional failure that takes the data centre down requires rebuilding the cluster on new infrastructure. The etcd snapshot from before the failure is the seed for the new cluster’s state.

Security incident response

A cluster compromised by a malicious operator account or a leaked ServiceAccount token can have state owned by the attacker. A snapshot pre-incident is the recovery state; the response sequence is restore the snapshot, revoke all credentials, change all Secrets, then re-issue access.

Failure modes an etcd snapshot does NOT recover

A snapshot does not replace a workload backup. It is a control-plane backup.

The backup-and-restore layered model

flowchart TB
    L1["Layer 1: etcd snapshot (control plane)"] --> R1["etcd snapshot restore"]
    L2["Layer 2: workload backup"] --> R2["CSI snapshots, database dumps"]
    L3["Layer 3: Git desired state"] --> R3["git reapply"]

A complete backup strategy combines all three layers:

• Layer 1: etcd snapshot covers API state.
• Layer 2: workload backup (Velero, database dumps) covers the bytes applications wrote into PVCs.
• Layer 3: Git holds the desired state; a complete restore would reapply it.

In a complete DR scenario, the recovery sequence is:

1. Restore etcd from snapshot (control-plane state).
2. Bring up the storage platform (so PVs can bind PVCs).
3. Restore workload bytes from workload backups.
4. Re-apply Git for any state that drifted from snapshot.
5. Validate cluster and workloads.

What a backup protects against

What an etcd snapshot is for:

• Recovery from cluster corruption.
• Recovery from accidental mass deletion.
• DR across site or region failure.
• Compliance requirement for point-in-time cluster state.
• Sandbox restores for testing (“give me a copy of last week’s cluster”).

What a snapshot is not for:

• Real-time replication (the cluster itself replicates writes; snapshots are point-in-time).
• Selective object recovery (a snapshot is whole-cluster; restore is whole-cluster).
• Workload data recovery (PVC bytes are separate).

$ etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db

"saved snapshot to /backup/etcd-snapshot-20260816-1200.db"

The “I don’t need backups” anti-pattern

A common anti-pattern: “we have a high-availability 3-member etcd, so we don’t need off-cluster backups”. The HA covers availability — the cluster keeps serving writes. Backup covers data integrity — the cluster’s state can be recovered after corruption or human error.

These are distinct properties:

• HA without backup. A cluster survives host loss; if all three hosts are corrupted, the cluster’s state is gone.
• Backup without HA. Snapshots can be restored to a new cluster, but during a member loss the cluster is offline (in the case of a 1-member setup).

Both are necessary.

The matrix of recovery scenarios

The pattern: anything that affects etcd’s stored state needs a snapshot.

Operational placement of the snapshot

A production snapshot plan:

gantt
    title etcd snapshot cadence
    dateFormat HH:mm
    axisFormat %H:%M
    section Hourly
    Snapshot to off-cluster :a1, 00:00, 5m
    Verify snapshot integrity :a2, after a1, 1m
    section Daily
    Upload to object storage :b1, 01:00, 5m
    section Weekly
    Test restore on isolated host :c1, Sat 02:00, 60m
    section Monthly
    Audit snapshot chain integrity :d1, 1st of month, 5m

The cadence:

• Hourly: snapshot to a known path on a control-plane host.
• Daily: off-cluster transfer (object storage, second region).
• Weekly: restore drill.
• Monthly: chain integrity audit.

This cadence covers the typical recovery scenarios within the hour they occurred, with off-cluster copies surviving site failure.

Quiz

Production discipline

• Snapshot hourly. Storage is cheap; data loss is expensive. An hour-old snapshot is the worst case.
• Snapshot off-cluster. A snapshot on the same host as the etcd member is a snapshot that is lost with the host.
• Test the restore quarterly. A snapshot that has never been restored is a snapshot you don’t know how to restore.
• Cover workload data separately. Etc snapshot does not cover PVC bytes; Velero or CSI snapshots cover that.
• Treat the snapshot’s staleness as a known lost window. Document it in the post-mortem; recover the lost window from Git or workload backups where possible.

Etcd backup is the foundation that every DR scenario relies on. Operating it well is what makes the rest of the cluster’s incident response possible.