Workload backup principles — etcd is not a backup

Workload backup in Kubernetes is not the same as backing up etcd. etcd is the source of truth for cluster state — manifests, RBAC, ConfigMaps, Service accounts — but it does not contain the contents of PersistentVolumes. An etcd restore brings the API back online with all objects reanimated, but every PVC is empty until its underlying storage is restored separately. This lesson establishes what must be backed up for a workload to be restorable, and the operational discipline that turns “we have backups” into “we can recover.”

What etcd contains and what it does not

etcd holds every Kubernetes object — Pods, Services, Secrets metadata, ConfigMaps, RBAC, CRDs, leases, the control-plane’s view of every node — but it holds references to data, not the data itself. A PVC in etcd is metadata: name, namespace, storage class, capacity, volume mode. The bytes inside the volume live on whatever the StorageClass provisions — a CSI volume on cloud block storage, a hostPath, an NFS mount, a Ceph RBD image, a Longhorn replica.

flowchart LR
    A[etcd snapshot] --> B[Cluster state]
    B --> B1[Workload manifests]
    B --> B2[RBAC]
    B --> B3[ConfigMaps]
    B --> B4[PVC metadata]
    B --> B5[Secrets metadata]
    C[Volume backup] --> D[Workload data]
    D --> D1[PVC bytes]
    D --> D2[Application state]
    D --> D3[Database files]
    E[Application backup] --> F[Consistent state]
    F --> F1[Quiesced DB]
    F --> F2[Pre-crash dump]

A cluster can be restored from an etcd snapshot alone only if the storage backend is preserved and the PV-to-volume mapping is intact. That is almost never true after a cluster loss. A workload backup needs both the manifest set and the volume contents.

The workload backup scope

A workload backup that supports real recovery must cover six categories:

Every category has a different storage mechanism and a different failure mode. Skipping any one of them makes a restore either impossible or partial.

The 3-2-1 rule and Kubernetes

The 3-2-1 rule: three copies of the data, on two different media, with one copy off-site. For Kubernetes this translates to:

• Three copies. Original (live cluster), one local backup (fast restore), one remote backup (off-cluster).
• Two media types. etcd snapshots are files; volume snapshots are CSI primitives; application dumps are export files. Mixing media prevents correlated failure.
• One off-site. The remote backup lives in a different failure domain — a different region, a different account, a different cloud. The point is that one bad region does not delete every copy.

# Three copies
velero backup create daily-full --include-cluster-resources=true

# Two media types — the Velero backup stores manifests in object storage
# and CSI snapshots in the storage backend; application dumps live in S3.
restic -r s3:s3.amazonaws.com/backup-bucket/etcd snapshots

# One off-site
aws s3 sync s3://primary-bucket s3://dr-bucket --source-region us-east-1 --region us-west-2

A 3-2-1 program without restore validation is still incomplete. The next lesson in this part covers the discipline of proving that a backup is restorable.

What must be backed up that is not in the cluster

A backup program that ignores out-of-cluster dependencies restores a cluster that cannot start its workloads:

• External databases. A workload that connects to an RDS Postgres needs the RDS snapshot, the credentials, and the network reachability documented and backed up on the same cadence.
• DNS records. A workload behind an Ingress depends on the DNS records pointing to the Ingress’s external address. The cluster’s restore is incomplete if the DNS still points to the old cluster.
• TLS certificates. A workload that serves HTTPS needs valid certificates at the new Ingress. cert- manager will reissue, but only if the ACME account or CA is intact.
• Object storage buckets. A workload that writes to S3 needs the bucket contents. Versioning and cross- region replication are the backup program here.
• Service mesh control plane. A workload that depends on Istio/Linkerd’s control plane needs the mesh configuration backed up or re-bootstrapped.
• CRDs and operators. Restoring manifests is meaningless if the CRD definitions themselves are gone. Back up CRDs first, then objects.

The operational discipline

Workload backup’s operational discipline:

• Document what is backed up. The cluster operator must list every category and the cadence.
• Verify backups run. The cluster operator must monitor the backup schedule for failures.
• Test restore quarterly. A backup that has never been restored is a hope, not a backup. The test is the proof.
• Test the worst case. The restore test must cover the complete cluster loss scenario, not just one PVC.
• Document the restore runbook. The runbook is the operator’s reference during an incident.
• Train the operations team. The team must be able to execute the restore under pressure.

Cross-course references

• The Linux course covers etcdctl snapshot, restic, borg, and pg_basebackup — the underlying tools a Kubernetes backup program rests on.
• The Observability course covers monitoring the backup schedule — alerting on velero_backup_total_status and the equivalent CSI metrics.
• The VyOS course covers off-site replication paths — the network the remote backup traverses.

Quiz

Production discipline

A workload backup program in production rests on five non-negotiable elements:

• Cover all six categories. Manifests, volumes, application-consistent state, secrets, external dependencies, and configuration. A program that misses any one of these is incomplete.
• Apply the 3-2-1 rule. Three copies, two media, one off-site. The rule is the minimum; more copies are fine, fewer are not.
• Schedule backups independently. etcd snapshots, Velero backups, and application exports should run on different cadences with different retention — so one broken job does not delete every copy.
• Validate restores quarterly. The restore test must include a complete cluster loss scenario, not just one PVC.
• Document and train. The restore runbook is the operator’s reference during the incident. The team must be able to execute it under pressure.

Workload backup is a production discipline, not a checkbox. A program without restore validation is hope, not a backup.