The Kubernetes backup model: four separate things

Backing up container data consistently settled what to copy and when to hold the writer still, and it settled it one volume at a time. A Kubernetes estate does not fail one volume at a time. Ask a team where their Kubernetes backup is and you will usually be shown one artefact — a Git repository, an etcd snapshot, a nightly volume job, a backup operator — and told that this is it. Each of those protects something real. None of them protects what the other three protect, and the distance between them stays invisible until a namespace disappears.

flowchart TB
    Service["Recovered Kubernetes service"]
    Git["Desired state: manifests, charts, values"] -->|"Git mirror or export"| Service
    Etcd["Cluster state: API objects and bindings"] -->|"etcd snapshot"| Service
    PV["Persistent application data"] -->|"application backup or independent volume copy"| Service
    External["External dependencies: registry, DNS, PKI, secrets"] -->|"their own recovery procedures"| Service

The service returns only where all four arrows meet. Reapplying Git can recreate an empty claim, an etcd restore can recreate a binding to missing storage, and a volume restore cannot recreate the identity or DNS path that makes it usable.

Four artefacts, four mechanisms, four ways to be uncovered

Nothing in a Kubernetes cluster deserves to be called the backup, because there are four separate things to protect, each held somewhere else, each protected by a different mechanism, usually on a different schedule and often by a different team.

The desired state is the material a human wrote: manifests, Helm values, Kustomize overlays, the chart versions pinned alongside them. It lives in a Git repository and it is protected by protecting that repository — an independent mirror, a clone that survives the loss of the hosting account, an export of the issues and pipeline definitions that surround it. What matters for recovery is less how it is protected than what it contains. The repository is a complete description of what should exist. It contains nothing whatsoever of what the running system produced.

The cluster state is the set of API objects as etcd holds them at a moment in time, and it is a strict superset of the desired state. The difference is the interesting part. A cluster accumulates objects nobody committed: a Secret created by hand at three in the morning and never written down, a ServiceAccount and its token, a PersistentVolume object minted by a provisioner, a binding between a claim and a volume, objects a controller or an admission webhook wrote on the way past, and the Lease and Endpoint churn that never belonged in a repository at all. Cluster state is protected by an etcd snapshot, and by nothing else on this list.

The persistent application data is the bytes inside the PersistentVolumes: the rows, the uploads, the queue, the ledger. This is the only one of the four that the business would describe as its data, and it is the only one that neither Git nor etcd holds even a copy of. It is protected by a volume-level backup — a storage or CSI snapshot copied somewhere else — or by an application-level backup taken from inside the workload, which is what the previous lesson was about.

The external dependencies are everything the cluster refers to but does not contain. Credentials whose source of truth is a secret manager outside the cluster. TLS certificates and the issuer that mints them. DNS records that point at a load balancer. Cloud resources created by a controller and owned by the cloud account rather than the cluster. The container images the manifests name, sitting in a registry that is a separate system with its own retention policy. A cluster can be rebuilt perfectly and still not start, because the image tag its manifests reference is no longer in the registry.

The design rule that follows is unglamorous: for every item in the estate, name which of the four covers it. Do it item by item rather than service by service, because a single service usually spans three of the four categories and is covered in two of them. And expect the exercise to find at least one item that nothing covers, because the categories are protected by different teams and nobody owns the seam between them.

The manifests came back; orders.csv did not

The claim that the desired state is not the data is easy to assert and easy to believe without ever having felt it, so it is worth executing. A single-node k3s cluster ran a namespace containing a Namespace, a PersistentVolumeClaim, a ConfigMap and a Pod that wrote two order lines to a file on the volume. The YAML was the whole of what a GitOps repository would have held. The order lines were not in it and could not have been.

$ kubectl delete namespace rbdr-shop

namespace "rbdr-shop" deleted
namespace fully removed after 0s

--- is the PersistentVolume data still on disk? ---
GONE - the local-path provisioner reclaimed the volume with the PVC

Deleting the namespace deleted the claim, and deleting the claim released the volume to the provisioner, which reclaimed it. The bytes were gone from the node before anyone had decided that a recovery was needed. Recovery attempt one is the move every team reaches for first, and the capture labels it in exactly the terms teams use: this is what people mean when they say everything is in Git.

$ kubectl apply -f rbdr-shop.yaml

namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 6s

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Read the two halves of that output against each other, because the whole lesson is in the contrast. Every object came back. The apply exited 0, the Pod reached Ready after 6s, the claim reached Bound, the ConfigMap was present, the container was 1/1 Running. A dashboard watching object health would have gone green, and a reconciliation controller would have reported the namespace as synced. Then ls on the mounted directory returned two entries — . and .. — and reading the file returned cat: can't open '/data/orders.csv': No such file or directory with exit code 1.

There is a second detail worth pausing on. Before the deletion, the claim was bound to pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf; after it, to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93. The object name rbdr-orders survived the round trip unchanged because it was written in the manifest. The volume identity did not, because it was never in the manifest to survive. Any restore procedure that hard-codes the volume path it saw yesterday is writing into a directory that the recreated claim is not using.

What the 1,208,352-byte snapshot is a backup of

Before the deletion the capture took a cluster-state snapshot, which is the second of the four mechanisms and the one most often confused with the third.

$ k3s etcd-snapshot save

time="2026-08-28T14:33:54Z" level=info msg="Snapshot rbdr-before-125e3e56d5eb-1787927634 saved."
>>> exit code: 0
snapshot file: rbdr-before-125e3e56d5eb-1787927634
drwx------ 4 root root    4096 Aug 28 14:32 ..
-rw------- 1 root root 1208352 Aug 28 14:33 rbdr-before-125e3e56d5eb-1787927634

One file, 1,208,352 bytes, holding the Namespace, the claim and its binding, the ConfigMap and the Pod spec — and holding them as the API server had them, which includes whatever had been created without passing through the repository. The capture states the boundary in its own words: the data actually protected by that file is Kubernetes objects only, and the data not protected by that file is every byte in every PersistentVolume. An etcd snapshot is a backup of the cluster’s mind and not of its contents.

The file also needs an owner, and the mechanism carries a seam that is easy to inherit without noticing. The k3s CLI documentation separates two kinds of snapshot on this datastore: the scheduled ones the server takes for itself, which are pruned down to a retention count, and the on-demand ones produced by k3s etcd-snapshot save, which that documentation states have no retention applied and stay in place until somebody removes them. The asymmetry cuts both ways. A snapshot taken deliberately before a risky change is exactly the artefact a recovery wants to find, and it is also the artefact that accumulates unwatched on a control-plane node. More important for this course is where the file landed: on the node whose datastore it describes. A copy that lives only there shares the failure domain it was taken to survive, so a cluster-state snapshot becomes a recovery mechanism only once something moves it somewhere the cluster cannot reach.

The objects-only boundary has a second direction, and it is the one that gets argued about. In an estate reconciled continuously from Git, the desired state is already covered, so an etcd snapshot can look redundant. It is not, because it is the only artefact that holds the objects nobody committed. If the answer to “what is in this cluster that is not in the repository” is “nothing”, that is a claim worth checking rather than assuming, and the checking is usually where the ad-hoc Secret and the hand-edited object turn up.

The dependencies that are not in the cluster at all

The fourth category is the one that gets discovered during the rebuild, because it is the only one with no artefact in the cluster to remind anybody it exists.

Start with credentials. A workload that fetches its database password from an external secret manager has, in the cluster, only a reference: a custom resource, a mounted projection, an annotation. Recreate the objects and the reference comes back pointing at a system that may not have been part of the recovery plan, holding a version of the credential that may no longer match what the restored database expects. The same holds for certificates: the Certificate object is desired state, the issued key material is cluster state, and the issuing authority is neither — it is a separate service with its own recovery story, and rebuilding a cluster whose issuer is unavailable produces a namespace full of workloads that will not accept a connection.

Then the resources the cluster asked the world to create on its behalf. A Service of type LoadBalancer produced an address in a cloud account. A DNS record points at that address, and the record lives in a zone the cluster does not own. An ingress controller obtained a certificate tied to that name. Every one of those is reconstructable, and none of them is reconstructed by re-applying a manifest — the manifest asks for a load balancer, gets a new one, and the DNS record still points at the old address until somebody changes it.

Images are the quietest failure of the four. Manifests reference images by tag or digest, and the registry that holds them is a separate system with its own retention, its own storage and its own credentials. A cluster rebuilt from a perfect repository into a working control plane will produce Pods that cannot start if the referenced image is no longer pullable, and the recovery then waits on rebuilding an image from a source tree that has moved on since the tag was cut. The registry belongs on the same inventory as the volumes.

Both, or neither

The capture finishes by doing the thing that actually works, which is using two of the four mechanisms together. The first disaster had already destroyed the original file, so the capture had to recreate the two order lines before it could demonstrate backing them up; the archive it then wrote from the node directory backing the second claim was 2560 bytes. The namespace and its volume were destroyed a second time, the manifests recreated the objects into a third directory — pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787_rbdr-shop_rbdr-orders, sharing no identifier with either of the first two — and the archive was unpacked into that new directory.

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Both md5 sums read 9eb4e2ad8e08e1dcaaf87ababab964b0, which is the value the capture recorded when the application first wrote the file and which the recreated copy reproduced exactly, so the comparison is a real one. The capture draws the conclusion itself, and it is worth quoting rather than paraphrasing: full recovery needed both the desired state to rebuild the objects and a separate data backup to refill the volume, and neither alone was enough.

Notice what the successful recovery required that the failed one did not. It needed a copy of the bytes taken before the loss, and it needed the restore to target the directory backing the new claim rather than the old one. The second requirement is a procedural one and it is where hand-written runbooks tend to break, because the path that was correct when the runbook was written is a path that no longer exists.

Production discipline

1. Classify before you buy. Decide which of the four categories a proposed tool covers before evaluating it. A tool that captures API objects and a tool that captures volume contents are answering different questions, and a product that does both still does them as two operations with two failure modes.
2. Keep an etcd snapshot even under GitOps. The 1,208,352-byte file in the capture held the objects as the API server had them, including anything created outside the repository. Reconciliation cannot restore an object that was never committed.
3. Never let an object-level restore stand in for a data restore. In the capture the apply exited 0, the Pod reached Ready after 6s and the claim reached Bound, and the file the business cared about returned cat: can't open '/data/orders.csv': No such file or directory.
4. Resolve volume paths at restore time. The claim came back as rbdr-orders and bound to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, a different volume from the pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf it had before, and the capture went through a third identifier before it finished. A procedure that hard-codes yesterday’s path restores into a directory nothing is reading.
5. Inventory the fourth category by name. List the registry, the DNS zone, the certificate issuer, the external secret store and the cloud resources as individual items with owners. They are the part of the estate that no cluster-side backup mechanism can see, and therefore the part that no cluster-side gap analysis will report.

Cross-course references

• Kubernetes for Production Sysadmins — Part LXVIII (etcd Backup) and Part XCVI (Workload Backup) build the two mechanisms this lesson only classifies, covering how a snapshot is taken and verified and what a workload-level backup captures; use this lesson to decide which of the four categories an item needs, and those parts to operate the mechanism you chose.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part LXXII (GitOps Foundations) and Part LXXV (Drift) explain the reconciliation loop that made the re-apply above succeed so completely, which is exactly why it is dangerous as a recovery story: reconciliation converges objects to the committed state and has no opinion at all about the bytes in a volume.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) covers the systems that hold the credentials a cluster only references, which is the fourth category in this lesson; a cluster rebuilt without its secret platform and issuer restores every object and still cannot authenticate anything.

Quiz