high riskservice affecting~90 min

Recover Kubernetes application data

1 · Prerequisites

Confirm every item is in place before any state change.

The manifests for the affected namespace, and the revision that was deployed when the data was lost — which is not necessarily the tip of the branch
A backup of the PersistentVolume contents taken independently of the manifests, carrying its own timestamp
A checksum, or a manifest of checksums, recorded at backup time for the files being recovered. Without it the validation step has nothing to compare against and the recovery cannot be declared
Cluster access sufficient to delete and re-create the namespace, and access to the node directory the volume provisioner writes into
A named owner for the application data who can authorise discarding anything written after the chosen recovery point
An agreed answer to whether the workload may be stopped, because one of the two restore routes requires it and the other does not
A second terminal on the node, so the volume directory can be inspected while the cluster objects are being rebuilt

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish which of the three things is missing. The manifests, the API objects, and the bytes in the volume are separate, and only the first lives in the repository. A namespace deletion removes the second and, with a provisioner that reclaims on release, the third as well. Recovering the wrong one wastes the window.
· Confirm the namespace is fully gone before re-applying anything. kubectl wait --for=delete namespace/rbdr-shop --timeout=300s, then kubectl get namespace rbdr-shop must report NotFound. A namespace in Terminating accepts no new content, so an apply against it reports unchanged for what still exists and creates nothing, and the pod that should follow never schedules. Exit code 0 does not distinguish the two cases.
· Read the archive before trusting it. tar tf "$SNAP" | head must list the paths you expect, relative to the volume root. An archive whose members are absolute, or rooted one directory too high, restores into the wrong place and looks like it worked.
· Confirm you hold a checksum recorded at backup time. A checksum computed after the restore, from the restored file, compares the file to itself and always matches.
· **Record the current volumeName before changing anything.** kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.spec.volumeName}'. The provisioner names the node directory after the volume, so this string is the only link between the claim and the bytes.
· Confirm the node has room for a second copy if you intend to restore into a fresh claim rather than into the existing one. Two copies of the dataset is the cost of keeping the failed state for investigation.
· Confirm nothing is still writing to the target volume. A pod with the file open will re-create or truncate what you extract, and the checksum comparison at the end will fail for a reason that has nothing to do with the archive.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Select the data recovery point separately from the manifest revision. These are two independent choices with two independent clocks. The manifest revision is chosen for compatibility — the schema, image tag and volume size the application expects. The data recovery point is chosen as the last archive taken before the destructive event. Nothing in the manifest names an archive, so no repository revision can select one for you.
2Write both choices down before touching the cluster. The manifest revision, the archive path, the archive timestamp, and the amount of work being discarded between that timestamp and the incident. This is the number the data owner is authorising, and it is the one people later dispute.
3Wait for the namespace to be fully terminated. kubectl wait --for=delete namespace/rbdr-shop --timeout=300s. Do not begin the apply on the assumption that the delete has finished; confirm it.
4Re-apply the manifests at the selected revision and expect the objects, not the data. Every object comes back and the volume comes back empty. That is the correct, complete behaviour of a manifest apply, and it is the reason this runbook has a second half.
5Prove the volume is empty rather than assuming it. Read the file the application needs. A directory listing showing only . and .., and a cat that fails with No such file or directory, is the evidence that the object recovery succeeded and the data recovery has not started.
6Locate where the volume actually stores files on the node. The claim does not tell you; the bound volume does. Resolve .spec.volumeName, then find the provisioner directory named after it. On the captured k3s local-path provisioner that directory is /var/lib/rancher/k3s/storage/<volumeName>_<namespace>_<claim>.
7Take the decision point: scale the workload to zero, or restore into a fresh claim. Scale to zero when the archive is a full copy of the volume, the claim is ReadWriteOnce and already bound, and the application cannot tolerate reading a half-extracted tree. Restore into a fresh claim when the recovery point is uncertain and you may need to try another, when the failed state must be preserved for investigation, or when the outage of stopping the workload is not authorised. The first is faster and destroys the evidence; the second is slower and leaves you a second claim to clean up.
8If scaling to zero, stop the workload and wait for the pods to be gone, not merely for the scale command to return. A terminating pod still holds the volume.
9Restore into the directory that exists now, not the one named in the archive. The volume is new, so its directory name contains a new volume identifier that did not exist when the archive was taken. Extract with an explicit destination — tar xf "$SNAP" -C "$NEWDIR" — and never rely on a path recorded inside the archive.
10Restore the ownership and mode the application expects. A tree that arrives owned by root under a container running as a non-root user reads as corruption to the application and as success to the responder.
11Start the workload and read the application, not the pod status. Running means the kubelet started a container. It says nothing about whether the file is there or parseable.
12Run the business validation: compare the recovered checksum against the one recorded at backup time, and make the comparison refuse two empty values. An equality test between two empty strings passes, which is exactly how a missing file and a missing checksum record are reported as a successful recovery.
13Exercise one real business transaction end to end. Read a record the business recognises, through the application, by the route a user would take. A file that is byte-identical and an application that cannot read it are different outcomes.
14Record the outcome while the terminal is still open: the manifest revision, the archive used, the old and new volume identifiers, both checksums, the elapsed time split between object rebuild and data restore, and the work discarded.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The namespace was confirmed deleted before the apply, by an explicit wait, not by elapsed time.
✓Every object from the manifests is present: the namespace, the claim in Bound, the ConfigMap, and the workload pod in Running.
✓Before the data restore, the volume directory listed only . and .., and the read of the application file failed. This negative result is recorded, because it is what separates an object recovery from a data recovery.
✓The node directory used for the restore matches the claim's current .spec.volumeName, and is not the directory recorded in the archive or in the pre-incident notes.
✓The restored tree carries the ownership and mode the workload runs as, checked from inside the pod rather than from the node.
✓The recovered checksum and the checksum recorded at backup time are both non-empty and equal. A comparison where either side is empty is reported as REFUSED, never as a match.
✓One business transaction has been read through the application and reconciled against a value the data owner recognises.
✓The elapsed time is recorded split into object rebuild and data restore, because the two scale with entirely different things and a single total hides which one breached the recovery time objective.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Before the extract, everything is reversible: the objects can be deleted and re-applied at no cost beyond the time. The archive is untouched, so a wrong recovery point costs one more extract.
↶If the extract went into the wrong directory — the old volume identifier, or the provisioner root — remove only what the archive placed there, using tar tf to enumerate its members. Do not clear the directory wholesale; on a shared provisioner root that removes other applications' volumes.
↶If the workload was scaled to zero, restore the original replica count from the manifest revision you recorded, not from memory. Scaling back to a remembered number is how a two-replica service comes back as one.
↶If you restored into a fresh claim and the result is wrong, leave the original claim bound and unmodified and delete only the new one. This is the whole reason that route exists.
↶Reverse the reflex mutations too: the alert silence, the paused GitOps sync or suspended reconciliation, the temporarily relaxed readiness probe, the scaled-down neighbour that was in the way. Each one was made in seconds and none of them expire on their own.
↶If the application was started and wrote to a partially restored volume, do not attempt to merge. Stop it, restore the volume again from the archive, and treat the writes since the start as discarded — with the data owner told, not assumed.
↶Keep the archive and the failed volume until the recovered application has been validated and has served real traffic. Deleting the evidence is the last step, not the first.

6 · Escalation

When the runbook isn't enough, contact:

· The namespace will not leave Terminating within the timeout: escalate to whoever owns the finalizers on the blocked resources. Force-removing a finalizer strands the resources it was protecting, and that is a decision with an owner.
· The archive does not list the paths you expect, or lists nothing: escalate to the backup owner before extracting. An archive whose shape is wrong restores silently into the wrong place.
· No checksum was recorded at backup time: escalate rather than substituting one computed after the restore. Without an independent reference the recovery cannot be validated, only asserted, and the responder should not be the one who decides that is acceptable.
· The newest archive predates a schema migration the current application cannot read: escalate to the application owner. Recovering to a point the code cannot open is a joint decision between the data recovery point and the manifest revision, and it is not the responder's alone.
· The volume contains data written after the incident that nobody has agreed to discard: escalate to the data owner before extracting. The extract is the moment that choice becomes irreversible.
· The recovery is on the critical path of an outage and the data restore is exceeding the recovery time objective: escalate to the incident owner with the measured split between object rebuild and data restore. Bringing the service up on an empty volume is somebody else's decision.
· The loss looks deliberate rather than accidental: escalate to security before restoring. The restore overwrites the state that shows what else was touched.

Three separate things exist for a stateful workload: the manifests, the API objects the cluster holds, and the bytes in the volume. A repository is a complete description of the first. This runbook exists because a recovery that only rebuilds the first two looks finished.

The apply returns every object and an empty volume

Re-applying the manifests is not a partial recovery. It is a complete recovery of the thing the manifests describe.

Configuration changere-apply the desired state after the namespace was deleted— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 5 of docs/courses/backup-dr/execution-evidence/kubernetes-desired-state-vs-application-data.txt.

$ kubectl apply -f rbdr-shop.yaml

  namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created

The capture recorded exit code 0 and the pod Ready after 6s. Six seconds is the whole of the object recovery, and it is the number people quote.

Then read the file the business needs. This is the measurement that matters, and it is the one nobody takes.

Read-only / Safethe volume that came back with the objects— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 5 of docs/courses/backup-dr/execution-evidence/kubernetes-desired-state-vs-application-data.txt.

$ kubectl -n rbdr-shop exec rbdr-orders-writer -- sh -c 'ls -la /data; cat /data/orders.csv'

  total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Record that failure. It is the boundary between the half of the recovery that is finished and the half that has not started.

Wait for the namespace to be gone

A namespace in Terminating accepts no new content. An apply against it reports unchanged for the objects the old namespace still holds and creates nothing new, so the command succeeds, the reader believes the rebuild worked, and the pod that should have followed never schedules. Confirm the deletion instead of timing it.

kubectl delete namespace rbdr-shop --wait=false
kubectl wait --for=delete namespace/rbdr-shop --timeout=300s
kubectl get namespace rbdr-shop

The third line is the one that matters: it must report NotFound.

Recovery point selection, which the manifests cannot make for you

The manifest revision and the data recovery point are two choices with two clocks. Pick the revision for compatibility — the image tag, the schema and the volume size the application expects. Pick the archive as the last one taken before the destructive event, and state how much work falls between that timestamp and the incident.

Nothing in the repository can make the second choice. Across the captured sequence the claim was bound to pvc-6edd5db0-..., then pvc-a29538af-..., then pvc-ef5cf541-..., while the manifests never changed. The identifier that names the data is generated at bind time and appears in no revision.

Find where the volume actually is

The claim does not tell you where the files are. The bound volume does.

Read-only / Safethe node path behind the claim— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 2 of docs/courses/backup-dr/execution-evidence/kubernetes-desired-state-vs-application-data.txt.

$ find /var/lib/rancher/k3s/storage -name orders.csv

  /var/lib/rancher/k3s/storage/pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf_rbdr-shop_rbdr-orders/orders.csv

Resolve it from the claim rather than by searching, so the path you restore into is provably the one the pod is using:

VOLNAME=$(kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.spec.volumeName}')
NEWDIR="/var/lib/rancher/k3s/storage/${VOLNAME}_rbdr-shop_rbdr-orders"
test -d "$NEWDIR" || { echo "ABORT: no provisioner directory for $VOLNAME"; exit 2; }
echo "$NEWDIR"

Decision point: scale to zero, or restore into a fresh claim

	Scale the workload to zero	Restore into a fresh claim
Choose when	The archive is a full copy, the claim is RWO and bound, a partial read would corrupt the application	The recovery point is uncertain, the failed state is evidence, or the outage is not authorised
Costs	An outage for the length of the extract	A second copy of the data, a manifest edit, and a claim to clean up
Destroys	The failed volume, immediately	Nothing; both remain until you choose

kubectl -n rbdr-shop scale deployment/rbdr-orders --replicas=0
kubectl -n rbdr-shop wait --for=delete pod -l app=rbdr-orders --timeout=120s

Wait for the pods to be gone, not for the scale command to return.

Restore into a directory that did not exist when the archive was taken

The volume is new. Its directory carries a volume identifier generated after the archive was written, so no path inside the archive is usable.

The captured run took its archive from pvc-a29538af-... and extracted it into a directory that only came into existence afterwards. Step 7 of the transcript, verbatim:

  new PersistentVolume directory: /var/lib/rancher/k3s/storage/pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787_rbdr-shop_rbdr-orders
$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

Name the archive from the recovery point you selected, and list it before you extract it. The listing is the last cheap check you get:

SNAP=/var/backups/rbdr/rbdr-pv-backup-2026-08-28T02-00Z.tar
tar tf "$SNAP" | head

Data-loss riskextract into the directory behind the new claim— EXECUTION VALIDATED on k3s v1.36.3+k3s1. The command form is step 7 of docs/courses/backup-dr/execution-evidence/kubernetes-desired-state-vs-application-data.txt; no output was captured for it, so none is shown.

$ tar xf "$SNAP" -C "$NEWDIR"

Business validation

Running is a statement about the kubelet. The validation is a comparison against a value recorded at backup time.

Step 7 of the transcript closes with the comparison, verbatim. The recovered sum and the sum recorded before the loss are both present, and both are the same:

  ORDER-1001,4500.00
  ORDER-1002,1250.00
  recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
  original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
  RECOVERED - the application data is back, byte-identical

The comparison has to refuse two empty values. A missing file and a missing checksum record both yield an empty string, and an equality test between two empty strings passes:

RECOVERED=$(kubectl -n rbdr-shop exec rbdr-orders-writer -- md5sum /data/orders.csv | awk '{print $1}')
ORIGINAL=$(cat /var/backups/rbdr/orders.csv.md5)
if [ -z "$RECOVERED" ] || [ -z "$ORIGINAL" ]; then
  echo "REFUSED: one side is empty; this is not a match"
  exit 2
elif [ "$RECOVERED" = "$ORIGINAL" ]; then
  echo "VALIDATED: $RECOVERED"
else
  echo "MISMATCH: recovered=$RECOVERED original=$ORIGINAL"
  exit 1
fi

Then read one order through the application, by the route a user takes, and reconcile its value with someone who recognises it.

Abort criteria

Stop, and escalate, rather than continuing when:

the namespace has not left Terminating within the timeout;
tar tf lists paths that are absolute, or rooted at an unexpected directory;
the provisioner directory for the current volumeName does not exist;
no checksum was recorded at backup time, so the validation has no reference;
the newest archive predates a schema migration the application cannot read;
the volume holds writes made after the incident that nobody has agreed to discard.

What to record

The manifest revision, the archive and its timestamp, the old and new volume identifiers, both checksums, the discarded window, and the elapsed time split between object rebuild and data restore. In the captured sequence the objects came back in six seconds. The data took a separate archive, a separate decision, and a separate proof.