Recover Kubernetes application data onto a rebuilt namespace

Objective

A namespace is not one thing. It is a manifest in a repository, a set of objects in etcd, and a directory of bytes on a node. Three stores, three different backup mechanisms, and only the first one is what people mean when they say everything is in Git.

You will build all three, destroy the namespace, rebuild it from the same YAML, and watch a perfect reconstruction hand you an empty volume. Then you will restore the volume separately and prove the bytes came back.

Architecture

flowchart TB
    GIT["manifest bundle in Git<br/>Namespace, PVC, ConfigMap, Pod"]
    ETCD["objects in etcd on the node<br/>PVC binding, ConfigMap, Pod spec"]
    PV["PersistentVolume directory on the node<br/>pvc-XXXX_rbdr-shop_rbdr-orders/orders.csv"]
    GIT -->|kubectl apply| ETCD
    ETCD -->|local-path provisioner| PV
    GIT -.->|re-apply the repository| R1["every object returns<br/>PVC Bound, pod 1/1 Running"]
    ETCD -.->|k3s etcd-snapshot, 1208352 bytes| R2["API objects only"]
    PV -.->|tar on the node, 2560 bytes| R3["orders.csv, md5 9eb4e2ad...964b0"]
    R1 --> EMPTY["/data is empty, exit 1"]
    R3 --> BACK["/data holds the orders again"]

Follow the two dotted paths on the right. The repository and the etcd snapshot both terminate at objects. Only the node-level archive reaches the bytes.

Requirements

• A disposable single-node k3s cluster with the embedded etcd datastore, and root or sudo on the node. The capture quoted throughout ran on k3s v1.36.3+k3s1 (5aed4d7b) with etcd snapshots as the datastore. Without the etcd datastore, k3s etcd-snapshot has nothing to snapshot. Every output block below is quoted from that capture. The pre-state, validation and cleanup steps were added to the later complete run captured in docs/courses/backup-dr/execution-evidence/backup-dr-lab-20-kubernetes-data-onto-a-clean-cluster-2026-08-29.txt, which supports this page’s last_executed date.
• The capture environment relaxed the kubelet eviction thresholds to nodefs.available<2%, because the host filesystem it ran on was above 90% full and the default 10% threshold tainted the node NoSchedule:

--kubelet-arg=eviction-hard=nodefs.available<2%

That is a property of the capture environment, not a recommendation for production. Running a node to 98% full removes the headroom the kubelet exists to defend and is a straightforward way to wedge a cluster. On a normally provisioned node, leave the defaults alone.

• A cluster you can throw away. Task 5 deletes a namespace and Task 7 deletes it again.
• No rbdr- objects already present. Every object here carries that prefix so cleanup can be scoped and asserted; Task 1 records what was there first.

Scenario

A platform team runs their shop namespace from a GitOps repository. Every object is declared, reviewed and reconciled. They also run a nightly k3s etcd-snapshot, retained for fourteen days.

Somebody deletes the namespace. You are asked how long recovery takes. The honest answer depends on which of the three stores held the thing that mattered.

Tasks

Task 1 — Record pre-lab state

LAB="$HOME/rbdr-lab-20"
SNAPDIR=/var/lib/rancher/k3s/server/db/snapshots
PVROOT=/var/lib/rancher/k3s/storage
mkdir -p "$LAB"

{
  k3s --version | head -1
  echo "--- namespaces ---"
  kubectl get ns -o name | grep 'rbdr-' || echo "none"
  echo "--- persistentvolumes ---"
  kubectl get pv -o name | grep 'rbdr-' || echo "none"
  echo "--- snapshots ---"
  sudo ls -1 "$SNAPDIR" 2>/dev/null | grep 'rbdr-' || echo "none"
  echo "--- node taints ---"
  kubectl get node -o jsonpath='{.items[*].spec.taints}'; echo
} | tee "$LAB/pre-state.txt"

Cleanup compares against pre-state.txt, so the first three inventories must read none now. The taint line is the eviction check: if it names node.kubernetes.io/disk-pressure, the kubelet has already tainted the node and nothing you apply will schedule.

Task 2 — Apply the desired state

cat > "$LAB/rbdr-shop.yaml" <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: rbdr-shop
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbdr-orders
  namespace: rbdr-shop
spec:
  accessModes: ['ReadWriteOnce']
  storageClassName: local-path
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rbdr-config
  namespace: rbdr-shop
data:
  currency: 'GBP'
---
apiVersion: v1
kind: Pod
metadata:
  name: rbdr-orders-writer
  namespace: rbdr-shop
spec:
  containers:
    - name: writer
      image: alpine:3
      command: ['sh', '-c', 'sleep 3600']
      volumeMounts:
        - name: orders
          mountPath: /data
  volumes:
    - name: orders
      persistentVolumeClaim:
        claimName: rbdr-orders
EOF

apply_shop() {
  kubectl apply -f "$LAB/rbdr-shop.yaml" || true
  for _ in $(seq 1 30); do
    kubectl -n rbdr-shop get serviceaccount default >/dev/null 2>&1 \
      && kubectl apply -f "$LAB/rbdr-shop.yaml" \
      && return 0
    sleep 1
  done
  echo 'default ServiceAccount or workload did not become available' >&2
  return 1
}

apply_shop
kubectl -n rbdr-shop wait --for=condition=Ready pod/rbdr-orders-writer --timeout=120s

$ kubectl apply -f rbdr-shop.yaml

  namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 9s

Task 3 — Write data and find where it lives

kubectl -n rbdr-shop exec rbdr-orders-writer -- \
  sh -c 'printf "ORDER-1001,4500.00\nORDER-1002,1250.00\n" > /data/orders.csv'
kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv
kubectl -n rbdr-shop exec rbdr-orders-writer -- md5sum /data/orders.csv

PVNAME=$(kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.spec.volumeName}')
PVDIR="$PVROOT/${PVNAME}_rbdr-shop_rbdr-orders"
echo "$PVDIR" | tee "$LAB/pvdir-before.txt"
sudo ls -l "$PVDIR/orders.csv"

$ kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv

  ORDER-1001,4500.00
ORDER-1002,1250.00
orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0

--- where that data physically lives on the node ---
/var/lib/rancher/k3s/storage/pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf_rbdr-shop_rbdr-orders/orders.csv

The directory name embeds the PersistentVolume identifier, and that identifier changes every time the claim is recreated. Any runbook that hardcodes it points at a directory that stops existing.

Task 4 — Snapshot the cluster state

sudo k3s etcd-snapshot save --name rbdr-before
sudo ls -la "$SNAPDIR"

$ sudo k3s etcd-snapshot save --name rbdr-before

  time="2026-08-28T14:33:54Z" level=info msg="Snapshot rbdr-before-125e3e56d5eb-1787927634 saved."
>>> exit code: 0
snapshot file: rbdr-before-125e3e56d5eb-1787927634
drwx------ 4 root root    4096 Aug 28 14:32 ..
-rw------- 1 root root 1208352 Aug 28 14:33 rbdr-before-125e3e56d5eb-1787927634

That file records the namespace, the claim binding, the ConfigMap and the pod spec. An etcd snapshot is not a backup of application data; it holds the cluster’s record of what should exist and none of what the application wrote. On the etcd 3.7.1 build Lab 19 captured, snapshot status and snapshot restore are no longer subcommands of etcdctl either — they moved to etcdutl, which is a detail worth discovering before an incident rather than during one.

Task 5 — Delete the namespace, and wait

kubectl delete namespace rbdr-shop

until ! kubectl get namespace rbdr-shop >/dev/null 2>&1; do
  echo "still terminating"
  sleep 2
done
echo "namespace fully removed"

PVDIR=$(cat "$LAB/pvdir-before.txt")
sudo ls -la "$PVDIR" || echo "GONE - the provisioner reclaimed the volume with the PVC"

$ kubectl delete namespace rbdr-shop

  namespace "rbdr-shop" deleted
namespace fully removed after 0s

--- is the PersistentVolume data still on disk? ---
GONE - the local-path provisioner reclaimed the volume with the PVC

Task 6 — Rebuild from the repository

apply_shop
kubectl -n rbdr-shop wait --for=condition=Ready pod/rbdr-orders-writer --timeout=120s
kubectl -n rbdr-shop get pvc,configmap,pod

kubectl -n rbdr-shop exec rbdr-orders-writer -- ls -la /data
kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv
echo ">>> exit code: $?"

$ kubectl -n rbdr-shop get pvc,configmap,pod

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

This is the failing case, and it fails cleanly: the claim is Bound, the pod is 1/1 Running, the ConfigMap is present, the reconstruction finished with the pod Ready after 6s, and cat exits 1. Note the new PersistentVolume identifier — pvc-a29538af-... where Task 3 had pvc-6edd5db0-.... The rebuild leg did exactly what it was asked. The orders were never in the thing that was asked.

Task 7 — Take the backup that was missing, then destroy again

kubectl -n rbdr-shop exec rbdr-orders-writer -- \
  sh -c 'printf "ORDER-1001,4500.00\nORDER-1002,1250.00\n" > /data/orders.csv'

PVNAME=$(kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.spec.volumeName}')
PVDIR="$PVROOT/${PVNAME}_rbdr-shop_rbdr-orders"
echo "PersistentVolume directory on the node: $PVDIR"
sudo tar cf /tmp/rbdr-pv-backup.tar -C "$PVDIR" .
sudo ls -l /tmp/rbdr-pv-backup.tar

kubectl delete namespace rbdr-shop
until ! kubectl get namespace rbdr-shop >/dev/null 2>&1; do sleep 2; done
apply_shop
kubectl -n rbdr-shop wait --for=condition=Ready pod/rbdr-orders-writer --timeout=120s
kubectl -n rbdr-shop exec rbdr-orders-writer -- ls -la /data

$ sudo tar cf /tmp/rbdr-pv-backup.tar -C "$PVDIR" .

  PersistentVolume directory on the node: /var/lib/rancher/k3s/storage/pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93_rbdr-shop_rbdr-orders
-rw-r--r-- 1 root root 2560 Aug 28 14:34 /tmp/rbdr-pv-backup.tar

--- destroy the namespace and its volume again ---
namespace fully removed after 0s
pod Ready after 6s
namespace recreated from YAML; volume is empty:
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:35 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:35 ..

The archive is 2560 bytes. Every recovery point this namespace has ever had was in a file that small, and no controller was going to produce it.

Task 8 — Restore the volume and time it

RESTORE_START=$(date +%s)
NEWPV=$(kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.spec.volumeName}')
NEWDIR="$PVROOT/${NEWPV}_rbdr-shop_rbdr-orders"
echo "new PersistentVolume directory: $NEWDIR"
sudo tar xf /tmp/rbdr-pv-backup.tar -C "$NEWDIR"

{
  kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv
  echo "recovered md5 : $(kubectl -n rbdr-shop exec rbdr-orders-writer -- md5sum /data/orders.csv | cut -d' ' -f1)"
  echo "original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0"
  echo "restore seconds: $(( $(date +%s) - RESTORE_START ))"
} | tee "$LAB/restore-proof.txt"

$ sudo tar xf /tmp/rbdr-pv-backup.tar -C "$NEWDIR"

  new PersistentVolume directory: /var/lib/rancher/k3s/storage/pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787_rbdr-shop_rbdr-orders

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Three PersistentVolume identifiers across one lab, and the archive went into the third. The tar was taken from a directory that no longer exists, and unpacked into one that did not exist when it was taken.

The capture above did not time itself, so the restore seconds line the block writes into restore-proof.txt is absent from it. That number is yours to measure, and it is the one the Expected Outcome asks for.

Validation

REC=$(kubectl -n rbdr-shop exec rbdr-orders-writer -- md5sum /data/orders.csv 2>/dev/null | cut -d' ' -f1)
ORIG=9eb4e2ad8e08e1dcaaf87ababab964b0
if [ -n "$REC" ] && [ "$REC" = "$ORIG" ]; then
  echo "PASS both checksums present and equal"
else
  echo "FAIL rec=[$REC] orig=[$ORIG]"
fi

kubectl -n rbdr-shop get pvc rbdr-orders -o jsonpath='{.status.phase}'; echo
kubectl -n rbdr-shop get pod rbdr-orders-writer -o jsonpath='{.status.phase}'; echo
grep -c 'restore seconds' "$LAB/restore-proof.txt"
grep -c 'none' "$LAB/pre-state.txt"

The -n "$REC" half is load-bearing. If the exec fails — wrong namespace, pod not ready, file absent — REC is the empty string, and an equality test against an empty ORIG would report success. Two empty strings compare equal, and a check that can pass while measuring nothing is worse than no check. -n makes an unmeasured restore fail.

Expected Outcome

The namespace was rebuilt twice from the same four-object manifest. Both times every object returned, the claim bound and the pod reached Ready, and both times the volume arrived empty. The orders came back only from a 2560-byte archive taken separately, at the node level, and unpacked into a directory whose name did not exist when the archive was made.

Record both numbers from your own run:

• Actual restore time: _______ seconds, from restore-proof.txt, measured in Task 8 from the tar xf to the checksum comparison. It excludes the rebuild that had to happen first, and the time to notice.
• Actual RPO observed: _______ seconds, the interval between the tar cf in Task 7 and the kubectl delete namespace that followed it. Every write inside that window is not in the archive. That is a property of when the archive was taken, never of the tool.

Troubleshooting

cat: can't open '/data/orders.csv': No such file or directory, exit 1, in Task 6. Expected. The claim was recreated, so the provisioner made a new empty directory. Continue to Task 7.

kubectl apply reports objects unchanged and the Pod stays Pending. You applied while the namespace was still Terminating. Delete again and let the until loop in Task 5 finish first.

Every Pod stays Pending and the node carries node.kubernetes.io/disk-pressure:NoSchedule. The kubelet crossed its eviction threshold and tainted the node. Free disk on the node; do not reach for the threshold override outside a disposable capture host.

k3s etcd-snapshot save exits non-zero and no file appears in $SNAPDIR. The cluster was installed without the embedded etcd datastore, so there is no etcd for the subcommand to snapshot. This is a property of how the cluster was installed and cannot be fixed from the snapshot command; see the k3s backup and restore reference for the datastores it applies to.

The PVC stays Pending and no volume directory appears. The storageClassName: local-path does not resolve on this cluster. kubectl get storageclass and use the name that exists.

tar xf succeeds and /data is still empty. The archive went into the previous PersistentVolume directory. Re-read .spec.volumeName after the rebuild; the identifier changed.

Recovered md5 differs from 9eb4e2ad8e08e1dcaaf87ababab964b0. The printf in Task 3 was retyped and lost or gained a trailing newline. Rewrite it exactly as given.

Cleanup

LAB="$HOME/rbdr-lab-20"
SNAPDIR=/var/lib/rancher/k3s/server/db/snapshots

kubectl delete namespace rbdr-shop --ignore-not-found
until ! kubectl get namespace rbdr-shop >/dev/null 2>&1; do sleep 2; done
sudo rm -f /tmp/rbdr-pv-backup.tar
sudo find "$SNAPDIR" -name 'rbdr-before-*' -delete

{
  k3s --version | head -1
  echo "--- namespaces ---"
  kubectl get ns -o name | grep 'rbdr-' || echo "none"
  echo "--- persistentvolumes ---"
  kubectl get pv -o name | grep 'rbdr-' || echo "none"
  echo "--- snapshots ---"
  sudo ls -1 "$SNAPDIR" 2>/dev/null | grep 'rbdr-' || echo "none"
  echo "--- node taints ---"
  kubectl get node -o jsonpath='{.items[*].spec.taints}'; echo
} > "$LAB/post-state.txt"

diff "$LAB/pre-state.txt" "$LAB/post-state.txt" && echo "CLEAN: post-state matches pre-state"

diff exiting 0 with CLEAN printed is the assertion. Anything else names the objects still present.

Production notes

• Back up the volumes, not only the cluster. A cluster-state snapshot and a GitOps repository both describe objects; neither holds a byte of application data.
• Never hardcode a PersistentVolume path in a restore procedure. This lab produced three different identifiers for one claim, in one sitting.
• Restore into the volume the rebuilt claim actually bound to, read from .spec.volumeName at restore time.
• Wait for Terminating to finish before re-applying, in automation as well as by hand. An apply that reports unchanged is not an apply.
• Write the checksum guard so it cannot pass on an empty measurement, and record the restore duration in the same file as the proof.

What You Learned

• A complete object rebuild returned an empty volume, with the claim Bound, the pod 1/1 Running, and cat exiting 1.
• The etcd snapshot was 1208352 bytes of API objects and protected none of the application data.
• The PersistentVolume identifier changed on every recreation, so the restore target had to be read back, not remembered.
• A 2560-byte node-level tar returned the orders byte-identical, md5 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides of two destructions.
• A checksum comparison alone can pass on nothing. The -n test is what stops two empty strings reporting success.