Lab 25: Restore a cluster from an etcd snapshot

Objective

By the end of this lab you will have deleted a namespace on a running cluster, restored the whole cluster from a snapshot taken minutes earlier, and be able to state precisely which objects came back, which objects the restore itself destroyed, and how wide the data-loss window was. The command is the easy part. What this lab is for is the fact that a restore is a choice between two kinds of loss, and the only way to develop a feel for that trade is to watch both kinds happen to objects you created yourself.

Architecture

One disposable control-plane node running a single-member etcd. A single member is enough to exercise every step of the procedure: the tools, the flags, the shutdown order, the validation gate and the data-loss window are identical. What a single member does not exercise is the per-node identity problem, which the “Production notes” section maps out at the end.

flowchart LR
    OP[Operator on cp-1] --> KB[kubectl]
    KB --> API[kube-apiserver static Pod]
    API --> E[etcd static Pod, one member]
    E --> DD["/var/lib/etcd on the host"]
    KL[kubelet] --> API
    KL --> WL[restore-demo and restore-later Pods]
    SNAP[etcdctl snapshot save] --> FILE["/var/backups/etcd/*.db"]

Requirements

• One disposable virtual machine: 2 vCPU, 4 GiB RAM, 40 GiB disk. Nested virtualisation is fine; this lab never nests further.
• At least 6 GiB free on the filesystem holding /var. The restore writes a second copy of the data directory alongside the first, and the original is kept for the rollback path.
• A single-control-plane kubeadm cluster at v1.34.x, already built and healthy, with a CNI installed and CoreDNS Ready. Lab 1 builds it; the short form is below if you need a fresh one.
• sudo on the node and a shell that survives a control-plane outage. This lab never touches networking, the firewall or sshd, so no out-of-band console is required.
• Outbound network access to fetch the etcd release tarball from GitHub and to pull registry.k8s.io/pause.

If you do not already have a cluster, this is the smallest one that works. Substitute your own node address; 192.0.2.10 is a documentation address that will not route.

# Only if you do not already have a disposable cluster.
NODE_IP=192.0.2.10

sudo kubeadm init \
  --apiserver-advertise-address="$NODE_IP" \
  --pod-network-cidr=10.244.0.0/16 \
  --kubernetes-version=v1.34.0

mkdir -p "$HOME/.kube"
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

Then install the CNI you used in Lab 1, with a configuration whose Pod CIDR matches --pod-network-cidr. The lab needs Pod networking only so far as CoreDNS reaches Ready; it does not depend on any particular plugin. The control-plane taint stays in place — the lab’s workloads tolerate it explicitly rather than mutating the node.

Scenario

Your team runs a cleanup script that removes finished sandbox namespaces. This afternoon somebody edited the label selector, ran it, and deleted restore-demo — a namespace that was still in use. The objects are gone, their Pods have been terminated, and the ConfigMap that the namespace carried was created in-cluster rather than from Git, so re-applying the repository does not bring it back.

There is an hourly etcd snapshot. It is twenty minutes old. Restoring it returns restore-demo exactly as it was — and rewinds every other write the cluster has taken in those twenty minutes. In the lab you will create objects on both sides of the snapshot so that the second half of that sentence is not an abstraction.

Tasks

Task 1 — Put the offline tools on the node

etcdctl is the network client and speaks to a running member. etcdutl is the offline utility and operates on data files. etcd v3.6.0 removed etcdctl snapshot restore and etcdctl snapshot status, so on a 1.34 cluster the restore is an etcdutl command with no alternative.

etcdutl is also not in the image kubeadm runs: registry.k8s.io/etcd ships etcd and etcdctl only, so there is nothing to crictl exec into. Read the image tag first and match the tarball to it.

$ sudo grep -- 'image:' /etc/kubernetes/manifests/etcd.yaml

    image: registry.k8s.io/etcd:3.6.5-0

# Match this to the tag printed above: leading v, and drop the -0 suffix.
ETCD_VER=v3.6.5

curl -fsSL "https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz" \
  -o /tmp/etcd.tar.gz

sudo tar xzf /tmp/etcd.tar.gz -C /usr/local/bin --strip-components=1 --no-same-owner \
  "etcd-$ETCD_VER-linux-amd64/etcdctl" "etcd-$ETCD_VER-linux-amd64/etcdutl"

etcdctl version
etcdutl version
etcdutl snapshot restore --help | grep -E 'bump-revision|mark-compacted'

That last line is a version probe, not decoration. If it prints nothing, your etcdutl predates 3.6; omit both flags in Task 9 and read the callout there for what you give up.

Now set the client environment once. Every etcdctl call in this lab reuses it, which is why the later commands look short.

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

sudo -E etcdctl endpoint health -w table
sudo -E etcdctl member list -w table

If etcdctl answers unauthenticated, swap ETCDCTL_CERT and ETCDCTL_KEY for the healthcheck-client pair in the same directory; kubeadm issues it from the same CA specifically for client calls.

Record the member’s own identity now. Task 9 needs it, and by then the API server will be gone and this manifest will be the only copy.

sudo grep -E -- '--(name|initial-advertise-peer-urls|initial-cluster|data-dir)=' \
  /etc/kubernetes/manifests/etcd.yaml

Write the three values into your notes: the member name, its peer URL, and the data directory. This lab’s examples use cp-1, https://192.0.2.10:2380 and /var/lib/etcd.

Task 2 — Create the state that must survive

Two namespaces, created twenty minutes apart in the story and about five in the lab. This one goes in before the snapshot.

mkdir -p "$HOME/etcd-restore-lab/evidence"
cd "$HOME/etcd-restore-lab"

# ~/etcd-restore-lab/before-snapshot.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: restore-demo
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: recovery-point
  namespace: restore-demo
data:
  created: before-snapshot
  note: This object is inside the snapshot and must come back.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keepalive
  namespace: restore-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: keepalive
  template:
    metadata:
      labels:
        app: keepalive
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10.1

kubectl apply -f before-snapshot.yaml
kubectl -n restore-demo wait --for=condition=Available deploy/keepalive --timeout=180s
kubectl -n restore-demo get cm,deploy,pod

The toleration is what lets these Pods schedule onto a single-node cluster without removing the control-plane taint. Leaving the taint alone keeps the node in the state kubeadm left it, which matters when you compare the restored cluster against the original.

Task 3 — Take the snapshot, and verify it before you trust it

snapshot save reads the member’s bbolt file from local disk. On a single-member lab cluster that read competes with the same member’s WAL fsync path, which is exactly the effect Part LXVIII tells you to avoid in production by pointing the command at a follower. Here there is no follower, and the cluster is idle, so the cost is invisible — note that the reason it is invisible is the lab, not the command.

sudo mkdir -p /var/backups/etcd
SNAP=/var/backups/etcd/etcd-$(date -u +%Y%m%dT%H%M%SZ).db

sudo -E etcdctl snapshot save "$SNAP"
echo "$SNAP" | tee "$HOME/etcd-restore-lab/evidence/snapshot-path.txt"

Now verify it. An unverified snapshot is not a recovery point, it is a file — and the failure mode this catches is a snapshot that saves successfully and restores into an empty database.

SNAP=$(cat "$HOME/etcd-restore-lab/evidence/snapshot-path.txt")

etcdutl snapshot status "$SNAP" -w table | tee "$HOME/etcd-restore-lab/evidence/snapshot-status.txt"
sudo sha256sum "$SNAP" | tee "$HOME/etcd-restore-lab/evidence/snapshot-sha256.txt"

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| a1b2c3d4 |    41289 |       1204 |      6.6 MB |
+----------+----------+------------+------------+

Write the REVISION down. It is the single number that says how far back the restore will wind the cluster, and in Task 6 it is what turns “we will lose some state” into a number you can put in front of an approver. A zero or unparseable HASH means the file is corrupt: stop, take another snapshot, and do not proceed with this one.

Task 4 — Write the state that will not survive

This is the half of the exercise that the command reference leaves out.

# ~/etcd-restore-lab/after-snapshot.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: restore-later
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: written-later
  namespace: restore-later
data:
  created: after-snapshot
  note: This object is outside the snapshot and the restore will destroy it.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: written-later
  namespace: restore-later
spec:
  replicas: 1
  selector:
    matchLabels:
      app: written-later
  template:
    metadata:
      labels:
        app: written-later
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10.1

kubectl apply -f after-snapshot.yaml
kubectl -n restore-later wait --for=condition=Available deploy/written-later --timeout=180s

date -u +%Y-%m-%dT%H:%M:%SZ | tee evidence/window-opened.txt
kubectl get ns restore-demo restore-later

Both namespaces now exist. Only one of them exists in the snapshot.

Task 5 — The incident

Run the deletion. This is the failure the rest of the lab responds to.

kubectl delete namespace restore-demo --wait=true

kubectl get ns
kubectl -n restore-demo get cm recovery-point

The last command must fail. A deleted namespace takes its ConfigMap, its Deployment, its ReplicaSet and its Pods with it, and none of them are in any controller’s desired state any more. Nothing in the cluster is going to bring them back on its own.

Task 6 — Decide, and write the decision down

Part LXIX opens with a decision tree for a reason. Walk it here, on paper, before touching etcd. The cluster is still serving requests, so the first branch says “state is corrupted?” — and the honest answer is that the state is not corrupt, it is wrong: an object that should exist does not. That reaches the restore branch, but only just, and the alternatives deserve a hearing.

“Hold” is a first-class option and needs the same rigour as the other two: an owner, and an end time. Write it as “Alex holds until 16:30 while we check whether the namespace contents are reconstructable from Git; at 16:30 we restore or we accept the loss”. A hold without an end time is not a decision, it is a stall.

For the lab, choose the restore, and write down the three lines that make it a change rather than a reflex:

Decision:  restore from snapshot <path>, revision <n from Task 3>
Because:   restore-demo/recovery-point was created in-cluster and is not in Git
Cost:      every write since <snapshot timestamp> is discarded, including
           namespace restore-later and everything in it

Task 7 — Capture the state you are about to discard

The pre-restore capture is the recovery of last resort. It costs a minute. Without it, anything created after the snapshot is gone with no record that it ever existed — you will not even be able to list what you lost.

cd "$HOME/etcd-restore-lab"

kubectl get namespaces -o yaml > evidence/pre-restore-namespaces.yaml
kubectl get all,configmaps,secrets,serviceaccounts -A -o yaml > evidence/pre-restore-objects.yaml
kubectl get pods -A --no-headers | wc -l > evidence/pre-restore-pod-count.txt
kubectl get clusterrolebindings,rolebindings -A --no-headers | wc -l > evidence/pre-restore-rbac-count.txt
kubectl get ns -o name | sort > evidence/pre-restore-ns-list.txt

wc -l evidence/pre-restore-*.txt

Then snapshot the cluster as it is now, broken. This is your second way back: if the restore goes wrong, this file returns you to the state you are standing in rather than to nothing at all.

BROKEN=/var/backups/etcd/pre-restore-$(date -u +%Y%m%dT%H%M%SZ).db

sudo -E etcdctl snapshot save "$BROKEN"
etcdutl snapshot status "$BROKEN" -w table
echo "$BROKEN" | tee evidence/pre-restore-snapshot-path.txt

Task 8 — Stop the control plane

kubeadm runs the control plane as static Pods, so there is no systemd unit to stop and crictl stop on its own achieves nothing: the kubelet restarts anything whose manifest is still in the watched directory. Moving the manifests out is what stops them.

The API server goes first because it is the only component that writes to etcd. The controller manager and scheduler write through it, so once it is down they cannot reach etcd anyway — moving their manifests is a confirmation rather than a fix. etcd goes last.

sudo mkdir -p /etc/kubernetes/manifests.stopped

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml       /etc/kubernetes/manifests/kube-controller-manager.yaml       /etc/kubernetes/manifests/kube-scheduler.yaml       /etc/kubernetes/manifests/etcd.yaml       /etc/kubernetes/manifests.stopped/

# The kubelet notices and removes the Pods within about 30 seconds.
sleep 45
sudo crictl ps --state running | grep -E 'kube-apiserver|kube-scheduler|kube-controller-manager|etcd' && echo 'STILL RUNNING - do not continue' || echo 'control plane stopped'

Do not continue until you see control plane stopped. kubectl is dead from this moment; that is expected, and on a single-node cluster there is no second API server to fall back on.

Now look at what is still running.

sudo crictl ps --state running | grep -E 'pause|coredns'

The restore-later Pod is still there. Its container never stopped: the kubelet keeps running the Pods it already knows about, buffers the status updates it cannot deliver, and retries. The data plane survives a control-plane outage. Remember this Pod — Task 10 is where it dies.

Task 9 — Restore

Move the live data directory aside. Do not delete it: it is the rollback path and the only evidence of what the cluster held before the restore.

TS=$(date -u +%Y%m%dT%H%M%SZ)

sudo mv /var/lib/etcd "/var/lib/etcd.broken-$TS"
sudo ls -ld /var/lib/etcd.broken-*
sudo du -sh /var/lib/etcd.broken-*
echo "/var/lib/etcd.broken-$TS" | tee "$HOME/etcd-restore-lab/evidence/preserved-datadir.txt"

Now the restore itself. It writes a fresh data directory; it does not touch the running cluster, because there is no longer one.

SNAP=$(cat "$HOME/etcd-restore-lab/evidence/snapshot-path.txt")

# Substitute the three values you recorded in Task 1.
MEMBER=cp-1
PEER_URL=https://192.0.2.10:2380

sudo etcdutl snapshot restore "$SNAP" --name "$MEMBER" --initial-advertise-peer-urls "$PEER_URL" --initial-cluster "$MEMBER=$PEER_URL" --initial-cluster-token "k8s-restore-$(date -u +%Y%m%d)" --bump-revision 1000000000 --mark-compacted --data-dir /var/lib/etcd-restore

sudo ls -l /var/lib/etcd-restore/member/
sudo stat -c '%U:%G %a %n' /var/lib/etcd-restore

Move the restored directory into the path the static Pod mounts. The manifest is untouched by this route, which is one fewer thing to get wrong; the kubernetes-rb-restore-etcd runbook instead repoints the etcd-data hostPath and leaves the restored directory where it is. Both reach the same state. Pick one and use it consistently.

sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo ls -l /var/lib/etcd/member/

Task 10 — Start etcd first, then everything else

Return etcd.yaml alone. The member must be healthy before an API server is allowed to write to it.

sudo mv /etc/kubernetes/manifests.stopped/etcd.yaml /etc/kubernetes/manifests/

sleep 60
sudo crictl ps --name etcd
sudo crictl logs "$(sudo crictl ps -q --name etcd)" 2>&1 | tail -30

Expect the log to show the member publishing its peer URL and electing itself leader of a one-member cluster. Then check it through the API, which is etcdctl territory again — these are live calls, not data-file operations.

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

sudo -E etcdctl member list -w table
sudo -E etcdctl endpoint health -w table
sudo -E etcdctl endpoint status -w table
sudo -E etcdctl alarm list

One member, healthy, leader true, and no alarms. A NOSPACE alarm here means the restored database is already over its quota and the API server will fail its first write — deal with that before going on.

sudo mv /etc/kubernetes/manifests.stopped/kube-apiserver.yaml       /etc/kubernetes/manifests.stopped/kube-controller-manager.yaml       /etc/kubernetes/manifests.stopped/kube-scheduler.yaml       /etc/kubernetes/manifests/

sleep 60
sudo rmdir /etc/kubernetes/manifests.stopped

rmdir refuses to remove a directory that is not empty, which is a free check that no manifest was left behind.

Validation

These commands prove the outcome rather than restate it. Run them in order and record what each one returns.

The API server is serving the restored state:

kubectl get --raw '/readyz?verbose'
kubectl get nodes
kubectl -n kube-system get pods -l k8s-app=kube-dns

The node may report NotReady for a minute: the Lease object in the snapshot is stale, and the node is Ready again once the kubelet renews it. If it is still NotReady after two minutes, that is a real problem — see Troubleshooting.

The objects inside the snapshot came back:

kubectl get ns restore-demo
kubectl -n restore-demo get cm recovery-point -o jsonpath='{.data.created}'
kubectl -n restore-demo get deploy keepalive
kubectl -n restore-demo get pods

recovery-point must print before-snapshot. The keepalive Deployment must reach 2/2 — note that the Pods are new: the ReplicaSet controller recreated them from the restored desired state, so their names and UIDs differ from the ones you saw in Task 2. The restore returned the objects, not the running containers.

The objects outside the snapshot are gone. This is the data-loss window, and it is the check most restore write-ups omit:

$ kubectl get ns restore-later

Error from server (NotFound): namespaces "restore-later" not found

Now find the orphan. The written-later container has been running throughout, because the kubelet never stopped it — but its Pod object no longer exists in the API server the kubelet is now talking to.

sudo crictl pods --state Ready | grep written-later
sleep 60
sudo crictl pods | grep written-later

Counts, compared against the capture from Task 7:

cd "$HOME/etcd-restore-lab"

kubectl get ns -o name | sort > evidence/post-restore-ns-list.txt
diff evidence/pre-restore-ns-list.txt evidence/post-restore-ns-list.txt

kubectl get clusterrolebindings,rolebindings -A --no-headers | wc -l
cat evidence/pre-restore-rbac-count.txt

The namespace diff should show exactly one difference: restore-later present before, absent after. restore-demo is absent from both lists, because Task 7 captured the cluster after the deletion — which is a useful reminder that the pre-restore capture records the broken state, not the good one. The RBAC counts should match; RBAC predates the snapshot, so the restore returns it intact.

Finally, prove the cluster can still take a write:

kubectl run restore-check --image=registry.k8s.io/pause:3.10.1 --restart=Never \
  --overrides='{"spec":{"tolerations":[{"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}]}}'
kubectl wait --for=condition=Ready pod/restore-check --timeout=120s
kubectl delete pod restore-check

Scheduling, admission, the kubelet and the CNI all have to work for that to pass. It is the cheapest end-to-end check the cluster offers.

Expected Outcome

• etcdctl endpoint health reports the single member healthy, with a leader and no alarms.
• kubectl get --raw '/readyz?verbose' reports every check ok and the node is Ready.
• restore-demo exists with recovery-point reading before-snapshot and keepalive at 2/2, on newly created Pods.
• restore-later does not exist, and its container has been reaped by the kubelet.
• /var/lib/etcd.broken-<timestamp> still exists on the host, and you can say what is in it and when you intend to delete it.
• You can state the data-loss window in minutes and name every object inside it.

Troubleshooting

Rollback

The restore stays reversible for as long as the preserved data directory exists. Backing out returns the cluster to the state at the end of Task 7 — which is a cluster with restore-demo still deleted. That is the point worth internalising: rollback returns you to the problem, not to a working cluster.

PRESERVED=$(cat "$HOME/etcd-restore-lab/evidence/preserved-datadir.txt")

sudo mkdir -p /etc/kubernetes/manifests.stopped
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml       /etc/kubernetes/manifests/kube-controller-manager.yaml       /etc/kubernetes/manifests/kube-scheduler.yaml       /etc/kubernetes/manifests/etcd.yaml       /etc/kubernetes/manifests.stopped/
sleep 45

sudo mv /var/lib/etcd /var/lib/etcd.restored
sudo mv "$PRESERVED" /var/lib/etcd

sudo mv /etc/kubernetes/manifests.stopped/etcd.yaml /etc/kubernetes/manifests/
sleep 60
sudo -E etcdctl endpoint health -w table

sudo mv /etc/kubernetes/manifests.stopped/kube-apiserver.yaml       /etc/kubernetes/manifests.stopped/kube-controller-manager.yaml       /etc/kubernetes/manifests.stopped/kube-scheduler.yaml       /etc/kubernetes/manifests/
sleep 60
kubectl get ns

If that succeeds you will see restore-later back and restore-demo gone — the exact state you decided to leave. The third path, if both the restore and the rollback have failed, is the pre-restore snapshot from Task 7: restore it with the same command as Task 9.

Cleanup

Run this once you have finished comparing. It removes what the lab created and leaves the cluster working.

kubectl delete namespace restore-demo restore-later --ignore-not-found

sudo rm -rf /var/lib/etcd.broken-* /var/lib/etcd.restored
sudo rm -f /var/backups/etcd/*.db /tmp/etcd.tar.gz
rm -rf "$HOME/etcd-restore-lab"

kubectl get ns
sudo ls -ld /var/lib/etcd
kubectl get --raw '/readyz?verbose'

Leave etcdctl and etcdutl in /usr/local/bin; both are worth having on a control-plane node, and the version probe in Task 1 is how you check them against the cluster next time. If the cluster itself was built only for this lab, sudo kubeadm reset -f removes it.

Production notes

Map this exercise onto a real change window before you rely on it.

Three members, not one. The single biggest gap. On a three-node control plane every node runs its own etcdutl snapshot restore, from a byte-identical copy of the same snapshot, with its own --name and --initial-advertise-peer-urls and the same --initial-cluster and --initial-cluster-token. The restore does not coordinate across nodes; the cluster forms afterwards, when each static Pod starts and the members find each other. Restoring all three with the same --name is the classic failure: each node believes it is the only member it knows about and the cluster never forms. kubernetes-rb-restore-etcd is the three-node procedure.

The outage is real and it is the whole procedure. From the manifest move to the API server’s return, nothing can be scheduled, rescheduled or scaled. Existing workloads keep serving, as Task 8 showed, but a Pod that crashes during the window stays down. Communicate that before you start.

The data-loss window needs written acceptance. Task 3’s REVISION and timestamp, plus the current time, define exactly what is being discarded. An approver who has seen those two numbers is agreeing to something specific. One who has been told “we might lose a bit of recent state” is not.

Pre-flight the things this lab could not break. Part LXIX’s checklist exists because certificate expiry, peer URLs that no longer resolve and snapshots taken from a different cluster are all invisible until the restore is halfway through. A snapshot that pre-dates a credential rotation restores cleanly and then cannot authenticate anything.

Rehearse on a disposable cluster, on a schedule. This lab is that rehearsal. A restore procedure that has been read but never executed is a document, not a capability — and the first execution should not be the one with an incident channel watching.

What you learned

• A snapshot is a recovery point only after it has been verified. etcdutl snapshot status and a recorded sha256 are what turn a file into something you can plan around; the REVISION is the number that makes the data-loss window concrete.
• A restore is a choice between two losses. You recovered restore-demo by destroying restore-later. Every restore does this; most write-ups only describe the half that recovers.
• The membership flags belong on the restore command. etcd reads the bootstrap flags only when the data directory is empty, so the static Pod manifest cannot correct a restore that omitted them.
• Stop the writers before you touch the data. The API server is the only component that writes to etcd, and moving its manifest out is what stops it under kubeadm.
• The data plane outlives the control plane. Containers kept running through the outage, and the orphaned one was reaped only when the kubelet got an authoritative Pod list again.
• Moving the data directory aside, rather than deleting it, is what makes the restore reversible — and reversible only back to the problem you started with, which is why the decision in Task 6 mattered more than any command that followed it.