Pre-flight checks — what to verify before running the restore

The pre-flight is the set of checks that confirm the restore will succeed before the operator commits to it. A restore that begins without pre-flighting risks discover during the restore that a cert has expired, a peer URL is wrong, or the snapshot is for a different cluster. This lesson walks each check.

The pre-flight checklist

flowchart TB
    PF[Pre-flight] --> CHK1[Snapshot integrity]
    CHK1 --> CHK2[Peer URL mapping]
    CHK2 --> CHK3[Certificate mapping]
    CHK3 --> CHK4[Network reachability]
    CHK4 --> CHK5[Host readiness]
    CHK5 --> READY[Restore ready]

The five checks:

A pre-flight runs in ~15 minutes and saves hours of restore-time debugging.

Check 1 — Snapshot integrity

# 1.1 Size and existence
ls -la /backup/etcd-snapshot.db
# expected: size matches the cluster's DB size approximately

# 1.2 Hash matches upload verification
sha256sum /backup/etcd-snapshot.db
# expected: hash matches the one stored in the manifest

# 1.3 Status
etcdutl snapshot status /backup/etcd-snapshot.db --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

The expected output:

• HASH is non-zero.
• REVISION is at or near the cluster’s last-known good revision.
• TOTAL KEY is close to the cluster’s known object count.
• TOTAL SIZE is consistent with the cluster’s bbolt size.

Check 2 — Peer URL mapping

The restore uses --initial-cluster=<NAME1>=<URL1>,<NAME2>=<URL2>,<NAME3>=<URL3>. Each peer URL must match the host where that member will run.

# On each host that will be a member:
hostname
ip addr show | grep -E 'inet '
# Output: the host's IP that matches the URL in --initial-cluster

# Example expected mapping:
# cp-1 -> 10.0.1.10
# cp-2 -> 10.0.1.11
# cp-3 -> 10.0.1.12

The mapping is often documented in the kubeadm cluster-info or in the original kubeadm init output. If not, it is recoverable from the static-pod manifest:

sudo cat /etc/kubernetes/manifests/etcd.yaml | grep -E 'initial-cluster|listen-peer-urls'

- --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380
- --listen-peer-urls=https://10.0.1.10:2380
- --initial-advertise-peer-urls=https://10.0.1.10:2380

The mapping must be preserved for the restored cluster. A wrong peer URL means the new member cannot reach its peers, and the cluster cannot form.

Check 3 — Certificate mapping

Each etcd member uses:

• A server cert (the API server connection).
• A peer cert (the member-to-member connection).
• A CA cert (the chain).

For a kubeadm-managed cluster, these live in /etc/kubernetes/pki/etcd/:

ls -la /etc/kubernetes/pki/etcd/

-rw-r--r-- 1 root root 1090 Aug  1 10:00 ca.crt
-rw------- 1 root root 2867 Aug  1 10:00 ca.key
-rw-r--r-- 1 root root 1489 Aug  1 10:00 peer.crt
-rw------- 1 root root 1675 Aug  1 10:00 peer.key
-rw------- 1 root root 1389 Aug  1 10:00 server.crt
-rw------- 1 root root 1679 Aug  1 10:00 server.key

The certs are per-host. A kubeadm cluster has each control-plane host with its own set of peer.crt, server.crt, etc. For a restore, the certs are not regenerated (the restore preserves them; the cluster is restored, but the certs are unchanged).

# Verify certs are valid (not expired)
sudo openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates

notBefore=Aug  1 10:00:00 2026 GMT
notAfter=Aug  1 10:00:00 2027 GMT

Expired certs break the restore; the API server cannot connect to etcd. If a cert is expired, renew it first before the restore.

Check 4 — Network reachability

The peer ports must be reachable between members:

# From cp-1, can I reach cp-2's peer port?
nc -zv cp-2 2380
# From cp-1, can I reach cp-3's peer port?
nc -zv cp-3 2380

A network partition or firewall rule that blocks peer traffic prevents cluster formation. The restore is not the right action if the network is broken; the network is.

# Also check the API server's etcd client port
nc -zv cp-1 2379
nc -zv cp-2 2379
nc -zv cp-3 2379

The 2379 port is for API server to etcd; 2380 is for etcd member-to-member.

Check 5 — Host readiness

Each host that will host a restored member needs:

• Disk space for the new data dir (the snapshot decompresses into /var/lib/etcd).
• Writable data dir (no leftover filesystem locks or mounts).
• Correct kubelet configuration for the etcd static pod.

# Disk space
df -h /var/lib/etcd
# Expected: at least 5 GiB available

# Writable
ls -ld /var/lib/etcd
# Expected: drwx------ 2 etcd etcd

# Inspect the kubelet's perspective
sudo crictl ps -a | grep etcd
# Expected: prior pod stopped; new pod will start after manifest update

If the data dir is on a separate volume, confirm the volume is mounted and writable:

mountpoint /var/lib/etcd
# Returns the device or "not a mountpoint" if not mounted

The data-dir move step

The pre-flight includes preparing the new data dir.

On each host:

# Container id from `sudo crictl ps --name etcd`:
ETCD_CONTAINER_ID=3f5c9a1b7e2d4

# 1. Stop etcd (if still running)
sudo crictl stop "$ETCD_CONTAINER_ID"

# 2. Move the existing data dir aside (do NOT delete yet)
sudo mv /var/lib/etcd /var/lib/etcd.broken-$(date +%Y%m%d-%H%M)

# 3. Confirm the moved dir is still readable
ls -la /var/lib/etcd.broken-*/

# 4. Confirm /var/lib/etcd is now an empty path
ls -la /var/lib/etcd
# Expected: No such file or directory

The *.broken-* directory is held for forensic analysis post-restore. After the cluster is verified, the operator deletes it.

Cross-host consistency

The pre-flight verifies that every host is configured consistently:

• Same etcd version on every host (etcd --version).
• Same --initial-cluster value in every host’s static-pod manifest.
• Same --initial-cluster-token (typically the cluster’s UUID, set at kubeadm init time).
• Same --data-dir path (typically /var/lib/etcd).

A drift between hosts is a restore-time surprise.

# On each host, check the static pod manifest
diff /etc/kubernetes/manifests/etcd.yaml <(ssh cp-2 cat /etc/kubernetes/manifests/etcd.yaml)
# expected: no output (drift detected if any)

The pre-flight output

After running all five checks, the operator should be able to confirm:

• The snapshot is valid and consistent.
• Each member’s peer URL maps to the host that will run it.
• Each member’s certs are valid and not expired.
• Network reachability between hosts is working.
• Each host is ready to host a new member.

A “yes” to each question means the restore is safe to begin. A “no” to any question is a blocker that must be resolved before the restore proceeds.

Quiz

Production discipline

• Pre-flight is mandatory. A restore without pre-flight is a restore expecting trouble.
• 15 minutes of pre-flight saves 3 hours of restore-time debugging.
• Move, do not delete. Forensic state matters; the broken data dir is held until post-restore verification.
• Renew expired certs before the restore. Cert validity is the sneakiest failure.
• Document the pre-flight. Each check recorded; each finding recorded; the restore then begins with a clean pre-flight report.

The pre-flight is the safety net behind the restore. A clean pre-flight produces a clean restore; a dirty pre-flight produces a dirty restore.