Validating the restored cluster — production readiness checks

A restored cluster that “looks up” is not yet production- ready. The validation gate is a checklist that proves the cluster serves requests correctly, its workloads reconcile to a stable state, and the operational discipline (encryption, backup, monitoring) is re-armed. This lesson walks the validation gate and the production-readiness criteria.

The validation gate

flowchart LR
    A[etcd is healthy] --> B[API server is healthy]
    B --> C[Namespaces visible]
    C --> D[Workloads reconcile]
    D --> E[Secrets accessible]
    E --> F[Storage and networking]
    F --> G[Operational discipline]
    G --> READY[Production ready]

Each step produces a clear answer; a “no” at any step blocks production traffic.

Step 1 — API server health

# From a control-plane host, before kubectl is configured:
curl -k https://10.0.1.10:6443/livez
# Expected: ok

curl -k https://10.0.1.10:6443/readyz
# Expected: ok

# With kubectl context (after admin.conf is restored):
kubectl get ns
# Expected: list of namespaces

kubectl get componentstatuses
# (Deprecated but still useful)
# Expected: Healthy for all components

The API server responds; authentication works; namespaces are returned. If any of these fail, the API server is not serving the etcd-snapshotted state — investigate before proceeding.

Step 2 — Object counts match

The cluster’s expected object count should match the snapshot’s:

# Total objects
kubectl get all,cm,secret,sa,role,rolebinding,\
clusterrole,clusterrolebinding,crd,cr,leases,events -A \
  --no-headers | wc -l

# Compare to the snapshot's reported totalKey
# (from etcdutl snapshot status before restore)

The pre-restore capture (saved at the previous lesson) provides the expected counts. A close match indicates the cluster’s state is the snapshot’s state.

# Namespaces
kubectl get namespaces
# Expected: same names as before the restore

# Pods
kubectl get pods -A --no-headers | wc -l
# Expected: same count approximately (some pods may have terminated)

Step 3 — Workloads reconcile

A production cluster has many workloads; their Pods run on worker nodes. After restore, the cluster’s schedulers and controllers watch the API and reconcile.

# All Deployments
kubectl get deployments -A

# All StatefulSets
kubectl get statefulsets -A

# All DaemonSets
kubectl get daemonsets -A

Each workload should converge to its desired replicas.

# Wait for reconciliation
kubectl get deployments -A -o json | \
  jq '.items[] | select(.status.replicas != .status.availableReplicas) | {name: .metadata.name, namespace: .metadata.namespace}'

# Output should be empty (all deployments have replicas == available)

flowchart LR
    D[Deployment - desired: 5] -->|reconcile| P[Pods: 5]
    P -->|Ready| ALL[All Pods Ready]
    ALL -->|No?| S[Investigate: scheduler, kubelet, image, resources]

A 5-minute observation is the typical wait; longer indicates a problem in the cluster (image pull, networking, scheduler).

Step 4 — Secrets accessibility

For each high-value Secret:

# Confirm a test secret is readable (assuming proper RBAC)
kubectl get secret -n prod db-credentials -o jsonpath='{.data.password}' | base64 -d

# Run a workload that uses the secret; check the workload can read it

For an automated check, run a workload that mounts the Secret and reads it; if the workload succeeds in authenticating against the database, the Secret is intact.

Step 5 — RBAC sanity

# Check role bindings
kubectl get clusterrolebindings | wc -l
kubectl get rolebindings -A | wc -l

# Compare to pre-restore captures

If the RBAC count is significantly different, RBAC was lost. The fallback is to re-apply RBAC from a Git source (GitOps) or from the cluster’s bind logs.

Step 6 — Storage and networking

# PVCs
kubectl get pvc -A

# StorageClasses (cluster-scoped)
kubectl get storageclass

# Services
kubectl get svc -A

# EndpointSlices
kubectl get endpointslices -A

For each PVC, check that the PV is bound:

kubectl get pv | grep -E 'Bound|Released'
# Expected: most PVs are Bound

A common post-restore issue: PVCs that were pending before the restore are still pending. The PVC’s storage class and node affinity are restored from the snapshot; if the storage backend is unavailable, the PVC stays Pending.

Step 7 — Networking

# DNS resolution from a Pod
kubectl run -n default nettest --rm -it --image=busybox -- \
  nslookup kubernetes.default.svc.cluster.local

# Service reachability
kubectl run -n default nettest2 --rm -it --image=busybox -- \
  wget -O - http://kubernetes.default/api

These tests confirm CoreDNS, kube-proxy, and the CNI are all serving correctly.

Step 8 — Operational discipline

# Snapshot scheduled
sudo systemctl status etcd-snapshot.timer
# Expected: active

# Backup cron / cronjob running
kubectl get cronjob -n kube-system | grep etcd
# Expected: hourly schedule

# Monitoring targets
curl -k https://prometheus.example/api/v1/targets | jq '.data.activeTargets | length'

The cluster’s operational discipline (snapshots, monitoring, alerting) must be re-armed and validated.

$ etcdctl --endpoints=https://10.0.1.10:2379 --cacert=... --cert=... --key=... endpoint status --write-out=table

+---------------------------+------------------+---------+
|         ENDPOINT          |        ID        | RAFT INDEX |
+---------------------------+------------------+---------+
| https://10.0.1.10:2379    | <new-id>        |  41289312 |
| https://10.0.1.11:2379    | <new-id>        |  41289312 |
| https://10.0.1.12:2379    | <new-id>        |  41289312 |
+---------------------------+------------------+---------+

The reset of monitoring IDs

A subtle concern: external monitors, especially Prometheus’ etcd targets and any alerting tied to the old member IDs, may need updating.

# Prometheus target labels may reference member IDs
# Update as needed

# Alertmanager receivers usually unaffected

The post-restore observation period

A 30-minute observation window checks for:

• Steady-state write rate (close to pre-restore).
• Steady-state read latency.
• No growing error rates on the API server.
• All members’ raft index climbing.

gantt
    title 30-minute observation window
    dateFormat HH:mm
    axisFormat %H:%M
    section Cluster observation
    Initial validation :a1, 00:00, 5m
    Steady-state check :a2, after a1, 25m
    section Production traffic
    Reroute 10% :b1, 05:00, 10m
    Reroute 50% :b2, 15:00, 10m
    Reroute 100% :b3, 25:00, 5m

After 30 minutes of stable behaviour, production traffic is rerouted incrementally.

The validation failure modes

The post-incident discipline

After a restore, the team must:

• Document the incident: cause, decision, restoration time, data loss.
• Run a post-incident review: what went well, what was unclear, what should change.
• Update the runbook if the procedure diverged from the written steps.
• Schedule the next restore drill.

The restore is over; the lessons from it start.

Quiz

Production discipline

• Validate every layer before rerouting traffic. API, workloads, secrets, storage, networking.
• Treat readiness as a gate, not a checklist. A step fails and the gate is closed.
• 30 minutes of observation, minimum. Reconciliation drift surfaces over minutes; observation is the safety net.
• Document the data-loss window. Even successful restores lost state between the snapshot time and the restore time. Communicate this clearly.
• Schedule the next drill. A restore that has been done once is a restore that should be practised.

A restored cluster is production-ready when the operator can prove every layer is healthy and stable. The validation gate is the discipline that proves it.