Cluster state, workers, workloads, persistent data, and validation — phases 5-9

Phases 5-9 of complete cluster loss recovery are the final stretch — cluster state, workers, workloads, persistent data, and validation. This lesson walks each phase, the dependencies, the validation cadence, and the operational discipline.

Phase 5: Cluster state

flowchart LR
    A[Git repository] --> B[CRDs]
    A --> C[ConfigMaps]
    A --> D[Namespaces]
    A --> E["RBAC, ServiceAccounts"]
    F["Vault / sealed-secrets"] --> G[Secrets]
    B --> H[Cluster state restored]
    C --> H
    D --> H
    E --> H
    G --> H

Cluster state includes:

• CRDs — every CustomResourceDefinition used by workloads (cert-manager, ingress-nginx, Argo CD).
• ConfigMaps — configuration that workloads mount.
• Namespaces — the namespace layout.
• RBAC — Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, ServiceAccounts.
• Secrets — replicated from Vault, sealed-secrets, or external secret stores.

The order: CRDs first, then ConfigMaps and Secrets, then RBAC, then ServiceAccounts. CRDs must exist before any CR is applied.

Phase 6: Workers join

# Substitute the token and CA cert hash the kubeadm init output printed:
JOIN_TOKEN=abcdef.0123456789abcdef
CA_CERT_HASH=sha256:56b65c61aafa58096802e3926e7d7a04b78df0d3f1cc3d425d4dbcb2be3582d1

kubeadm join lb-endpoint:6443 \
  --token "$JOIN_TOKEN" \
  --discovery-token-ca-cert-hash "$CA_CERT_HASH"

The worker join:

1. Connects to the API server at the load balancer.
2. Authenticates with the bootstrap token.
3. Verifies the CA cert hash.
4. Starts the kubelet.
5. Registers with the API server as a Node.

After joining:

kubectl get nodes

NAME    STATUS   ROLES           AGE   VERSION
cp-1    Ready    control-plane   30m   v1.34.0
cp-2    Ready    control-plane   30m   v1.34.0
cp-3    Ready    control-plane   30m   v1.34.0
worker-1 Ready   <none>          1m    v1.34.0
worker-2 Ready   <none>          1m    v1.34.0
worker-3 Ready   <none>          1m    v1.34.0

All nodes should be Ready. A NotReady node has a kubelet, CNI, or runtime issue.

Phase 7: Workloads applied

flowchart LR
    A[Git repository] --> B[Argo CD or kubectl apply]
    B --> C[Deployments]
    B --> D[StatefulSets]
    B --> E[Services]
    B --> F[Ingress]
    C --> G[Controllers reconcile]
    D --> G
    E --> G
    F --> G

Workloads are applied from Git (via Argo CD or Flux) or via kubectl apply. The controllers reconcile the actual state toward the desired state:

kubectl get pods -A

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
prod-app      api-xxxxx-yyyyy                   1/1     Running   0          2m
prod-app      api-xxxxx-zzzzz                   1/1     Running   0          2m
prod-data     postgres-0                        1/1     Running   0          2m
ingress       ingress-nginx-xxxxx-yyyyy         1/1     Running   0          2m
cert-manager  cert-manager-xxxxx-yyyyy          1/1     Running   0          2m

All Pods should be Running. If any are Pending, the CNI, the StorageClass, or the secrets are not ready.

Phase 8: Persistent data restored

velero restore create cluster-data \
  --from-backup daily-full-20260816030000 \
  --include-resources persistentvolumeclaims

Velero restores the PVC objects. Each PVC references a VolumeSnapshot via dataSource; the CSI driver provisions a new volume from the snapshot.

kubectl get pvc -A

NAMESPACE   NAME                STATUS   VOLUME               CAPACITY   ACCESSMODES
prod-data   postgres-data       Bound    pvc-aaaaa-bbbbb      100Gi      RWO
prod-data   postgres-wal        Bound    pvc-ccccc-ddddd      10Gi       RWO

All PVCs should be Bound. An unbound PVC indicates a snapshot reference error, a missing StorageClass, or a CSI driver issue.

After PVCs are bound, the workloads may need to be restarted to pick up the data:

kubectl rollout restart statefulset/postgres -n prod-data

Phase 9: Validation

flowchart TD
    A[Pods Running] --> B[Services route]
    B --> C[Ingress responds]
    C --> D[TLS valid]
    D --> E[Data accessible]
    E --> F[End-to-end tests pass]

The validation chain:

# 1. Pods Running
kubectl get pods -A | grep -v Running | grep -v Completed

# 2. Services route
kubectl get endpoints -A

# 3. Ingress responds
curl -k https://prod-app.example.com/health

# 4. TLS valid
openssl s_client -connect prod-app.example.com:443 -servername prod-app.example.com < /dev/null

# 5. Data accessible
kubectl exec -n prod-data postgres-0 -- psql -c "SELECT count(*) FROM orders;"

# 6. End-to-end test suite
./run-e2e-tests.sh

Each check catches a different failure mode. Skipping any one leaves a gap.

The recovery time budget (phases 5-9)

Combined with phases 1-4 (1h 25m), the complete recovery is approximately 3h 50m — within a 4-hour RTO target for Tier 3.

The operational failure modes

Phases 5-9 fail for predictable reasons:

• CRDs not in Git. Workloads fail because their CRDs are missing. Recovery: add CRDs to Git.
• Secrets not replicated. Workloads fail to authenticate. Recovery: replicate from Vault or sealed-secrets.
• Workers not joining. The token expired or the CA hash is wrong. Recovery: re-create the token.
• Workloads Pending. The CNI, StorageClass, or secrets are not ready. Recovery: re-verify earlier phases.
• PVCs not binding. The CSI driver is not installed or the snapshot is not ReadyToUse. Recovery: re-verify phase 4.
• Validation skipped. The cluster is declared recovered but the workloads are not functional. Recovery: re-run validation.

Quiz

Production discipline

Phases 5-9 in production rest on five non-negotiable elements:

• Apply the order. Cluster state, workers, workloads, persistent data, validation. Skipping the order produces a non-functional cluster.
• Replicate secrets separately. Vault, sealed-secrets, or ESO. Kubernetes Secrets in etcd are not recoverable from Velero.
• Verify data, not just Pods. Phase 9 must include database counts, file counts, or whatever indicates the data is present.
• Test the full chain quarterly. The complete nine-phase sequence is the only valid test of recovery. Quarterly drills catch ordering bugs, missing prerequisites, and validation gaps.
• Document the recovery time budget. Each phase has an expected time; the sum is the RTO. The business commitment depends on the actual RTO.

The recovery is complete only when the validation passes. A cluster that has not been validated is a cluster whose recovery is unknown. The discipline is to validate end-to-end and to treat any failure as a blocker for declaring recovery complete.