HA validation — chaos testing, drills, observability

HA is a property of the cluster’s design and operations. A design that has not been tested in failure is not HA; it is design. This lesson walks the validation discipline: chaos tests, restore drills, observability for HA, and the cadence that keeps the cluster’s HA real.

The validation hierarchy

flowchart TB
    H1[Survive host loss] --> H2[Survive AZ loss]
    H2 --> H3[Survive HA failover LB]
    H3 --> H4[Restore from snapshot]
    H4 --> H5[Operate through full recovery]

Each level is harder than the previous. A production cluster validates each level periodically.

Chaos testing

Chaos testing introduces failures in a controlled way:

The test workflow:

sequenceDiagram
    participant Op as Operator
    participant C as Cluster
    participant O as Observability
    Op->>C: schedule chaos (kill host cp-1)
    Op->>O: capture pre-state (replicas, latency, errors)
    C->>C: chaos begins
    C->>O: monitor post-state
    Op->>O: validate SLOs met
    Op->>C: end chaos (or stay in failure mode for further testing)

$ kubectl get componentstatuses

scheduler       Healthy   ok
controller-manager   Healthy   ok
etcd-0          Healthy   {"health": "true"}

The “kill a host” test

The simplest chaos test: stop a kubelet on a worker.

# On the worker:
sudo systemctl stop kubelet
# Watch the node transition to NotReady:
kubectl get nodes

Expected behaviour:

• After node-monitor-grace-period, the node is NotReady.
• After node-eviction-timeout, Pods are evicted.
• The cluster reconciles: Deployments bring Pods on other nodes.
• The cluster reaches steady state with the lost node’s workloads re-scheduled.

The “kill a control-plane host” test

A larger chaos test: stop all the control-plane processes on one host.

sudo crictl stop $(crictl ps --name kube-apiserver -q)
sudo crictl stop $(crictl ps --name kube-controller-manager -q)
sudo crictl stop $(crictl ps --name kube-scheduler -q)
sudo crictl stop $(crictl ps --name etcd -q)

Expected behaviour:

• The cluster loses one control-plane host.
• The remaining control-plane hosts handle requests.
• The LB routes around the dead host (health check fails).
• Reconciliation continues on the remaining hosts.
• The cluster reaches steady state.

The “kill etcd” test

The most invasive test: take down etcd.

sudo crictl stop $(crictl ps --name etcd -q)

Expected behaviour:

• API server cannot write to etcd.
• The remaining etcd members (2 of 3 in stacked; 4 of 5 in external) may have quorum and continue.
• API server returns timeouts to clients.
• Once etcd restarts, the cluster recovers.

If the test kills all 3 etcd members simultaneously, the cluster is in quorum loss; the recovery is restore from snapshot (Part LXIX).

Restore drills

A restore drill is not chaos; it is a planned exercise to validate the restore path:

gantt
    title Restore drill timeline
    dateFormat HH:mm
    axisFormat %H:%M
    section Pick snapshot
    Find snapshot :a1, 00:00, 5m
    section Restore on sandbox
    Build sandbox :b1, after a1, 10m
    Run etcdutl snapshot restore :b2, after b1, 30m
    Spin up API server :b3, after b2, 10m
    section Validate
    Verify cluster :c1, after b3, 30m
    Document drill :c2, after c1, 15m

The drill:

1. Pick a recent snapshot (off-cluster).
2. Build a sandbox host (cloud VM or local VM).
3. Run etcdutl snapshot restore on the sandbox.
4. Start an etcd process on the sandbox.
5. Verify cluster health via etcdctl endpoint status.
6. Optionally: start the API server on the sandbox.
7. Document the drill outcome.

The drill validates the entire path from snapshot to running etcd. A drill that has been done in the past is reliable; a drill that has never been done is suspect.

Observability for HA

The control plane’s observability is critical:

The cluster’s SLOs are explicit and observable.

The chaos test cadence

gantt
    title Chaos test cadence
    dateFormat HH:mm
    axisFormat %H:%M
    section Daily
    Synthetic API requests :a1, 00:00, 1d
    section Weekly
    Kill a worker kubelet :b1, 1d, 1m
    Verify recovery :b2, after b1, 30m
    section Monthly
    Kill an etcd member :c1, 7d, 1m
    Verify recovery :c2, after c1, 30m
    section Quarterly
    Full cluster restore drill :d1, 90d, 1m

The cadence is the discipline:

• Daily: synthetic API requests validate the LB and authn/authz.
• Weekly: kill a worker kubelet; verify recovery.
• Monthly: kill an etcd member; verify quorum recovery.
• Quarterly: full cluster restore drill from snapshot.

The drill record

Each drill produces a record:

DRILL: Restore from snapshot on sandbox
TIMESTAMP: 2026-08-16 12:00 UTC
SNAPSHOT: s3://prod-etcd-backups/drill/20260715.db
OPERATOR: <name>
OUTCOME: success / partial / failed
TIME-TO-RESTORE: 30 minutes (vs 60-minute target)
OBSERVATIONS:
  - Sandbox host build was slow (10 minutes for apt update)
  - Certs on sandbox need updating (TTL was 90 days)
  - Drill is repeatable

The drill’s output is the procedure’s validation.

Chaos test anti-patterns

The production rule: chaos on a separate test cluster (or test windows on prod), with rollback procedures ready.

The runbook entry for HA validation

PRODUCTION HA VALIDATION
========================

Drill cadence: weekly, monthly, quarterly

Weekly (worker kubelet kill):
  1. Schedule window
  2. Pick a worker
  3. systemctl stop kubelet
  4. Watch node transition NotReady
  5. Watch eviction
  6. Watch workloads re-scheduled
  7. Bring kubelet back: systemctl start kubelet
  8. Verify health
  9. Document

Monthly (etcd member kill):
  1. Schedule window
  2. Pick an etcd member
  3. crictl stop <etcd-pod>
  4. Watch leader election
  5. Watch reconciliation
  6. Restart etcd: crictl rm <stopped-pod>; kubelet restarts
  7. Verify health
  8. Document

Quarterly (full restore drill):
  1. Pick snapshot from N days ago
  2. Build sandbox host
  3. Restore snapshot
  4. Spin up cluster from snapshot
  5. Verify API server, etcd, namespace visibility
  6. Document; update runbook

Quiz

Production discipline

• Validate HA continuously. Drift accumulates; unnoticed HA regressions are a real risk.
• Quarterly restore drill. Validates snapshot integrity and operator ability.
• Monthly chaos test. Validates in-life HA (etcd member kill, control-plane host kill).
• Weekly synthetic checks. Daily API requests from the kubeconfig validate the basic path.
• Update the runbook from drill findings. A runbook that doesn’t change is a runbook that wasn’t exercised.

HA validation is the discipline that converts HA design into working HA. Operating it well is keeping the discipline.