Health validation after a change - the safety net

After every change, validate. The validation is the safety net that catches issues before the next node. This lesson covers what to validate and how.

The validation checklist

After every change:

• Service responds: smoke test the application.
• Performance baseline: latency and throughput within expected range.
• No new errors: log volume and error rate.
• All resources running: cluster status is healthy.
• Synthetic monitoring: external check passes.

Smoke tests

A smoke test is a quick check that the basic function works:

# HTTP service
curl -sf https://service/health
# 200 = OK

# TCP service
nc -zv service 5432
# succeeded = OK

# Database
psql -c "SELECT 1" -h db-host
# 1 = OK

Smoke tests are fast (seconds) and catch obvious failures.

Performance baseline

Compare current performance to baseline:

# Latency
time curl -sf https://service/

# Throughput
ab -n 1000 -c 10 https://service/

If latency is within 10% of baseline and throughput is within 10%, the change is healthy.

Logs

Check logs for new errors:

# Application logs
journalctl -u myapp --since "10 minutes ago" | grep -i error

# Cluster logs
journalctl -u pacemaker --since "10 minutes ago" | grep -i error

A new error pattern after a change is a red flag.

Cluster status

pcs status

All resources running, no failed actions, no warnings.

Synthetic monitoring

If configured, the external check is the most important validation:

# Check synthetic monitor
curl -sf https://synthetic-monitor/check

The synthetic monitor makes a real request from outside the cluster, exercising the full path.

Roll back on failed validation

If validation fails:

1. Stop the change. Do not proceed to the next node.
2. Roll back to the previous state.
3. Investigate the cause.
4. Plan the fix.
5. Test the fix in staging.
6. Re-apply when ready.

The discipline: validate first, roll back if needed. Never proceed on a failed validation.

Document the validation

CHANGE: 2026-08-09 14:00 - nginx update
NODES: node1 (canary), node2, node3 (after canary)
VALIDATION:
- Smoke test: PASS
- Latency: 45ms (baseline 47ms)
- Errors: 0 new in 10 min
- Cluster status: HEALTHY
- Synthetic: PASS
DECISION: proceed to node2

The validation log is the audit trail. If something goes wrong later, the log shows what was tested.

Knowledge check