The three layers
Monitoring must cover all three:
- Host: CPU, memory, disk, network. node_exporter.
- Container: per-container CPU, memory, network, restart count. cAdvisor.
- Application: request rate, latency, error rate. Application /metrics endpoint.
Missing any one layer produces blind spots at incident time.
Critical alerts every production stack should have
- OOM kill (
container_oom_events_total > 0) - Container restart loop (
increase(container_restart_count[15m]) > 3) - Disk fill (
predict_linear(node_filesystem_avail[6h], 24*3600) < 0) - Certificate expiry (
probe_ssl_earliest_cert_expiry - time() < 30 * 86400) - Registry reachability (synthetic probe)
- Backup failure (cron exit code != 0)
Cadence
- Continuous: Prometheus / Grafana scrape.
- Daily: alert routing test.
- Weekly: this checklist.
- Quarterly: alert-testing exercise (synthetic incident).