Before deploymentkubernetes-observability-readiness

Cluster Observability Checklist

13 items ·7 critical ·5 warn ·1 info

How to use this checklist

The test for this checklist is not whether dashboards exist. It is whether an engineer paged at 3 a.m. can answer ‘what changed and what is broken’ without shell access to a node.

Mark an item N/A when it genuinely does not apply, and write down why. An unexplained N/A is the most common way a checklist stops working.

Sign-off

Every critical item must pass. A failing critical item blocks the deployment or the maintenance window; it is not a note for later. Record the date, the reviewer, and the disposition of every item that did not pass.

Critical7 items

metrics-server is installed and healthy, so that `kubectl top` and the HPA have a data source.
```
kubectl top nodes
```
API server, etcd, scheduler, and controller-manager metrics are scraped, not just node and Pod metrics.
Container logs ship off the node. A node that has failed is the node whose logs you need, and local logs die with it.
Kubernetes Events are captured to the log pipeline. They expire from the API after an hour by default and are frequently the whole story.
```
kubectl get events -A --sort-by=.lastTimestamp | tail
```
Alerts fire on user-visible symptoms, not only on component health. A cluster can be entirely green while the service is down.
Alerts exist and have been test-fired for: node NotReady, Pod CrashLoopBackOff, PVC nearly full, certificate expiry, etcd leader changes, API latency.
Every alert routes to someone who is on call and empowered to act, and the routing has been tested outside business hours.

Warning5 items

Metrics retention is long enough to cover the longest plausible investigation, and the storage for it is sized and monitored.
Log retention meets the incident-review window and any regulatory requirement, and rotation is verified rather than assumed.
The alert set has been pruned. An alert that fires daily and is always ignored is training the team to ignore alerts.
A dashboard exists for cluster health, node capacity, and per-workload saturation, and each is linked from the runbook.
Every alert carries a link to the runbook entry for that alert.

Info1 item

Distributed tracing is available for request-path services, or its absence is a recorded decision.