Checklist: Cluster monitoring readiness

Use this checklist quarterly to verify that cluster monitoring is intact. Any Critical failure must be resolved before the quarter ends.

How to use this checklist

Run it once per cluster each quarter — but run the node-scoped commands (corosync-cfgtool -s, the BMC reachability check, pcs stonith status) on every node. A link or a fence path that is broken on one node only is precisely the case the cluster cannot survive, and it is invisible from anywhere else.

A Critical failure here does not mean the cluster is down. It means the cluster is running without the alerting that would tell you it had degraded: the failure is already possible, you would simply not hear about it. Resolve it before the quarter closes rather than waiting for it to arrive as an incident.

Eight items have no command, because no command can answer them — the performance baseline, whether alerts are actionable and correctly routed, whether runbooks and escalation paths are current, whether STONITH and split-brain handling have actually been exercised, and whether alert volume is being reviewed. Those are answered with artefacts: the baseline document, a sample alert showing symptom, action and runbook link, the dated record of the last fencing test. Attach them to the quarterly review ticket alongside the query output.

Reference expressions

The metric checks in this checklist are Prometheus queries, not shell commands. They run against the ClusterLabs ha_cluster_exporter. Pass the item only when the query returns data — a query that returns nothing means the alert built on it can never fire.

# quorum lost on this partition
ha_cluster_corosync_quorate == 0

# a cluster member is not online
ha_cluster_pacemaker_nodes{type="member", status="online"} == 0

# a resource is active on no node
sum by (resource) (ha_cluster_pacemaker_resources{status="active"}) == 0

# a resource is failing to start
ha_cluster_pacemaker_fail_count > 0

# the exporter itself stopped reporting
up{job="ha_cluster"} == 0

Never compare a metric against a bare word such as expected_count. PromQL reads it as another metric name, the comparison finds nothing to match, and the rule sits inactive forever while looking correct in the config.

Confirm the cluster’s own view separately:

pcs quorum status
pcs status --full
pcs resource failcount show
corosync-cfgtool -s          # every link must report: connected
pcs stonith status           # every fence device must be Started

Two of those are routinely missed. A Corosync link that goes down silently leaves the cluster running on one path, so the next failure on that path is a partition rather than a failover. And a fence device that cannot reach its target BMC fails only at the moment it is needed - the cluster then blocks recovery rather than fencing, because Pacemaker will not promote resources it cannot safely take. Monitor the BMC path continuously, not at build time.

Sign-off

• Operator: _________________ Date: ___________
• Reviewer: ________________ Date: ___________