Cluster monitoring design - what to monitor in a cluster

A cluster has monitoring needs beyond a single host. This lesson covers what to monitor: quorum, membership, failovers, resources, and the actionable alerts.

What to monitor

For a cluster:

• Quorum: are enough nodes online?
• Membership: are the right nodes in the cluster?
• Resources: are the right resources running on the right nodes?
• Failovers: when did the last failover happen?
• Performance: latency, throughput, error rate.
• Dependencies: DNS, NTP, identity.

For Pacemaker specifically:

• crm_mon or pcs status for cluster state.
• Corosync logs for membership and quorum.
• Node_exporter for host metrics.
• The ClusterLabs ha_cluster_exporter for Prometheus metrics.

The exporter and its metric names

Prometheus does not read the CIB. Something has to translate cluster state into metrics, and for Pacemaker that is the ClusterLabs ha_cluster_exporter. It scrapes crm_mon, corosync-quorumtool and corosync-cfgtool, then exposes everything under the ha_cluster prefix.

The metrics used in this lesson:

Write your alerts against these names. An expression that names a metric no exporter produces is silently valid PromQL and permanently empty.

Quorum monitoring

Quorum is a Corosync property, not a Pacemaker one. Alert when the local partition loses it:

# fires per node, with the instance label of the reporter
ha_cluster_corosync_quorate == 0

Comparing a vector against the numeric literal 0 filters the samples: every sample whose value is 0 is kept, so the rule fires with the labels of the node that reported it.

If you would rather alert on the arithmetic, compare the votes held against the votes required. Both sides are real series, so the match must be spelled out:

sum by (instance) (ha_cluster_corosync_member_votes)
  < on (instance) ha_cluster_corosync_quorum_votes{type="quorum"}

On the host, two commands answer two different questions. corosync-quorumtool is authoritative for the vote arithmetic, because quorum is a Corosync property:

sudo corosync-quorumtool -s

Quorum information
------------------
Nodes:             3
Expected votes:    3
Highest expected:  3
Total votes:       3
Quorum:            2
Flags:             Quorate

Pacemaker reports only whether its partition is quorate, as part of the cluster summary:

sudo crm_mon -1 | head -8

Cluster Summary:
  * Current DC: node1 (version 2.1.6-...) - partition with quorum
  * 3 nodes configured
  * 7 resource instances configured

Alert when Flags: no longer contains Quorate, or when crm_mon reports partition WITHOUT quorum. Treat it as an outage rather than a warning: with no-quorum-policy=stop, the default, an inquorate partition stops every resource it holds.

Membership monitoring

ha_cluster_pacemaker_nodes emits one series for every node and every possible status, so the online series exists even while the node is down — its value is simply 0. That makes the per-node form the better alert, because it carries the name of the node that left:

# a cluster member is not online
ha_cluster_pacemaker_nodes{type="member", status="online"} == 0

If you want to alert on the size of the membership instead, count the online members and compare against the literal node count of this cluster:

# a three-node cluster is short a member
count(ha_cluster_pacemaker_nodes{type="member", status="online"} == 1) < 3

The literal 3 is deliberate. It belongs in the rule, next to the cluster it describes, not in a phantom metric.

Or in crm_mon:

Online: [ node1 node2 ]
OFFLINE: [ node3 ]

node3 is offline. Alert if unexpected.

Failover monitoring

Alert when a failover occurs:

# Corosync log
journalctl -u corosync | grep -i member
journalctl -u pacemaker | grep -i migrate

A failover event is informational but should be alerted if unexpected.

Resource monitoring

ha_cluster_pacemaker_resources carries a status label, and the active series exists for every configured resource. Sum it per resource: a total of 0 means the resource is running nowhere in the cluster.

# no instance of this resource is active on any node
sum by (resource) (ha_cluster_pacemaker_resources{status="active"}) == 0

A resource that keeps failing to start is the more urgent signal, and Pacemaker counts those for you:

ha_cluster_pacemaker_fail_count > 0

Or in pcs status:

pcs status --full
pcs resource failcount show

Look for resources that are not Started.

The two silent killers

Quorum, membership and resource state are the signals every cluster monitoring plan gets right. Two more decide whether you find out about the outage before your users do, and both are routinely missing.

Fencing failures

When a STONITH action fails, Pacemaker cannot prove the peer’s resources have stopped. It therefore refuses to start them anywhere - starting a resource that might still be running on the other node is how you corrupt shared storage. The result is the worst state a cluster can be in: quorate, no failed resources on the dashboard, and the service down indefinitely.

Nothing in the ordinary alert set catches this. The resource is not “failed”, it is unstarted by design. Alert on the fence history directly:

# Any non-empty output is a page-now condition
pcs status --full | sed -n '/Failed Fencing Actions/,/^$/p'
pcs stonith history show

# Machine-readable form for a check script
stonith_admin --history '*' --output-as=xml

In Prometheus, the fence agents are Pacemaker resources like any other, so their failures land in the fail-count series:

# a fence agent that is failing to fence
increase(ha_cluster_pacemaker_fail_count{resource=~"stonith.*|fence.*"}[15m]) > 0

Pair it with a periodic proof that fencing still works. A fence device whose credentials expired six months ago looks perfect until the day you need it, because nothing has asked it to do anything.

Degraded but running

A failover that succeeds produces no user-facing symptom and no alert. It also consumes the entire failure budget. The cluster now runs on one node where it was designed to run on three, and the next failure - the one you could have prevented - is a total outage.

Alert when the cluster is running with no redundancy left:

# running, but no failure budget remaining
count(ha_cluster_pacemaker_nodes{type="member", status="online"} == 1) < 3

# a promotable resource with a Promoted instance and no healthy peer
count by (resource) (ha_cluster_pacemaker_resources{status="unpromoted"} == 1) < 1

And on the storage layer, where “running” and “redundant” are most easily confused:

# replicated storage: both sides must read UpToDate/UpToDate
drbdadm status
# md RAID under the cluster
awk '/\[.*_.*\]/ {print "DEGRADED: " $0}' /proc/mdstat

Severity matters here. Degraded-but-running is a ticket, not a page: nobody should be woken for it, and nobody should be able to ignore it for a week either. Give it a deadline and put it on a dashboard the team actually reads.

Dependency monitoring

DNS, NTP, identity — the same as for any service.

Actionable alerts

For each alert:

• Symptom: what is the user seeing?
• Action: what should the operator do?
• Runbook: link to the procedure.

Example:

ALERT: ClusterQuorumLost
Symptom: Pacemaker cannot make decisions; resources are
         frozen on their current nodes.
Action: Investigate the partition; restore connectivity
         or fence the minority group.
Runbook: /runbooks/cluster/cluster-failover.md

The alert is actionable. The operator knows what to do.

Prove the rule can fire

An alert rule that is never tested is a rule you assume works. Check the syntax, then check the semantics:

# syntax and rule structure
promtool check rules /etc/prometheus/rules/cluster.yml

# does the expression return anything right now?
promtool query instant http://localhost:9090 \
  'ha_cluster_pacemaker_nodes{type="member", status="online"}'

check rules will not catch a misspelled metric name — the expression is valid PromQL either way. The instant query will: a healthy cluster must return one series per node. If it returns nothing, the name is wrong or the exporter is not scraped, and the alert built on it is decorative.

Finish with promtool test rules, which feeds synthetic samples through the rule and asserts that it fires. That is the only evidence that the alert works.

Knowledge check