Monitoring failovers and resources - the cluster activity

Failovers and resource state are the cluster activity signals. This lesson covers how to monitor them and how to alert on anomalies.

Track failover events

A failover is when a resource moves from one node to another. Track:

• When: timestamp of the failover.
• What: which resource.
• Why: cause (planned maintenance, node failure, etc.).

# Failed actions and current placement, in one view
sudo pcs status --full

# Pacemaker failure history per resource and node
sudo pcs resource failcount show

# One-shot view with fail counts and migration summary
sudo crm_mon -1 -A -f -r

# Corosync and Pacemaker logs
sudo journalctl -u corosync -S -24h | grep -i member
sudo journalctl -u pacemaker -S -24h \
  | grep -E 'Transition|Recover|Move|Promote|fence'

Pacemaker has no “show me the failover history” subcommand. The history is assembled from three places: the Failed Resource Actions section of pcs status --full, the fail counts, and the transition lines in the Pacemaker journal. If you find a runbook that offers a single command for it, check that the command exists before you rely on it in a post-incident review.

For the why behind a specific move, replay the saved policy-engine input for that transition:

ls -t /var/lib/pacemaker/pengine/pe-input-*.bz2 | head
sudo crm_simulate -Sx /var/lib/pacemaker/pengine/pe-input-42.bz2 -s

Pacemaker writes one of those files per transition. Replaying it shows the scores the cluster used and the actions it chose, which is the difference between “the resource moved” and “the resource moved because the monitor on node1 failed twice and the fail count exceeded the migration threshold”.

Monitor resource state

# Substitute your own values before running:
RESOURCE=web-vip

# Current state
sudo pcs status --full

# Per resource
sudo pcs resource config "$RESOURCE"

# One-shot cluster view with fail counts
sudo crm_mon --one-shot --failcounts

pcs resource show <resource> was removed in pcs 0.11. On a current cluster it either errors or prints the status list rather than the configuration you asked for. The replacement is pcs resource config <resource>.

In Prometheus, cluster resource state comes from the ClusterLabs ha_cluster_exporter. It publishes ha_cluster_pacemaker_resources, with one series per resource per status, valued 1 when the resource holds that status and 0 when it does not.

Alert when a resource is running nowhere:

sum by (resource) (ha_cluster_pacemaker_resources{status="active"}) == 0

The sum by (resource) collapses the per-node series, so a resource that has moved between nodes still counts as one. A total of 0 means no node is running it.

Alert when a node drops out of the cluster:

ha_cluster_pacemaker_nodes{status="online"} == 0

Alert when a resource cannot start:

ha_cluster_pacemaker_fail_count > 0

As alerting rules, with a for clause so a normal, successful failover does not page anyone:

- alert: PacemakerResourceNotRunning
  expr: sum by (resource) (ha_cluster_pacemaker_resources{status="active"}) == 0
  for: 2m
- alert: PacemakerNodeNotOnline
  expr: ha_cluster_pacemaker_nodes{status="online"} == 0
  for: 1m

Use the metric names published by the exporter you actually deployed. Check them against curl -s localhost:9664/metrics on a node before writing the rule: a rule built on a metric name that nothing exports loads cleanly, stays inactive forever, and is indistinguishable from a healthy cluster.

Both compare a vector against a numeric literal, which is a filter: the matching samples are returned with their labels, so the alert names the resource.

Build a resource dashboard

A typical cluster dashboard shows:

• Cluster status (quorum, members).
• Per-resource state (which node, running or not).
• Recent failovers.
• Performance per node.

Tools:

• HAProxy + keepalived for floating IP.
• Pacemaker + pcs for cluster state.
• Prometheus + Grafana for metrics.
• Grafana dashboards for visualisation.

Alert on anomalies

Alert when:

• A resource fails over unexpectedly.
• A resource is not running for too long.
• A node leaves or rejoins the cluster.
• Quorum is lost (handled separately).
• A failover fails (resource cannot start on the target).

For each, the runbook is the action.

Track resource history

Pacemaker records a fail count per resource, per node, per operation:

sudo pcs resource failcount show

Output (illustrative):

Failcounts for resource 'web'
  node1: 3
  node2: 1

Fail counts are cumulative until they are cleared, so treat them as a trend, not a live state. Clear them deliberately with pcs resource cleanup <resource> once the cause is fixed; a stale fail count will keep a resource off a node that is now healthy.

This is the audit trail of cluster activity. Use it to identify flaky resources or recurring issues.

Knowledge check