What cluster monitoring cannot tell you

The five lessons before this one build a monitoring system for a cluster. This one is about its boundary, because the most expensive cluster incidents happen inside things monitoring is structurally incapable of reporting - not because the rules were wrong, but because no rule could have been right.

Monitoring answers one question: what is the state of the things I asked about? Here are seven questions it cannot answer, and what to do instead of each.

1. Will a failover work?

Monitoring reports that a failover did work, after one. It cannot report that the next one will.

A standby that has never taken production load, a resource agent whose start path has not run in eighteen months, a filesystem that mounts fine empty and behaves differently under load - all of these look identical to working ones, because nothing has asked them to do anything.

Instead: a drill calendar. Fail over deliberately, on a schedule, and let the standby carry real load for long enough to be a test. The date of the last successful drill is a monitoring signal in its own right, and one you can alert on when it goes stale.

2. Is the fence device working?

A fence device sits idle for months. Its credentials expire, a firewall rule changes, the management VLAN is renumbered - and none of that produces a failure, because nothing has asked it to fence anything.

The alert set from linux-cluster-monitoring-design catches a fence that failed. It cannot catch a fence that has never been attempted, and the difference between “no failures” and “working” is the whole gap.

Instead: exercise it. A status call to the device proves credentials and reachability; a real fence proves the rest. Record the date, and treat a fence device unproven this quarter as a ticket with a deadline.

3. Am I inside the partition?

4. Is “not running” correct or wrong?

A resource that is stopped looks the same whether it is stopped because an operator put the node in standby for patching, or because a fence failed and Pacemaker refuses to start it anywhere.

Monitoring has no concept of intent. It reports state, and both states are “stopped”.

Instead: supply the intent from outside.

• Put nodes and resources into the cluster’s own maintenance mode, and have the monitoring respect it, so a planned stop is distinguishable from an unplanned one.
• Alert on the reason separately, which is why failed fencing actions get their own rule rather than being inferred from resource state:

pcs status --full | sed -n '/Failed Fencing Actions/,/^$/p'
pcs stonith history show

5. Is the data correct?

Every signal in this part is about processes, states and counters. None of them says the bytes are right.

A replica that is connected and reporting zero lag can still have diverged. A database that answers every query can be serving a table that lost rows during a failover. A filesystem mounts, reads and writes with a silently corrupted extent in the middle of it.

Instead: verification that reads the data.

• Restore tests. A backup that has never been restored is a file, and restoring it is the only check that reads all of it.
• Application-level consistency checks - row counts, checksums, reconciliation against a system of record.
• Storage-level scrubs, where the filesystem or array supports them.

These are periodic jobs, not metrics, and their result is what monitoring watches.

6. Why did this happen?

Monitoring shows state over time. It does not show causation, and the most common cause is a change.

A graph that turns red at 14:07 tells you when. Whether it was the config push at 14:05, the certificate that expired at 14:00, or the batch job that starts at the top of the hour is not in the metrics.

Instead: annotate. Deployments, config pushes, package upgrades and maintenance windows on the same time axis as the metrics turn a twenty-minute correlation hunt into a glance. The change record is monitoring data.

7. Is no data good news?

The failure mode that defeats every rule above: the rule stops evaluating and silence reads as health.

linux-cluster-monitoring-design covers two mechanisms - an aggregation over an empty selector returns nothing, and a misspelled metric name is valid PromQL that matches nothing forever. There are more: an exporter that crashed, a scrape target removed by a config change, a node whose firewall now blocks the scrape.

Instead: monitor the monitoring.

# the exporter is not being scraped successfully
up{job="ha_cluster"} == 0

# no cluster samples at all in the last 10 minutes
absent_over_time(ha_cluster_pacemaker_nodes[10m])

And a dead man’s switch: an alert rule that is always firing, routed to a receiver that pages if it ever stops arriving. It is the only construction that detects a monitoring pipeline which has failed silently end to end, because every other alert depends on the pipeline it is trying to check.

Prove rules can fire before trusting them:

promtool check rules /etc/prometheus/rules/cluster.yml
promtool test rules /etc/prometheus/rules/cluster_test.yml

check rules validates the syntax. test rules feeds synthetic samples through the rule and asserts that it fires, which is the only evidence that the alert works.

What is worth waking someone for

The gaps above explain why alert sets grow: each one, once understood, suggests another rule. The counterweight is a decision procedure, applied to every alert before it gets a paging route.

Page only if all three are true:

1. A user is affected now, or will be within the response time. Not “a component is unhealthy” - a user. Spent redundancy fails this test, which is why it is a ticket.
2. It gets worse without intervention. A cleanly completed failover is stable. A replication divergence is not: every minute adds data that has to be reconciled by hand.
3. The person on call can do something about it now. If the only action is to open a ticket for tomorrow, it is that ticket. A page whose runbook says “investigate in the morning” trains people to silence pages.

If any answer is no, it is a ticket with a deadline. If all three are no, it belongs on a dashboard - and if it does not belong on a dashboard either, delete the rule rather than leaving it to be ignored.

The honest summary

Cluster monitoring tells you, reliably: what the cluster believes about itself right now, and what it believed recently. That is genuinely valuable and it is bounded.

It does not tell you whether the parts you have never used will work, whether the view you are reading is from inside the failure, whether a stopped thing was stopped on purpose, whether the data is correct, why anything happened, or whether the absence of an alert means anything at all.

Each of those gaps has a countermeasure, and none of the countermeasures is a metric. They are drills, external probes, maintenance records, restore tests, change annotations and a dead man’s switch. A monitoring plan that lists only rules has covered the easy half.

Knowledge check