Actionable cluster alerts - the alert that leads to action · Linux

An actionable alert leads to a specific action. An inactionable alert is noise. This lesson covers how to design alerts that are actionable.

What makes an alert actionable

An actionable alert has:

Symptom: what the user is experiencing.
Action: what the operator should do.
Runbook: link to the procedure.
Severity: how urgent is the response.

A good alert:

“API latency p99 > 1 second for 5 minutes” is actionable: investigate the application, check the recent deploy, roll back if needed.
“Cluster quorum lost” is actionable: investigate the network, restore connectivity, fence the lost node.
“Disk will be full in 7 days at the current rate” is actionable: archive old data or grow the volume. Note what makes it good - it is a prediction of user impact with a deadline attached, not a level crossed.

A bad alert:

“CPU > 80%” is not actionable: CPU is high, but the operator does not know what to do. Compare it with the disk alert above: 80% CPU says nothing about when, or whether, anything breaks.

Avoid alert fatigue

Alert fatigue is when the operator ignores all alerts because most are noise:

Alert only on user impact: a user is experiencing a problem.
Alert only on actionable signals: the operator can do something.
Route resource metrics to the right lane: CPU, memory and disk rarely justify waking someone. They are not forbidden - they belong in a ticket lane, and they page only when they predict imminent user impact. See below.
Deduplicate and group: one alert per incident, not per metric.

Match alert to runbook

Every alert should have a runbook:

ALERT: ClusterQuorumLost
SEVERITY: critical
SYMPTOM: Pacemaker cannot make decisions
ACTION: Investigate the network, restore connectivity, or
        fence the lost node
RUNBOOK: /runbooks/cluster/quorum-loss.md

The runbook is the procedure. The alert is the trigger. The operator knows what to do.

Alert taxonomy

For a cluster, define a taxonomy of alerts:

Alert	Lane	Action
ClusterQuorumLost	page	Network or fence
NodeOffline	ticket	Investigate; the cluster is still serving, but the redundancy is spent
ResourceNotRunning anywhere	page	Check logs, restart — no node is running it, so the user is affected
ResourceNotRunning on one node	ticket	It failed over; the service is up. Diagnosis, not a page
ResourceFailsToStart	page	Check constraints, target — it cannot recover on its own
FailoverFrequencyHigh	ticket	Review constraints, capacity
SplitBrainDetected	page	Fence, restore data

The split of ResourceNotRunning into two rows is the whole point. “This resource is not running here” is normal — it is what a successful failover looks like. Alert on the aggregate instead:

# nobody is running it: this is user impact, page
count by (resource) (ha_cluster_pacemaker_resources{status="started"}) == 0

Each alert is clear, has a lane, and has a runbook.

Which lane, and why

Page on user-facing symptoms and on states the cluster cannot recover from by itself: unreachable service, latency outside objective, quorum lost, split brain, a resource running nowhere.
Ticket on degraded-but-running and on resource pressure: one node left, a failover that already succeeded, a filesystem at 85%, a fence device not proven this quarter.
Never neither. A cluster that has spent its redundancy is one failure from an outage, and without the ticket nobody is looking. This is the same split cluster monitoring design makes; the two lessons agree.

For resource metrics, the lane follows from the query. A level is a ticket; a prediction is a page:

# ticket: a filesystem is filling up
node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.15

# page: it will be full within four hours at the current rate
predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0