Failure detection - how a cluster decides a node has failed

Quorum answers “which partition may act”. Before that question can be asked, something has to decide that a node is missing at all. That decision is the least examined part of most cluster designs, and it is the one that fires at three in the morning.

Detection is inference from silence

No cluster ever observes a node dying. It observes that no message arrived within a time limit, and it infers failure from that silence.

This is worth stating in those words, because the whole rest of the discipline follows from it:

• Silence and death are not the same event. A node can be silent and fully alive - stalled on I/O, swapping, its corosync process starved of CPU, or on the far side of a broken switch.
• The inference is therefore sometimes wrong, and being wrong is not free: the cluster fences a live node and moves its resources, which is an outage it caused itself.
• Nothing in the detection layer can close that gap. Only fencing converts “we heard nothing” into “it has stopped”, which is why linux-why-fencing-exists exists as a separate discipline rather than a configuration option.

Node-level detection: the corosync token

Corosync passes a token around the ring. A node that holds the token and does not pass it on is retransmitted to a fixed number of times, and when the token still does not complete its circuit within the token timeout, the ring is declared failed and reformed without that node.

The parameters live in the totem section of /etc/corosync/corosync.conf. Read the values your cluster is actually running rather than trusting any published default - several of them are derived at runtime from the node count, so the effective value is not the one in the file:

# corosync-cmapctl | grep -E 'totem\.(token|consensus|join)|runtime\.config\.totem'

runtime.config.totem.token (u32) = 3650
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
runtime.config.totem.consensus (u32) = 4380
runtime.config.totem.join (u32) = 50

The two that matter for detection latency:

Detection is not instantaneous and not exactly token either: the ring has to fail, a new membership has to be agreed, and Pacemaker then has to act on it. Measure it on your own cluster rather than computing it - stop corosync on a node and time the gap to the membership change in the log on a survivor:

# On a survivor, before you break anything
journalctl -u corosync -u pacemaker -f -o short-iso

Then compare the timestamp of the last healthy token to the Members left line. That number is your real detection latency, and it belongs in the runbook next to the recovery time objective.

Ring health is separate from membership and is the signal that tells you detection is about to get exciting:

sudo corosync-cfgtool -s
sudo corosync-cfgtool -n

A link showing anything other than a healthy status, or a log full of token retransmits, means the cluster is close to declaring a failure that has not happened.

The false-positive trap

Every operator eventually wants faster failover and reaches for the token timeout. It works, and it is one of the more effective ways to cause an outage.

The default is deliberately slack. Corosync is tuned to tolerate a busy production Linux host rather than to minimise failover time on an idle one.

Resource-level detection: monitor operations

Corosync detects nodes. Pacemaker detects resources, and it does so by running the resource agent’s monitor action on a schedule. If the monitor fails, or takes longer than its timeout, Pacemaker treats the resource as failed.

# A monitor operation with an explicit interval and timeout
pcs resource update pgsql op monitor interval=20s timeout=30s

# What is configured now
pcs resource config pgsql

Three properties of this deserve attention:

• A monitor timeout is treated as a failure. The agent that took too long may have been perfectly healthy behind a slow disk. The same false-positive logic as the token timeout applies, one layer up.
• The interval bounds your detection time. A resource that died one second after a 20-second monitor completed is undetected for nineteen seconds. Shortening the interval costs CPU and increases false positives.
• Repeated failures escalate. Pacemaker counts failures per resource per node, and migration-threshold decides when to stop retrying locally and move the resource elsewhere. Without it, a resource can restart in place forever while the service is down.

# Move the resource after 3 failures on a node rather than retrying forever
pcs resource update pgsql meta migration-threshold=3

# Inspect and clear the counters
pcs resource failcount show
pcs resource cleanup pgsql

What happens after the decision

Detection produces a membership change; quorum arithmetic then decides whether this partition may act. If it may not, no-quorum-policy decides what happens to the resources this partition is already running - and the choice is a real decision with real consequences, not a default to inherit.

demote is the one worth understanding, because it is the honest description of a very common requirement. A demoted database replica is still running and still answers reads; what it will not do is accept writes, because it is no longer promoted. A partition that has lost quorum can therefore keep serving read traffic while being structurally incapable of becoming a second writer.

# Read what is set now, in a form that works on any pcs version
sudo cibadmin --query --scope crm_config

# Set it deliberately
sudo pcs property set no-quorum-policy=demote

The allowed values, and the name of the self-fencing option in particular, have changed across Pacemaker releases. Check the documentation for the version you are running rather than copying a value from a blog post; an unrecognised value is rejected, but a recognised value with different semantics than you expected is silent.

The detection budget

Put the numbers end to end and the recovery time objective becomes arithmetic rather than hope:

Node dies
  + token timeout                  ring declared failed
  + consensus                      new membership agreed
  + fence action                   node proven stopped
  + resource start + monitor       service actually serving
  ------------------------------------------------------
  = time to recovery

The fence term is usually the largest and the most variable, because it depends on a BMC responding. Measuring each term once, on your own hardware, is worth more than any published default - and it is the only way to find out that your fence device takes ninety seconds before anybody is depending on it.

Knowledge check