Failure: cluster node loss, quorum loss and dependency outages

Cluster failures break the single-host habits. The host you are logged into may be the one that is wrong, the service may be down because the cluster is protecting it, and a healthy dashboard is not evidence of a healthy service.

Node loss: three different states

The cluster reports a node as “offline”. That single word covers three situations with different causes and different recoveries:

Always check from two nodes. A node that considers itself online while its peers consider it offline is the definition of a partition, and only comparing views reveals it:

# corosync-quorumtool -s; corosync-cfgtool -s

Quorum information
------------------
Nodes:             1
Node ID:           1
Quorate:           No

Ring ID:           1.1a2
Membership information
----------------------
  Nodeid      Votes Name
       1          1 node1 (local)

LINK ID 0 udp
addr	= 10.0.0.11
status	= 1 3 3

Quorate: No with one member, while the other two nodes report themselves quorate with two members, is a partition, not a double node failure. Check the network before touching the cluster.

Quorum loss: resources freeze, and that is correct

When a partition loses quorum, Pacemaker stops making decisions. Resources are not moved and, depending on no-quorum-policy, are stopped. Operators read the frozen state as “the cluster is broken” and start forcing things.

The freeze is the safety mechanism. Two partitions that both believed they held quorum would both start the database on shared storage, and that is how a cluster corrupts data permanently rather than being down for twenty minutes.

The order that keeps the data:

1. Establish, from outside the cluster, whether the missing nodes are running. BMC power state or a console session - not a ping from the surviving node, which is on the network you already suspect.
2. If they are running, fix the network. The cluster recovers by itself when the rings come back.
3. If they are genuinely down and cannot be recovered in the incident window, power them off at the BMC so they cannot rejoin unexpectedly.
4. Only then relax quorum on the survivor, and record it as a deliberate act with a time and an owner.
5. Restore the original quorum configuration as soon as the peers return.

A failed fence is why the service will not come back

This is the cluster failure most often misdiagnosed, because every dashboard is green.

When Pacemaker cannot fence a node, it cannot prove that node has stopped its resources. It therefore refuses to start them anywhere - starting a second copy of a database on shared storage is the outcome fencing exists to prevent. The cluster is quorate, no resource is “failed”, and the service is down indefinitely.

# pcs status --full | sed -n '/Failed Fencing Actions/,/^$/p'; pcs stonith history show

Failed Fencing Actions:
* reboot of node3 failed: delegate=node1, client=pacemaker-controld, origin=node1,
  last-failed='Mon Aug 10 04:19:55 2026'

Common causes, in the order they occur in practice: BMC credentials changed or expired, the BMC is on a management network the surviving node cannot reach, the fence device configuration names the wrong port or host, or the fence agent package was upgraded and the parameters changed.

Test the fence device directly before assuming the cluster is at fault:

pcs stonith status
stonith_admin --list-registered
# a status call, not a reboot - it proves credentials and reachability

Shared dependencies fail the whole cluster at once

A cluster removes single points of failure inside itself and then quietly acquires new ones outside itself. When every node fails at the same moment, stop looking at the nodes.

• DNS. Nodes resolve each other by name for the cluster link, for storage, and for identity. A DNS outage can look exactly like a total cluster failure. Cluster membership should not depend on DNS: put the peers in /etc/hosts.
• Time. Corosync tolerates skew badly, Kerberos rejects it outright (typically past five minutes), and TLS validation fails against a wrong clock. Check chronyc tracking on every node, and compare them to each other, not just to a server.
• Identity. With LDAP or Active Directory unavailable, sudo and ssh logins can hang for the NSS timeout on every node simultaneously. The service may still be serving traffic while no operator can log in to see it. That is why a local break-glass account exists.
• Storage. A shared LUN or NFS export that disappears takes every node with it. dmesg on any node shows path or connection errors at the same timestamp.

Knowledge check