Monitoring cluster storage and replication - watching the standby too

linux-cluster-monitoring-design established the shape: page on user impact, ticket on spent redundancy. Storage is where that rule is hardest to apply, because the signal that matters most comes from the node nobody is looking at.

The failure that monitoring is designed to miss

# Run on every node, active and standby alike
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT /dev/mapper/shared-lun
sudo multipath -ll
ls -l /dev/disk/by-id/ | grep -c wwn-

A standby that cannot see the device is a cluster with one node that pretends to have two.

Reachable is not the same as usable

Even a visible LUN is only half the claim. The checks worth running on the standby, in increasing order of strength:

The gap between rows one and three is where the interesting failures live: a LUN can be visible and still be masked to the wrong initiator group, so reads fail with a reservation conflict at exactly the wrong moment.

Do not test by mounting. A shared filesystem mounted on two nodes at once is the corruption that linux-two-nodes-mounting-the-same-block-risk describes; the cluster is what enforces single-writer, and a manual mount for a health check bypasses it.

Replicated storage: running is not healthy

DRBD, and replication in general, has two states that both look like “up”:

# drbdadm status

r0 role:Primary
disk:UpToDate
node2 role:Secondary
  peer-disk:UpToDate

r1 role:Primary
disk:UpToDate
node2 role:Secondary
  replication:SyncSource peer-disk:Inconsistent done:41.28

r0 has two good copies. r1 is serving perfectly and has one copy of the data - the peer is Inconsistent and will be until the resynchronisation completes. A failover during that window has no usable target.

The states worth encoding into checks:

• UpToDate/UpToDate - the only healthy steady state.
• UpToDate/Inconsistent with SyncSource - resynchronising. Expected after a failure, alarming if it never finishes or restarts repeatedly.
• UpToDate/Outdated - the peer is known stale and may not be promoted. Correct behaviour, spent redundancy.
• Diskless, DUnknown, or a connection state that is not Connected - degraded, and the peer may not be reachable at all.
• Split brain detected - both sides diverged. This is a data event, and linux-drbd-primary-secondary-and-split-brain covers the recovery. It never resolves itself.

Replication lag is a recovery point objective

For asynchronous replication - streaming replication in a database, an async DRBD protocol - the lag is not a performance metric. It is the amount of data you lose if you fail over right now.

-- PostgreSQL primary: how far behind is each standby, in bytes
SELECT client_addr,
       state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

Set the threshold from the RPO the service promised, not from what looks tidy on a graph. If the RPO is “no more than five seconds of writes”, then a lag alert at five seconds is the RPO alarm, and it deserves the same seriousness as the RPO.

Two traps:

• A standby that has disconnected has no lag at all. It vanishes from pg_stat_replication entirely, so a rule written as “lag greater than X” goes quiet exactly when things are worst. Alert on the count of connected standbys as well as on the lag of the ones present - the same aggregation-over-nothing problem as the membership rule in linux-cluster-monitoring-design.
• Lag measured in time can look fine on an idle system. A standby with zero write traffic reports zero lag whether or not it is actually replicating. Measure in bytes or LSN position where you can.

Redundancy underneath the cluster

The layers below the cluster degrade silently by design, which is the whole point of them:

# md RAID: a bracket with an underscore is a missing member
awk '/^md/ {name=$0} /\[.*_.*\]/ {print "DEGRADED: " name}' /proc/mdstat

# Multipath: count active paths per map, compare with expected
sudo multipath -ll | grep -E 'status=(active|enabled|failed)'

# The cluster's own view
sudo pcs status --full

A RAID array with a failed member and a multipath map down to one path both serve every request correctly. Nothing fails, no user notices, and the next failure is total. This is precisely the degraded-but-running category: a ticket with a deadline, never a page, and never nothing.

Which lane

The third row is the one that differs from most published guidance, and it follows directly from the top callout. A standby that cannot mount is not a redundancy warning - it is an outage that has already happened and has not yet been noticed. The cluster is running unprotected, and unlike a degraded RAID array, nothing is working to repair it.

Knowledge check