Cluster IR: shared storage loss - when a stop cannot succeed

The nodes are fine. The network is fine. The cluster is quorate. And over the next four minutes the cluster fences every node in turn, until there is nothing left running.

Shared storage loss is the incident where the cluster’s own safety mechanism becomes the amplifier. Understanding the cascade is what lets you stop it, and the sequence that stops it is not obvious.

The cascade

It runs like this, and it takes minutes rather than hours:

1. The shared LUN, NFS export or DRBD peer becomes unreachable. Nothing in the cluster has failed yet.
2. Processes doing I/O to that storage enter uninterruptible sleep (D state). They cannot be signalled, and they cannot be killed.
3. The resource agent monitor operation for the filesystem or database times out, because it also does I/O.
4. Pacemaker decides to recover the resource, which means stopping it first.
5. The stop operation also hangs, because stopping a filesystem resource means unmounting it, and the unmount blocks on the same dead storage.
6. A failed stop means Pacemaker cannot prove the resource is not running. Its only remaining tool for obtaining that proof is to fence the node.
7. The node is fenced. The resource is started on the next node, which reaches the same dead storage, and the cycle repeats.

Step 6 is the one that surprises people. A failed stop is not treated as “the resource is unhealthy” — it is treated as “the state of this resource is unknown”, and an unknown resource state on shared storage is the exact condition fencing exists to resolve. The cluster is behaving correctly. It is just behaving correctly about the wrong problem.

Recognising it from the cluster side

The tell is D-state processes on the node that owns the resource:

$ ps -eo state,pid,wchan:32,comm | awk '$1 ~ /D/'

D    3412 folio_wait_bit_common          kworker/u16:3
D    9188 nfs_wait_bit_killable           mysqld
D    9204 nfs_wait_bit_killable           mysqld
D   14771 __wait_on_freeing_inode         df

The wchan column names the kernel function each process is blocked in, which points straight at the layer that is stuck. nfs_wait_bit_killable is an NFS server that has stopped answering. Block-layer waits point at the SAN or multipath.

The df process in that output is the second tell: routine monitoring commands hang too. A monitoring agent that runs df every minute accumulates one stuck process per minute, and the node’s load average climbs into the hundreds without any CPU being used. High load with idle CPUs is a storage symptom, not a CPU symptom.

Then confirm at the kernel:

# Path and connection errors, with kernel timestamps
sudo dmesg -T | tail -40
sudo journalctl -k --since '-30 min' --utc -o short-iso-precise \
  | grep -iE 'i/o error|path|timeout|nfs|iscsi|scsi|dm-'

# Multipath: are the paths failed, or is the whole map gone?
sudo multipath -ll

# iSCSI sessions, if that is the transport
sudo iscsiadm -m session

Containment: stop the cluster making it worse

Before repairing anything, stop the fencing cascade. Maintenance mode makes every resource unmanaged: Pacemaker stops monitoring, stops recovering, and stops fencing.

# Halt the cascade. The cluster stops acting entirely.
sudo pcs property set maintenance-mode=true
sudo pcs status

If only one node is affected and the storage is genuinely available elsewhere, a narrower option is to stop that node being chosen while you work:

# Substitute your own values before running:
NODE=node2

# Marks resources on this node unmanaged, leaves the rest of the
# cluster fully operational
sudo pcs node maintenance "$NODE"
sudo pcs status nodes

Repair, in this order

1. Fix the storage. Restore the SAN path, the NFS server, the switch port, the DRBD link. Nothing on the cluster side can be fixed while the underlying device is gone.
2. Confirm the path is back at the block or mount layer, from every node, not just the one you are on. multipath -ll should show active paths; an NFS server should answer a mount probe.
3. Check what the hung I/O did. Processes in D state usually complete once the storage returns, but some do not, and a node with unrecoverable D-state processes needs a reboot rather than an argument.
4. Only then clear the cluster state: read the failed actions, then pcs resource cleanup, then pcs property set maintenance-mode=false.
5. Verify the service from a client, and verify the filesystem. Storage that vanished mid-write may need a consistency check that the resource agent will not run for you.

# Substitute your own values before running:
RESOURCE=fs-data

# Read the failure record BEFORE erasing it
sudo pcs resource failcount show "$RESOURCE"
sudo pcs status --full

# Then clear it
sudo pcs resource cleanup "$RESOURCE"
sudo pcs property set maintenance-mode=false
sudo pcs status

The unmount that will not unmount

Once the storage is back, a stale mount may still refuse to release. The two options are not interchangeable:

# Substitute your own values before running:
MNT=/srv/data

# Who is holding it? -n avoids DNS lookups, which also hang
sudo lsof -n "$MNT" 2>/dev/null | head
sudo fuser -vm "$MNT" 2>/dev/null

# Substitute your own values before running:
MNT=/srv/data

sudo umount -f "$MNT" || sudo umount -l "$MNT"
findmnt "$MNT" || echo "unmounted"

Knowledge check