Shared storage failure modes - the single point of failure you bought · Linux

Shared storage solves a real problem: the data is reachable from every node, so the cluster manager can start the service anywhere without copying anything. The price is a component every node depends on, and this lesson is about what that component’s failure looks like from a Linux host.

The short version is that it usually does not look like an error. It looks like nothing happening.

What is actually shared

Draw the path from the application to the bytes and count the things that only exist once:

The array or the NFS server. Dual controllers help; one chassis, one firmware version and one configuration database do not stop being single.
The fabric. Two switches with a shared control plane (a stack, or MLAG with a peer link) share a failure mode.
The LUN or the export itself. One namespace, one set of permissions, one place a mistake lands.
The storage network. Covered in the cluster networking part; here it matters because losing it looks exactly like losing the array.

Redundancy inside the array removes component failures. It does not remove a bad firmware upgrade, a full filesystem on the appliance, an expired certificate on the management interface, or an operator deleting the wrong LUN. Those reach every node at once, and that is the definition of a single point of failure regardless of how many power supplies the chassis has.

Linux does not fail storage fast by default

This is the part that surprises people. Every layer in the storage path is tuned, sensibly, for a transient glitch: retry, wait, retry again, on the assumption that the storage will come back and losing the I/O would be worse than waiting.

When the storage does not come back, that tuning becomes an indefinite hang.

NFS. The default is hard, which means the client retries forever. Processes doing I/O enter uninterruptible sleep - state D in ps - where they cannot be killed by an ordinary signal, cannot be reaped, and hold whatever locks and file descriptors they had. The load average climbs while the CPU is idle.

ps -eo state,pid,wchan:24,cmd | awk '$1 ~ /^D/'

The kernel says so, once, and then stops:

journalctl -k | grep -i 'not responding'

soft is not the fix, and intr no longer exists

The obvious response to a hang is soft, which makes the client give up after timeo and retrans and return an error to the application. For read-mostly, restartable workloads that is reasonable. For anything that writes, it is a data-integrity setting dressed as a timeout: the application receives EIO on a write it believed had succeeded, and most applications handle that badly or not at all. NFS documentation has warned against soft for writable mounts for two decades.

The other half of the folklore is intr, which was supposed to make hard mounts interruptible. It has been a no-op since Linux 2.6.25 - the kernel made hard mounts killable by SIGKILL instead, so intr and nointr are accepted and ignored. A mount option in your Ansible role that has done nothing for fifteen years is not the safety net you think you have.

The workable middle ground for a writable mount is hard with a bounded retry budget, so the hang is long enough to survive a switch reload and short enough that the cluster can act:

hard,timeo=600,retrans=2,_netdev

timeo is in tenths of a second, so 600 is 60 seconds per attempt.

iSCSI. When a session drops, open-iscsi blocks I/O to the device for node.session.timeo.replacement_timeout seconds while it tries to re-establish the session. The default is 120 seconds, which is right for a single-path setup where waiting beats failing.

It is wrong for a multipath setup. With multipath above it, you want the path to fail quickly so multipathd can route around it. The documented value for multipath deployments is 5 seconds, in /etc/iscsi/iscsid.conf:

node.session.timeo.replacement_timeout = 5

Existing sessions keep the value they were created with, so change the file, then log out and back in - or reboot the node during a maintenance window.

Multipath. no_path_retry decides what happens when the last path dies:

no_path_retry queue (or the queue_if_no_path feature) queues I/O indefinitely. Nothing fails; everything waits. This is the setting most vendor defaults ship.
no_path_retry fail (or 0) returns errors immediately.
no_path_retry <n> retries for roughly n polling intervals and then fails.

sudo multipath -ll
sudo multipathd show config | grep -i no_path_retry

Why one hung mount can fence the whole cluster

Follow the consequence through Pacemaker.

The array goes away. The Filesystem monitor operation runs and touches the mount.
The monitor blocks in D state.
The operation exceeds its timeout. Pacemaker records a monitor failure and starts recovery: stop the resource, then start it elsewhere.
The stop runs umount. umount blocks in D state too, because the filesystem cannot flush.
The stop times out. A failed stop means Pacemaker cannot prove the node stopped writing, so it fences the node.
Every other node is attached to the same array and is at step 2.

The result is a cluster that fences itself into the ground while the actual fault is a storage outage no amount of fencing addresses. Worse, the resource cannot start anywhere, because “anywhere” has the same problem.

queue_if_no_path plus a cluster is a fencing loop

no_path_retry queue is a good default for a standalone database server: the array reboots, I/O resumes, nobody noticed. On a cluster node it converts a storage outage into a node outage, and then into every node’s outage.

If the cluster is going to fence on a hung stop - and it is, because that is correct - then the storage layer beneath it must be configured to fail rather than to wait. Bound no_path_retry to a number of retries that comfortably covers a controller failover and no more, so the I/O error arrives while the node is still able to act on it.

The general rule: the storage timeout budget must be shorter than the cluster’s stop timeout, which must be shorter than the fencing timeout. Write those three numbers down and check they are in that order.

Verify the ordering you actually have:

Read-only / Safecluster timeout budget— Compare these with the storage-side timeouts. A 100s stop above a multipath layer that queues forever will never complete.

$ sudo pcs resource config app_fs | grep -E 'timeout|on-fail'

              start interval=0s timeout=60s (app_fs-start-interval-0s)
            stop interval=0s timeout=100s on-fail=fence (app_fs-stop-interval-0s)
            monitor interval=20s timeout=40s on-fail=fence (app_fs-monitor-interval-20s)

Illustrative output

HA for the shared component itself

Making the shared thing redundant moves the problem rather than removing it, but it moves it somewhere much better.

An HA NFS server - two nodes, the export following the service, backed by DRBD or by a shared LUN - removes the server from the SPOF list and leaves the storage behind it. Two details decide whether it works:

The NFS server’s state directory must move with the service. Client lock and open state lives there; if it stays on the failed node, clients silently lose their locks after failover. The ocf:heartbeat:nfsserver agent takes an nfs_shared_infodir parameter pointing at a directory on the shared volume for exactly this.
Clients hang for the grace period after failover. NFSv4 refuses new locks while it waits for existing clients to reclaim theirs - 90 seconds by default. That is correct behaviour and it is also your RTO. Test with a client under load and measure it, because “the failover took two seconds” is a statement about the server, not about the application.

A redundant fabric - two independent switches, two HBAs or two NICs, multipath on top - removes the path from the SPOF list. Verify it by pulling a cable in a maintenance window and watching multipath -ll, not by reading the diagram.

What redundancy cannot remove is the array as a configuration and firmware entity. The answer to that is not more redundancy inside the box; it is a copy of the data outside it. Backups, or replication to a second array, or a design where the cluster degrades to read-only rather than failing.

The design questions worth answering in advance

If the array disappears for ten minutes, do the nodes hang or do they get errors? Test it, on the lab cluster, by blocking the storage network.
Is the storage timeout budget shorter than the cluster’s stop timeout?
Does the cluster do anything useful when every node loses storage simultaneously, or does it fence itself flat? A Filesystem monitor with a long timeout and on-fail=block for the shared-outage case is one deliberate answer; fencing everything is the accidental one.
What is the RTO of the client, including the NFSv4 grace period and the application’s own reconnect logic?
Where is the copy of the data that is not on this array?