Shared storage failure paths - reading a host whose storage has gone away · Linux

Shared storage has a property no local disk has: it can vanish while the host is otherwise perfectly healthy. The CPU is idle, memory is fine, the network is up, and every process that touched the array is in D state where no signal reaches it.

It also has a property that makes the incident harder than it needs to be. The obvious diagnostics hang. df hangs. ls on the mount point hangs. Tab-completing a path in your own shell hangs, and now you have lost the session you were investigating from. Knowing which commands are safe is not a detail here; it is the first skill.

Four shapes, and they need different responses

Shape	Host symptom	Storage still there?	Fix
Path loss, paths remaining	Nothing visible	Yes	Restore the path; no filesystem work
All paths lost	`D`-state processes, or sudden I/O errors	No	Restore paths, then check the filesystem
Filesystem shut down	`EIO` on everything, mount still listed	Yes	Unmount and remount; paths are not the problem
NFS server gone	`D` state, `df` hangs, no `EIO` ever	No	Restore the server; do not force-unmount

The middle two are the ones people confuse, and confusing them wastes the outage: hunting for a fabric problem when the fabric is fine and the filesystem shut itself down, or remounting a filesystem while the paths are still flapping.

Diagnostics that do not hang

The rule is simple. Anything that calls statfs() or stat() on the affected mount will block. Anything that reads /proc/self/mountinfo, /sys or /proc/<pid> will not.

Read-only / Safemount table without touching the filesystems— findmnt reads /proc/self/mountinfo, which the kernel maintains in memory. It never issues I/O to the filesystems it lists, so it returns instantly even when every one of them is unreachable. df is the opposite: it statfs()s each mount to get free space, and statfs on an unreachable NFS mount blocks in the same uninterruptible way an application does.

$ findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS -t nfs4,xfs,ext4

TARGET     SOURCE                        FSTYPE OPTIONS
/          /dev/mapper/vg0-root          ext4   rw,relatime
/srv/data  /dev/mapper/mpatha            xfs    rw,noatime
/srv/share nfs01.example.com:/export/dat nfs4   rw,_netdev,hard

Illustrative output

If you must use df, scope it to one filesystem you know is healthy (df -h /) or exclude the suspect types (df -h -x nfs -x nfs4 -x cifs). Running bare df during a filer outage is how the second engineer loses their session too.

Read-only / Safewho is blocked, and in what— D is uninterruptible sleep - the process is inside a kernel call that will not return and cannot be signalled. The WCHAN is the diagnosis: io_schedule means blocked on block-device I/O, so a SAN or iSCSI path. rpc_wait_bit_killable means blocked in the NFS client waiting for an RPC reply, so a filer. Two different incidents, told apart by one column.

$ ps -eo state,pid,ppid,wchan:28,comm | awk 'NR==1 || $1=="D"'

S   PID  PPID WCHAN                        COMMAND
D  2841  2837 io_schedule                  postgres
D  2903  2837 io_schedule                  postgres
D  4117     1 rpc_wait_bit_killable        rsync

Illustrative output

For the block-device case, walk from the process to the device:

# what the process was doing, from the kernel side (needs root)
sudo cat /proc/2841/stack

# what files it has open, without touching them
sudo ls -l /proc/2841/fd

# the SCSI device state for every path: running, offline or blocked
grep . /sys/block/sd*/device/state

# iSCSI session state, if this is iSCSI
grep . /sys/class/iscsi_session/session*/state

/sys/block/sdb/device/state is the fastest single answer for a SAN incident. running is healthy; offline means the SCSI layer gave up on the device; blocked means the transport layer has it quarantined while it waits for the fabric.

Shape 1 and 2: paths

Read-only / Safedegraded, not down— Two of four paths are failed faulty. I/O is continuing over the surviving group and no application has noticed anything. This is shape 1: nothing is broken today, and the host is now one HBA away from shape 2. It is invisible to any monitoring that checks whether the filesystem is mounted, because it is mounted and it works.

$ multipath -ll mpatha

mpatha (3600508b400105e210000900000490000) dm-3 NETAPP,LUN C-Mode
size=10T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 6:0:0:1 sdb 8:16 failed faulty running
| `- 6:0:1:1 sdd 8:48 failed faulty running
`-+- policy='service-time 0' prio=10 status=active
|- 7:0:0:1 sdf 8:80 active ready running
`- 7:0:1:1 sdh 8:112 active ready running

Illustrative output

Monitor the path count, not the mount

A check that asks “is /srv/data mounted and writable” passes for the entire life of a degraded LUN and fails only when the last path drops - at which point it is reporting an outage, not warning of one.

Ask multipathd directly for a machine-readable count:

sudo multipathd show maps status
sudo multipathd show paths format "%m %d %t %o"

show maps status prints the number of paths and the number active per map. Alert on active-path count below the configured total, not on zero. The window between “one path failed” and “the last path failed” is usually days, and it is the entire opportunity to fix it without an incident.

When the last path goes, the behaviour is whatever no_path_retry was set to - queue forever and hang, or fail and return EIO. That choice, and why a finite retry count is usually right, is covered in the multipath lesson; what matters here is recognising which one you are looking at.

Read-only / Safethe kernel timeline— Read it as a sequence. Paths fail at 03:14. The hung-task detector fires at 03:16 because kernel.hung_task_timeout_secs is 120 by default - that message is a report that something has been stuck for two minutes, not a new event. The I/O error on dm-3 arrives once the retry budget is exhausted. The two-minute gap between the path failure and the first hung-task warning is why the monitoring alert and the user complaint arrive out of order.

$ sudo dmesg -T | grep -E 'blocked for more than|rejecting I/O|Device offlined|failing path|I/O error, dev'

[Tue Aug 11 03:14:22 2026] device-mapper: multipath: 253:3: Failing path 8:16.
[Tue Aug 11 03:14:22 2026] device-mapper: multipath: 253:3: Failing path 8:48.
[Tue Aug 11 03:16:29 2026] INFO: task postgres:2841 blocked for more than 120 seconds.
[Tue Aug 11 03:16:29 2026] blk_update_request: I/O error, dev dm-3, sector 4194304 op 0x1:(WRITE)

Illustrative output

Shape 3: the filesystem shut itself down

This one is different in kind. The LUN is present, every path is active ready running, and the filesystem still returns errors for everything. XFS detected a metadata inconsistency or an unrecoverable write error and shut itself down deliberately.

XFS (dm-3): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x400 len 8 error 5
XFS (dm-3): Log I/O Error Detected. Shutting down filesystem
XFS (dm-3): Please unmount the filesystem and rectify the problem(s)

ext4’s equivalent is quieter and is the reason errors=remount-ro exists:

EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
EXT4-fs (dm-3): Remounting filesystem read-only

A shut-down filesystem is not fixed by fixing the paths

The instinct is to keep working on the fabric, because the filesystem is unusable and the last thing that happened was a path event. But the message says what to do: unmount and rectify. A shut-down XFS stays shut down for the life of the mount. No amount of path recovery brings it back, because the shutdown is a state of the mounted filesystem, not of the device.

The sequence is fixed and the order is not negotiable:

Confirm the paths are stable first. multipath -ll with every path active ready running, held for several minutes. Unmounting and remounting onto a flapping LUN reproduces the shutdown and adds a second incident to the first.
Stop everything that has the filesystem open. sudo fuser -vm /srv/data names them.
Unmount. sudo umount /srv/data.
Repair offline, from a backup position. sudo xfs_repair -n /dev/mapper/mpatha is the read-only dry run and tells you what would be changed. Only then consider running it without -n, and only after the LUN has been snapshotted on the array or imaged - xfs_repair can and does discard metadata it cannot reconcile, and files land in lost+found with numeric names.
Remount and verify the data, not just the mount.

Never run xfs_repair or fsck on a mounted filesystem, and never on a device whose paths are still unstable. Both turn a recoverable incident into a restore.

Shape 4: the NFS server is gone

A hard NFS mount never returns an error. That is the contract: retry until the server comes back, so that an application which does not check its write return codes cannot silently lose data. The cost is that every process touching the mount goes into D state and stays there.

Read-only / Safeconfirm hard versus soft— The effective options come from the kernel, not from fstab - somebody may have remounted with different ones. hard here means the D-state processes will never receive an error and will resume when the server returns. /proc/self/mountstats is safe to read and carries per-operation RPC counters and retransmission counts, which distinguish 'server is down' from 'server is slow'.

$ grep -c . /proc/self/mountstats; findmnt -t nfs4 -o TARGET,SOURCE,OPTIONS

412
TARGET     SOURCE                            OPTIONS
/srv/share nfs01.example.com:/export/data    rw,relatime,vers=4.1,hard,proto=tcp,timeo=600,retrans=2

Illustrative output

umount -l makes the problem invisible, not smaller

umount -f on a hung NFS mount often fails outright. umount -l

lazy - always appears to succeed, and that is the trap.

Lazy unmount detaches the mount from the filesystem namespace so that new lookups no longer find it. It does not terminate the outstanding I/O and it does not release the processes already blocked in it. They stay in D state, still holding references, on a mount that no longer appears in df or findmnt.

You have now removed your own evidence. The mount is gone from every listing, the processes are still stuck, and the next engineer sees blocked processes with no visible mount to explain them. Worse, mounting the same export at the same path now succeeds, so the host carries two mounts of the same export - one live, one draining - and files written through each go to different places in ways that are extremely hard to reason about.

Leave the mount alone. Fix the server. hard mounts recover on their own when the server returns, and the D-state processes resume as if nothing happened - which is exactly what hard is for.

What if the host has to be rebooted while storage is hung?

systemctl reboot will not complete. Shutdown unmounts filesystems, unmounting a hung mount blocks, and the shutdown job waits DefaultTimeoutStopSec per mount before giving up. With several affected mounts that is minutes of nothing happening, and on a host with a hung root-adjacent filesystem it may not complete at all.

The escalation ladder, in order:

Wait out the timeouts. Check with systemctl show -p DefaultTimeoutStopUSec so you know how long “not hung, just slow” looks like.
sudo systemctl reboot --force - man 1 systemctl describes this as skipping the shutdown of running services while still killing all processes and unmounting or remounting read-only every filesystem, then rebooting.
sudo systemctl reboot --force --force - executed by systemctl itself without contacting the manager, with no process termination and no unmount at all. The man page states plainly that this may result in data loss, and it works even when the system manager has crashed.
Sysrq, from the console only, when even that does not return.

Every step past the first skips filesystem flushing, so everything in the page cache that has not reached the array is lost. On a healthy local root that is usually a few seconds of writes. It is still a data-loss decision, not a reset button, and it belongs to whoever owns the service - not to the person who happens to be at the console.

Why D state cannot be killed

kill -9 delivers SIGKILL, and signal delivery happens when a task returns from the kernel to userspace. A task in uninterruptible sleep is inside a kernel call that has declared it will not check for pending signals - because it holds locks or is mid-way through a state change that cannot be abandoned half-done.

So the signal is recorded and delivered when the call returns. If the call never returns, the process never dies, and kill -9 genuinely does nothing.

NFS has a partial answer: the killable variants used by modern clients allow SIGKILL specifically, which is why the earlier ps output showed rpc_wait_bit_killable rather than rpc_wait_bit. Block I/O has no such escape. A process blocked in io_schedule on a dead multipath map with no_path_retry queue waits for the array, forever, and the only levers are restoring the path or resetting the host:

# release queued I/O with errors instead of waiting - a deliberate
# choice to convert a hang into application errors
sudo multipathd disablequeueing map mpatha

That one is not a diagnostic. It fails every queued and future write on the map, so the applications that were hanging start receiving EIO and the filesystem may shut down or remount read-only. Use it when you have decided the array is not coming back soon and errors are better than a frozen host - and say so out loud before you run it.

Recovery order

Identify the shape before touching anything. findmnt for the mounts, ps for D-state processes and their WCHAN, dmesg -T for the timeline, multipath -ll for the paths. All read-only, none of them hang.
Restore the transport. Fabric, switch, NIC, iSCSI session, filer. Nothing above this layer can be fixed while it is broken.
Wait for the paths to be stable, not merely present. Several minutes of every path active ready running, confirmed twice.
Only then address the filesystem. A shut-down XFS or a read-only ext4 needs an unmount and remount; a healthy filesystem that was merely queueing needs nothing at all.
Verify the data, not the mount. Application-level checks - row counts, checksums, a read of the most recently written file - because an I/O error that reached the application may have left a partial write behind.
Write down the path-count alert you did not have. Every one of these incidents was visible as a degraded path days earlier.

Four shapes, and they need different responses

Diagnostics that do not hang

Shape 1 and 2: paths

Shape 3: the filesystem shut itself down

Shape 4: the NFS server is gone

Recovery order

Knowledge check