Break/Fix: load average 340, every CPU idle, and nothing can be killed

Reported symptoms

• At 14:07 the monitoring system alerted on load average above 300 on db-node02. It has 16 cores.
• CPU utilisation is 0%. Memory is 40% used. Nothing is swapping.
• The existing SSH session still responds. ls /, top and dmesg all work. df hangs and never returns.
• New SSH sessions authenticate and then hang before the prompt.
• The on-call engineer tried kill -9 on the stuck processes. Nothing happened. The processes are still there.
• systemctl reboot was issued fifteen minutes ago. The host is still up.

Evidence provided

$ uptime
 14:22:41 up 61 days,  3:12,  2 users,  load average: 341.08, 288.44, 141.76

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0 214      0 9843200  22016 3812864    0    0     0     0  412  866  0  0 100  0  0
 0 214      0 9843200  22016 3812864    0    0     0     0  388  801  0  0 100  0  0
 0 214      0 9843200  22016 3812864    0    0     0     0  401  822  0  0 100  0  0

$ ps -eo state,pid,wchan:20,comm --no-headers | awk '$1=="D"' | head -5
D  1874 blk_mq_get_tag       postgres
D  1902 blk_mq_get_tag       postgres
D  2311 io_schedule          df
D  2455 io_schedule          rsyslogd
D  2790 io_schedule          sshd

$ ps -eo state --no-headers | grep -c '^D'
214

$ sudo multipath -ll mpatha
mpatha (3600a098038303735) dm-3 NETAPP,LUN C-Mode
size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 3:0:0:1 sdc 8:32 failed faulty running
| `- 3:0:1:1 sdd 8:48 failed faulty running
`-+- policy='service-time 0' prio=0 status=enabled
  |- 4:0:0:1 sde 8:64 failed faulty running
  `- 4:0:1:1 sdf 8:80 failed faulty running

$ sudo dmesg -T | grep -iE 'multipath|rport' | tail -6
[Tue Aug 11 14:06:12 2026] rport-3:0-1: blocked FC remote port time out: removing rport
[Tue Aug 11 14:06:12 2026] rport-4:0-1: blocked FC remote port time out: removing rport
[Tue Aug 11 14:06:13 2026] device-mapper: multipath: 253:3: Failing path 8:32.
[Tue Aug 11 14:06:13 2026] device-mapper: multipath: 253:3: Failing path 8:48.
[Tue Aug 11 14:06:14 2026] device-mapper: multipath: 253:3: Failing path 8:64.
[Tue Aug 11 14:06:14 2026] device-mapper: multipath: 253:3: Failing path 8:80.

$ grep -r no_path_retry /etc/multipath.conf /etc/multipath/conf.d/
/etc/multipath.conf:        no_path_retry           queue

$ sudo multipathd show paths | head -3
hcil     dev dev_t  pri dm_st  chk_st dev_st  next_check
3:0:0:1  sdc 8:32   0   failed faulty running orphan
3:0:1:1  sdd 8:48   0   failed faulty running orphan

Work the evidence before reading on

The load average is the misdirection, and it is a very effective one. Three things to reconcile before reading on:

1. vmstat shows r at 0 and b at 214. Nothing is runnable. 214 things are blocked. Work out which of those two columns the load average counts on Linux.
2. kill -9 did nothing. There is exactly one process state in which SIGKILL is not delivered. Name it, and name what puts a process there.
3. The last multipath message in dmesg is from 14:06:14, sixteen minutes ago. The host has been hung ever since and the kernel has said nothing in that time. Why would a subsystem that has completely failed stop logging?

Root cause

1. Linux load average counts uninterruptible sleep

On most Unix systems load average is a measure of CPU run-queue length. Linux is different: it counts tasks in state R (running or runnable) and tasks in state D (uninterruptible sleep, almost always waiting on I/O).

So a load average of 341 on a host with 0% CPU is not a contradiction. It is the signature of 214 processes blocked on storage, plus the queue growing as more arrive. Reading load as “CPU demand” is what sends the first thirty minutes of this incident into top, pidstat and a hunt for a runaway process that does not exist.

2. D state is why SIGKILL does nothing

A process in uninterruptible sleep is inside a kernel call that cannot be unwound safely — it holds locks, it has a request in flight, and the kernel will not let it return to userspace until the call completes. Signals are checked on the way back to userspace. A D-state process never gets there, so the signal is recorded as pending and delivered never.

There is no userspace action that frees a D-state process. Only the thing it is waiting for can.

3. queue_if_no_path means “wait forever”

The map’s features line is the answer:

size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

When device-mapper has no usable path it can do one of two things with an incoming request:

• Fail it — return EIO. The application sees an error, the filesystem probably goes read-only, and the host stays responsive.
• Queue it — hold the request in memory until a path returns.

no_path_retry queue selects the second, unconditionally and without a time limit. The reasoning is sound in its intended context: a fabric reconvergence or a controller failover takes seconds, and a clustered database that receives EIO during those seconds will abort transactions or fence itself. Queueing rides out the blip invisibly.

The failure mode is what happens when the blip is not a blip. There is no timeout, no upper bound, and no escape — and every new process that touches the mount joins the queue.

4. Why the kernel went quiet

The four Failing path messages are the kernel reporting each path going down. After that there is nothing to report: no I/O is being attempted against the paths, no errors are being returned, nothing is timing out. Silence in dmesg reads like “the problem stopped”, and it means the opposite. The subsystem is not failing repeatedly; it is waiting, permanently and quietly.

5. Why the host is half-alive

The root filesystem is on local disk. ls /, top and dmesg all work because none of them touches mpatha. What hangs is:

• df, because it stats every mounted filesystem including this one
• the database, because its data directory is there
• rsyslogd, because one of its output files is there
• new SSH logins, because PAM writes a session record and the shell reads a profile from a path on that mount

That last one is the operational sting. The host is up, reachable, answering ICMP and accepting TCP on port 22 — and you cannot get in.

Resolution

1. Confirm the shape of the problem before acting. vmstat 1 for r versus b, ps -eo state --no-headers | grep -c "^D" for the blocked count, and sudo multipath -ll for path state. Thirty seconds of this rules out every CPU and memory hypothesis
2. Find out what happened to the fabric. sudo dmesg -T | grep -iE "rport|fc_host|iscsi" gives the time the paths went away. Take that timestamp to the storage and network teams — this is almost always a change on the other side of the cable
3. Open a second way in now, before you need it. Console, IPMI or the hypervisor console does not depend on the mount. If new SSH logins already hang, this step is not optional
4. Try to restore the paths. This is the fix. Once one path returns, the queue drains on its own and every blocked process resumes with no data loss:
5. `` sudo multipathd show paths sudo multipath -r ``
6. If the fabric is coming back shortly, wait. Queued I/O costs memory and patience, not data. Riding it out is the outcome with the best data integrity, and it is the outcome queue_if_no_path was configured for
7. If the paths cannot be restored and the host must be freed, convert the queue into errors. Understand the cost first: in-flight writes are lost and the filesystem will need checking:
8. `` # Substitute your own values before running: MAP=mpatha sudo dmsetup message "$MAP" 0 fail_if_no_path ``
9. Expect a burst of errors and a read-only filesystem. Blocked processes wake with EIO, the load average collapses, and the host becomes responsive again. Stop the affected services and unmount the filesystem
10. When the paths return, restore queueing and re-scan:
11. `` sudo dmsetup message "$MAP" 0 queue_if_no_path sudo multipath -r sudo multipath -ll ``
12. Check the filesystem before remounting if you used fail_if_no_path. xfs_repair on an unmounted XFS, fsck on ext4. Do not skip this because the mount succeeded
13. Only then restart the application, after a manual write test has succeeded

Verification

1. Every path is back. sudo multipath -ll mpatha shows all four paths active ready running, and sudo multipathd show paths shows a non-zero priority on each
2. Nothing is still queued. sudo multipathd show maps format "%n %Q" reports zero queued I/O for the map
3. The blocked processes are gone. ps -eo state --no-headers | grep -c "^D" returns single digits, and the load average decays towards normal over the next few minutes rather than staying pinned
4. The device is usable, not merely present. sudo dd if=/dev/zero of=/srv/san/.iotest bs=1M count=16 oflag=direct completes in under a second and sudo rm /srv/san/.iotest succeeds. oflag=direct matters — a buffered write can be acknowledged by the page cache without ever reaching the device
5. **df returns.** The command that hung is the cheapest end-to-end proof that it no longer does
6. Redundancy is real, not assumed. Fail one path deliberately at the fabric or with sudo multipathd fail path sdc, confirm I/O continues on the survivors, then restore it with sudo multipathd reinstate path sdc. A map with four paths that all traverse one switch is a single path wearing a disguise
7. The filesystem is clean if fail_if_no_path was used — the repair tool reports no outstanding damage
8. Path-count alerting fires. Drop one path and confirm monitoring notices. The alert that matters is the one at three paths, not the one at zero

Prevention

• Alert on path count, not on path loss. Losing two of four paths is the actionable event; losing four is an outage you are now inside. A check that compares current active paths against the expected count per map catches fabric problems while they are still redundant.
• Set no_path_retry deliberately, per volume, and document why. An inherited queue on a mount whose application would have been perfectly happy with EIO is how an hour-long fabric change becomes a hung host.
• Keep the root filesystem, /var/log, /home and anything the login path touches off multipathed SAN storage. The difference between “the database is down” and “we cannot log in to the host that runs the database” is entirely down to this.
• Require SAN fabric changes to enumerate every attached host, and treat them as changes to those hosts. Nobody on the Linux side knew a switch was being reloaded.
• Test path failure on a schedule. multipathd fail path and multipathd reinstate path make this a routine, safe exercise on a non-production volume, and it is the only way to find out that all four of your paths go through one switch.
• Watch memory during a queueing event. Queued I/O is held in kernel memory, and a long outage on a busy volume can push the host towards OOM on top of everything else.