Storage and ZFS troubleshooting · Proxmox VE

ZFS faults reach you dressed as Proxmox faults. The storage shows inactive, a guest will not start, a backup fails, a VM is paused. The Proxmox layer is reporting accurately and has nothing more to tell you; everything useful is one layer down.

Four faults cover most of what you will meet, and each has a different first command.

Fault one: the pool will not import

pvesm status shows the ZFS storage inactive. zpool list shows nothing.

Read-only / Safewhat does ZFS see, and what could it import— Read-only. zpool import with no arguments scans for importable pools and reports why each one is or is not importable — the state line and the action line are the whole diagnosis. Scanning by-id paths matters because device names are not stable across reboots or controller changes.

zpool list

zpool import

zpool import -d /dev/disk/by-id

ls -l /dev/disk/by-id | grep -v part

Read-only / Safe

$ zpool import

   pool: tank
   id: 10913148573287938276
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
 see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

tank        ONLINE
  mirror-0  ONLINE
    ata-SAMSUNG_MZ7L3960_S6ABC0000000001  ONLINE
    ata-SAMSUNG_MZ7L3960_S6ABC0000000002  ONLINE

The common causes and their answers:

Symptom in `zpool import`	Cause	Action
Pool not listed at all	Devices not present, or names not scanned	`zpool import -d /dev/disk/by-id`; check `lsblk`, cabling, HBA
“last accessed by another system”	Hostid mismatch — reinstall, disk move, or another live host	`-f`, after confirming no other system has it
`FAULTED`, “insufficient replicas”	Too many devices missing for the redundancy level	Find the missing devices; do not force
“may be in use from other system” with a live peer	Genuinely dual-attached	Stop. Forcing this is how you destroy a pool
Imports but hangs	Damaged pool, or a device that responds slowly	`-o readonly=on`, and `-N` to skip mounting

What -f actually overrides

zpool import -f bypasses the check that a pool was last accessed by a different system. That check exists for one reason: two hosts importing the same pool at the same time destroys it. ZFS has no cluster coordination and no mechanism to detect a second importer once both are writing.

Before typing -f, answer one question: could any other system currently have this pool imported? If the disks are local to a rebuilt node, the answer is no and -f is routine. If they are on a shared enclosure, an external JBOD, or a SAN, confirm the other host is powered off — not shut down cleanly, powered off — before proceeding.

For rescue work, import read-only first. zpool import -o readonly=on -N tank gives you a look without mounting anything and without writing, which is the right first move on any pool whose health you are unsure of.

Fault two: the resilver has stopped moving

zpool status shows a resilver in progress with an estimate that grows every time you look.

Read-only / Safefind the device that is holding up the resilver— Read-only. zpool status -v shows per-device error counters; zpool iostat -v shows per-device throughput and latency. The device with error counts climbing, or with latency an order of magnitude above its peers, is the one to deal with.

zpool status -v

zpool iostat -v 5 3

zpool events -v | tail -40

journalctl -k --since "-6h" | grep -iE 'ata|scsi|nvme|I/O error|medium error' | tail -40

Read-only / Safe

$ zpool status -v tank

  pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Aug 11 22:04:11 2026
1.42T scanned at 41.2M/s, 388G issued at 11.3M/s, 6.71T total
388G resilvered, 5.64% done, 6 days 14:22:19 to go
config:

NAME            STATE     READ WRITE CKSUM
tank            DEGRADED     0     0     0
  raidz2-0      DEGRADED     0     0     0
    ata-...001  ONLINE       0     0     0
    ata-...002  ONLINE       0     0    47
    ata-...003  ONLINE       0     0     0
    replacing-3 DEGRADED     0     0     0
      old       OFFLINE      0     0     0
      ata-...09 ONLINE       0     0     0

The slow resilver is a second failing disk

An eight-day resilver estimate on a pool that resilvered in nine hours last time is not a ZFS performance problem. It is almost always one of two things, and the first diagnosis — “the array is just busy” — is usually wrong.

A second marginal device. Non-zero CKSUM or READ counts on a device that is not the one being replaced. Every checksum failure means a reconstruction, and reconstruction during a resilver is expensive because the resilver is already reading everything. Worse, a RAIDZ2 pool resilvering one device with a second device returning bad data has spent its redundancy: a third fault is data loss.

Client load. A resilver yields to production I/O by design, so a pool that is saturated by guests will resilver very slowly. zpool iostat -v distinguishes the two: high throughput to all devices means load, low throughput with rising error counts on one device means hardware.

What to do, in order:

Identify the marginal device precisely — zpool status -v, smartctl -a, and the kernel log for ATA/SCSI/NVMe errors.
Do not offline it while redundancy is already reduced. In the example above, offlining device 002 during the replacement of device 3 removes the last redundancy in a RAIDZ2 vdev.
Reduce client load on the pool if you can — pause backups, migrate the heaviest guests — so the resilver finishes and redundancy is restored.
Then replace the second device, one at a time, waiting for each resilver to complete.
Verify you have a current, restorable backup before any of this. A degraded pool with a second marginal disk is the state in which people lose data.

Fault three: the pool is full and nothing accounts for it

zfs list shows the pool nearly full. The guests on it add up to far less.

Read-only / Safeaccount for every byte— Read-only. zfs list -o space breaks usage into the categories that explain a full pool: data, snapshots, children and reservations. This is the command that ends the argument about where the space went.

zpool list -o name,size,alloc,free,cap,frag,health

zfs list -o space -r rpool

zfs list -t snapshot -o name,used,refer -s used | tail -30

zfs get -r -o name,property,value refreservation,reservation,used,usedbysnapshots rpool | head -40

Read-only / Safe

$ zfs list -o space -r rpool/data

NAME                        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool/data                  212G  3.42T         0      0B             0B     3.42T
rpool/data/vm-141-disk-0    212G  1.02T      640G    404G             0B         0B
rpool/data/vm-142-disk-0    712G  1.98T         0    478G          1.5T         0B
rpool/data/subvol-205-disk-0 212G   412G      398G   14.1G             0B         0B

The four things that consume a ZFS pool without being obvious:

Snapshots. USEDSNAP is space held only by snapshots. A guest that deletes 100 GB frees nothing while a snapshot references it. This is the most common answer by a wide margin.

Refreservations on zvols. A thick-provisioned zvol reserves its full size whether or not it is written. USEDREFRESERV shows it. This is space that exists to guarantee the guest can always write, and it is a deliberate choice — but it is often an accidental one, inherited from a storage created without thin provisioning.

Fragmentation and the 80% rule. ZFS performance degrades as a pool fills, because finding contiguous free space gets harder. zpool list shows FRAG. A pool above about 80% is one to plan capacity for, not one to keep filling.

Reservations on datasets, which do the same job as refreservations at the dataset level and are rarer but harder to spot.

A pool at 100% may not let you delete anything

ZFS is copy-on-write. Deleting a file is a write: metadata has to be updated, and the update is written to a new location before the old one is released. A pool with genuinely zero free space can therefore fail to accept a deletion, which is the most counter-intuitive failure in ZFS operations.

Free space in this order:

Destroy a snapshot. zfs destroy pool/dataset@snapname releases space without needing much, and it is the reliable escape. Pick the oldest snapshot whose loss you can accept, and check zfs list -t snapshot -o name,used -s used for which ones are actually holding data.
Remove a refreservation. zfs set refreservation=none pool/vm-142-disk-0 returns reserved-but-unwritten space immediately, at the cost of the guarantee it was providing. Understand what you are giving up before doing it in an emergency and forgetting afterwards.
Delete a clone or an unused zvol that no guest references — the unused volumes from the storage-migration lesson are a good place to look.
Add capacity. On a mirror, attaching a new mirror vdev is possible online; on RAIDZ it is a bigger conversation.

What does not work is deleting files inside a guest, which frees nothing at the pool level unless discard is plumbed all the way through — and even then, not while a snapshot references the blocks.

Fault four: guests are paused and the storage looks fine

A pool that hits an unrecoverable I/O error can enter a suspended state, in which all I/O to it blocks. Guests do not error; they stop.

Read-only / Safeis the pool suspended, and what are the guests doing— Read-only. A suspended pool reports its state in zpool status; QEMU guests whose disks have blocked report io-error. Check both, because the Proxmox-level symptom and the ZFS-level cause look nothing alike.

zpool status

zpool get failmode rpool

qm list

qm status 141

journalctl -k --since "-2h" | grep -iE 'zfs|zio|I/O failure|pool.*suspended' | tail -30

failmode, and why the default hangs

failmode controls what a pool does when it can no longer service I/O:

wait (the default) — block all I/O until the fault is repaired. Nothing errors, nothing corrupts, and everything hangs.
continue — return EIO to new writes while allowing reads that can be satisfied.
panic — panic the kernel.

The default is deliberate and conservative: hanging is recoverable, and errors propagated into guests are not. A guest that receives an I/O error typically remounts read-only or corrupts; a guest that is paused resumes cleanly when the underlying fault is fixed.

The operational consequence is that a suspended pool looks like a hung node. qm list may take a long time to return, tasks stack up, and the GUI becomes unresponsive for that node — none of which says “storage”. zpool status is the command that tells you, and it should be early in the checklist for any node that has gone unresponsive without an obvious cause.

Recovery is to fix the underlying device fault — reseat, replace, restore the path — and then zpool clear <pool>, which resumes I/O. Guests in io-error resume with qm resume <vmid> once the pool is healthy.

Under the hood

Why does ZFS report a pool as healthy in zpool list while individual devices are returning bad data?

Because the pool’s health and a device’s error counters answer different questions. zpool list reports whether the pool has enough redundancy to serve data; the per-device counters in zpool status -v report how much work it has had to do to achieve that.

Every read in ZFS is checksummed. When a checksum fails, ZFS reconstructs the block from redundancy, returns correct data to the caller, writes the corrected block back, and increments the CKSUM counter for the offending device. The application sees a successful read. The pool stays ONLINE. The only evidence is a counter.

That is the property this whole lesson depends on and it cuts both ways. It is why ZFS estates do not suffer the silent corruption that unchecked arrays do — and it is why a disk can be failing for weeks with no user-visible symptom at all, until it fails enough to be marked DEGRADED, or until a resilver has to reconstruct through it and takes eight days.

Two habits follow directly:

Scrub on a schedule and read the result. A scrub is what forces every block to be read and checksummed, which converts latent errors into counters you can see. A pool that is never scrubbed has no idea how healthy it is.

Alert on non-zero READ, WRITE and CKSUM counters, not only on pool state. By the time the state changes, the pool has already lost redundancy. The counters are the early warning, and zpool clear resets them — which is useful for establishing a new baseline and dangerous if used to make a warning go away.

First commands, by symptom

Symptom	First command
PVE storage inactive, ZFS type	`zpool list` then `zpool import`
Node unresponsive, no obvious cause	`zpool status` — check for a suspended pool
Guest paused, `io-error`	`zpool status`, then `zpool get failmode`
Backup or write fails, “no space”	`zfs list -o space -r <pool>`
Slow guests on a healthy-looking pool	`zpool status -v` for CKSUM counts, `zpool list` for `FRAG` and `CAP`
Resilver estimate growing	`zpool iostat -v` and `zpool status -v`
Pool degraded	`zpool status -v`, then `smartctl -a` on the named device

And three things to have configured before any of the above:

Scheduled scrubs, with the result reported somewhere a human sees it.
Alerts on device error counters and on pool capacity at 80%, not just at 95%.
A snapshot retention policy that somebody owns. Unbounded snapshots are the single most common cause of a full ZFS pool in a Proxmox estate.

What happens if you replace the wrong disk?

It is more survivable than it feels, and the recovery depends entirely on what “replace” meant.

If you pulled a healthy disk from a redundant vdev, the pool goes DEGRADED and keeps serving. Putting the same disk back usually results in ZFS recognising it and resilvering only what changed, which is fast. Do that immediately and do not start a replacement onto a new device in the meantime.

If you already ran zpool replace onto a new device, the pool is now resilvering onto the new disk while the wrong old disk is detached. The data is intact — you removed a redundant copy, not the data — but you are running with reduced redundancy until the resilver completes, and the disk that actually needed replacing is still in the pool. Let the resilver finish, then replace the genuinely faulty device.

If the vdev had no redundancy — a single-disk vdev, or a mirror already degraded — you have lost the pool and are restoring from backup. This is the scenario that makes the preparation worth it.

The preparation, which takes five minutes and prevents all of this:

Refer to devices by /dev/disk/by-id paths, never by /dev/sdX, which changes across reboots and controller enumeration.
Before pulling anything, map the ZFS device name to a physical slot and record it. ledctl or the enclosure’s own locate function, where available, is the reliable way; a written map of serial numbers to slots is the fallback.
Confirm with the serial number on the disk itself, physically, before pulling. The serial is in the by-id name.
Have the replacement runbook open, and confirm a current backup exists before starting.

Key takeaways

zpool import with no arguments is the diagnosis for a pool that will not import: it prints the state, the reason and the recommended action.
Scan by -d /dev/disk/by-id. Device names are not stable and a pool that “disappeared” has usually just been renamed by the kernel.
-f overrides the check that stops two hosts importing the same pool. Confirm no other system has it — powered off, not merely shut down — before using it. Import -o readonly=on -N first when in doubt.
A resilver whose estimate keeps growing usually means a second marginal device returning checksum errors, or heavy client load. zpool iostat -v distinguishes them. Do not offline a second device while redundancy is already reduced.
zfs list -o space -r accounts for a full pool: USEDSNAP for snapshots, USEDREFRESERV for thick zvol reservations, USEDCHILD for descendants.
A pool at 100% may be unable to accept a delete, because deleting is a write. Destroy a snapshot or drop a refreservation instead.
A suspended pool with the default failmode=wait blocks all I/O, which presents as an unresponsive node rather than as a storage error. zpool status belongs early in the checklist for any hung node; recovery is to fix the device and then zpool clear.
Checksum errors are invisible to applications because ZFS repairs them transparently. Scrub on a schedule and alert on the counters, not only on pool state.

Knowledge check

Why this matters

ZFS is unusually honest. It tells you about problems other storage layers hide, it repairs what it can, and it records what it repaired.

The catch is that the honesty is in counters rather than in alarms. A number in zpool status -v climbing from 0 to 47 over six weeks is the most valuable signal in the whole storage stack, and it produces no notification, no ticket and no GUI badge. It is available to anybody who looks, and almost nobody looks until the pool is degraded.

Every fault in this lesson has an earlier, cheaper version of itself: the pool that would not import was a device path assumption; the eight-day resilver was a checksum counter; the full pool was a snapshot policy nobody owned; the hung node was a disk that had been logging ATA errors for a fortnight.

Configure the scrub, alert on the counters, own the retention policy. Then this lesson is about incidents that mostly do not happen to you.