high riskdata loss risk~120 min

Expand a ZFS pool safely

1 · Prerequisites

Confirm every item is in place before any state change.

The pool is healthy: zpool status shows ONLINE with no errors, no degraded vdev and no resilver in progress
A recent scrub completed without errors, because expansion rewrites metadata and a pool with latent corruption is the wrong pool to grow
A current, verified backup exists of everything on the pool - expansion is a low-probability, high-consequence operation
The new disks are physically installed, visible to the operating system, and identified by stable path rather than by kernel device name
The existing pool layout is understood: how many vdevs, of what type, and what redundancy each one has
The chosen expansion method is decided in advance and written down, because the commands for the three methods differ by one word

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· zpool status POOL shows all vdevs ONLINE and no resilver or scrub running
· zpool list POOL records the current size, allocated and free values for later comparison
· zpool get autoexpand POOL shows whether autoexpand is on, because disk replacement does nothing visible without it
· ls -l /dev/disk/by-id/ identifies every new disk by a stable identifier
· smartctl -H reports PASSED on each new disk before it is added to a pool
· zpool status POOL shows the vdev names that any add or attach command will reference
· zpool history POOL shows what has been done to this pool before, which often explains an unexpected layout

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the current layout and capacity: zpool status, zpool list and zfs list output saved to a file
2Confirm the pool is healthy and no scrub or resilver is running
3Identify every new disk by /dev/disk/by-id path and confirm its health with SMART
4Choose the method: replace disks in place, add a matching vdev, or expand a raidz vdev
5Run the chosen command with the dry-run flag first, and read the layout it reports back
6Confirm the reported layout is exactly what was intended before running it for real
7Execute the expansion
8For disk replacement: wait for each resilver to complete before starting the next disk
9For raidz expansion: allow the reflow to complete, and expect it to take a long time on a full pool
10Verify the new capacity is visible to ZFS and to Proxmox storage
11Run a scrub after the expansion settles and confirm it completes with zero errors
12Update capacity monitoring thresholds and documentation to the new size

4 · Verification

Confirm the procedure actually fixed the problem.

✓zpool list POOL shows a larger SIZE than the value recorded before, and the increase matches the expected amount
✓zpool status POOL shows every vdev ONLINE with the expected redundancy, and the layout matches the intended design rather than merely being larger
✓No vdev in the pool is a single disk unless the pool was deliberately designed with no redundancy
✓zfs list shows increased AVAIL on the datasets that matter
✓pvesm status shows the increased size for the Proxmox storage backed by this pool
✓A scrub started after the expansion completes with zero repaired blocks and zero errors
✓zpool status shows no resilver or reflow still in progress before the change is declared complete

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A top-level vdev added with zpool add cannot be removed from a pool that contains raidz vdevs. There is no undo
↶Device removal exists only for mirror and single-disk top-level vdevs in a pool with no raidz vdev, and it must be planned rather than attempted as a recovery
↶The rollback for a wrong zpool add is: destroy the pool and restore from backup. That is the honest answer and it is why the dry run is mandatory
↶Disk-by-disk replacement is reversible until the last disk is replaced - the old disk can be replaced back in, at the cost of another resilver
↶raidz expansion cannot be reversed; the vdev keeps its new width permanently
↶If a resilver fails or the new disk errors during it, the pool returns to its prior state once that disk is detached - the pool was never dependent on it

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the storage owner before any zpool add, because it is the irreversible one
· Escalate if the dry run reports a layout that differs in any way from what was intended - do not adjust and retry without a second pair of eyes
· Escalate if a resilver reports read or checksum errors on an existing disk, because expansion has just exposed a second fault
· Escalate to the hardware owner if a new disk fails SMART or errors during resilver
· Escalate if the pool is above 80 percent capacity before the expansion begins, because ZFS performance and resilver time both degrade sharply there and the window will be longer than planned

Verified against Proxmox VE 9.2.4 with OpenZFS 2.3.

Growing a ZFS pool is a five-second command with a very long tail. The commands are simple, the concepts are simple, and the failure mode is that zpool add and zpool attach differ by one word while doing opposite things - one widens the pool with a new vdev, the other adds a device to an existing vdev.

Type the wrong one against a mirrored pool and you have permanently converted a redundant pool into one where the loss of a single new disk takes the whole thing with it. ZFS will do it without complaint, and there is no supported way back.

Hence the shape of this runbook: the dry run is not optional, and the output of the dry run is the thing you actually check.

When to use this runbook

A pool is approaching its capacity threshold.
Larger disks are available and the chassis has no free bays.
New disks are available and there are free bays.
A raidz vdev needs another disk (OpenZFS 2.3 and later).

The three methods, and when each applies

Method	Command	Requires	Reversible?
Replace disks in place	`zpool replace` per disk, `autoexpand=on`	No free bays; larger disks	Yes, until the last disk
Add a vdev	`zpool add`	Free bays; enough disks for a matching vdev	No
Expand a raidz	`zpool attach POOL raidzN-X DISK`	OpenZFS 2.3+; one free bay	No

The choice is usually made for you by the chassis. What is never made for you is whether the new vdev matches the existing ones - and that is the decision that determines whether your pool stays sane.

Step 1: Record the current state

Read-only / Safewhat the pool looks like now

POOL=rpool
OUT="/root/zfs-expand-$POOL-$(date +%Y%m%d-%H%M).txt"

{
zpool status "$POOL"
echo '== list'
zpool list -v "$POOL"
echo '== properties'
zpool get all "$POOL" | grep -E 'autoexpand|ashift|capacity|health|fragmentation'
echo '== datasets'
zfs list -o name,used,avail,refer -r "$POOL" | head -30
echo '== history'
zpool history "$POOL" | tail -20
} | tee "$OUT"

zpool list -v is the important one: it shows the pool broken down by vdev, which is the structure any expansion must respect.

Read-only / Safehealthy, and nothing already running

POOL=rpool

zpool status -x
zpool status "$POOL" | grep -E 'scan:|state:|resilver|errors:'

Step 2: Identify the new disks properly

Read-only / Safestable identifiers, not sdX

ls -l /dev/disk/by-id/ | grep -vE 'part[0-9]' | awk '{print $9, $11}'
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,MOUNTPOINT

Read-only / SafeSMART before trusting a disk with data

DISK=/dev/disk/by-id/nvme-SAMSUNG_MZQL2_S12345

smartctl -H "$DISK"
smartctl -A "$DISK" | grep -iE 'reallocated|pending|wear|percentage_used|power_on'

Step 3, method A: Replace disks in place

For a pool with no free bays. Replace one disk at a time with a larger one; when every disk in a vdev has been replaced, the vdev grows.

Configuration changeautoexpand must be on BEFORE the last replacement

POOL=tank

zpool get autoexpand "$POOL"
zpool set autoexpand=on "$POOL"
zpool get autoexpand "$POOL"

Destructiveone disk, then wait for the resilver

POOL=tank
OLD=/dev/disk/by-id/ata-OLD_DISK_SERIAL1
NEW=/dev/disk/by-id/ata-NEW_DISK_SERIAL1

zpool replace "$POOL" "$OLD" "$NEW"

# Watch it. Do not start the next one until this finishes.
watch -n 30 zpool status "$POOL"

Read-only / Safeafter the last disk in the vdev

POOL=tank

zpool status "$POOL" | grep -E 'scan:|resilver'
zpool list "$POOL"

# If size did not grow despite autoexpand, nudge each device:
DEV=/dev/disk/by-id/ata-NEW_DISK_SERIAL1
zpool online -e "$POOL" "$DEV"
zpool list "$POOL"

The pool grows only when every disk in a vdev is larger. Replacing three of four disks in a raidz1 gains nothing until the fourth is done - that is expected, not a fault.

Step 3, method B: Add a vdev

For a pool with free bays. This is the fast method and the dangerous one.

Read-only / Safethe dry run is the whole safety mechanism

POOL=tank
D1=/dev/disk/by-id/ata-NEW1
D2=/dev/disk/by-id/ata-NEW2
D3=/dev/disk/by-id/ata-NEW3

zpool add -n "$POOL" raidz1 "$D1" "$D2" "$D3"

-n prints the layout the pool would have and changes nothing. Read that output line by line and compare it against zpool list -v from Step

Specifically:

Is the new vdev the same type as the existing ones? A mirror pool should get a mirror; a raidz2 pool should get a raidz2.
Is it the same width? A raidz1 of three next to a raidz1 of six is legal and gives uneven performance and uneven failure characteristics.
Is anything listed as a bare disk? A top-level single-disk vdev in a redundant pool is the failure this whole step exists to prevent.

Destructiveonly after the dry run showed exactly what you intended

POOL=tank
D1=/dev/disk/by-id/ata-NEW1
D2=/dev/disk/by-id/ata-NEW2
D3=/dev/disk/by-id/ata-NEW3

zpool add "$POOL" raidz1 "$D1" "$D2" "$D3"
zpool status "$POOL"
zpool list -v "$POOL"

`zpool add` is permanent, and the classic mistake is a single disk

zpool add tank /dev/disk/by-id/ata-NEW1 - with no mirror or raidz1 keyword - adds that disk as its own top-level vdev with no redundancy. The pool accepts it immediately and starts striping new writes across it.

The pool is now only as reliable as that one disk: when it dies, the pool is gone, including all the data on the redundant vdevs. ZFS gives no warning, because striping across dissimilar vdevs is a legal configuration.

You cannot remove it. Device removal does not work on a pool containing raidz vdevs, and even where it is supported it is a planned operation rather than an undo. The recovery is: destroy the pool, rebuild it, restore from backup.

This is what -n is for. Run it, read it, and only then run the command without it.

Step 3, method C: Expand a raidz vdev

OpenZFS 2.3 (shipped with PVE 9) can add a disk to an existing raidz vdev. Note the command is attach, and it names the vdev, not the pool.

Read-only / Safefind the exact vdev name first

POOL=tank

zpool status "$POOL"
# The vdev names appear in the tree: raidz1-0, raidz2-1, mirror-0, ...

Destructiveattach a disk to an existing raidz vdev

POOL=tank
VDEV=raidz1-0
NEW=/dev/disk/by-id/ata-NEW4

zpool attach "$POOL" "$VDEV" "$NEW"
zpool status "$POOL"

The pool then performs a reflow, redistributing existing data across the wider vdev. It runs online and it takes a long time on a full pool.

Step 4: Verify against the recorded numbers

Read-only / Safebigger, and still the right shape

POOL=tank

zpool list "$POOL"
zpool list -v "$POOL"
zpool status "$POOL"
zfs list -o name,used,avail -r "$POOL" | head -20

# Compare against the file saved in step 1
OUT=/root/zfs-expand-tank-20260812-0300.txt
diff <(zpool list -v "$POOL") <(grep -A20 '== list' "$OUT") | head -30

The check that can fail is not “did SIZE increase” - it is “does the layout match the design”. A pool that grew by adding a single unprotected disk also shows a larger SIZE.

Read-only / Safedoes Proxmox see it

pvesm status
zfs get -H -o value available rpool/data 2>/dev/null

Step 5: Scrub

Service impact possibleprove the expanded pool is consistent

POOL=tank

zpool scrub "$POOL"
zpool status "$POOL" | grep -E 'scan:|errors:'

# Later, when it has finished:
#   errors: No known data errors
#   and 0 repaired

Run it after any resilver or reflow has completed, not concurrently. A scrub that repairs blocks after an expansion is telling you something - either a new disk is bad or the pool had latent damage that the expansion has now spread.

Step 6: Close out

Update the capacity alert thresholds. A pool that was at 85% and is now at 45% will not alert again for a long time, and the threshold that was tuned for the old size is now wrong.
Record the new layout in documentation, including which physical bays hold which vdev. The next person to replace a failed disk needs that.
Note the expansion in zpool history terms - it is already there, but a human-readable change record explains why.

Rollback

Method	Rollback
Disk replacement, mid-way	Replace the old disk back in. Another resilver, no data risk
Disk replacement, complete	The vdev is larger. Shrinking is not supported
`zpool add` a matching vdev	None. The vdev is permanent
`zpool add` a bare disk by mistake	None. Destroy and restore from backup
`zpool attach` to a raidz	None. The vdev keeps its new width
Resilver failed on the new disk	Detach it; the pool never depended on it

Common patterns

Symptom	Likely cause	Resolution
Size unchanged after replacing disks	`autoexpand` was off, or not every disk in the vdev was replaced	`zpool set autoexpand=on`, `zpool online -e` each device
Gained far less space than expected from raidz expansion	Old data keeps the old parity ratio	Expected. Improves as data is rewritten
Pool now shows a single disk as a top-level vdev	`zpool add` without a vdev-type keyword	Not fixable. Restore from backup onto a correct layout
Resilver very slow	Pool nearly full, or a slow disk under load	Free space; throttle other I/O; expect hours
Errors on an existing disk during resilver	Latent damage exposed by the full read	Stop. Get data off. This pool is one failure from gone
New disk not visible to ZFS	Stale partition table or a foreign label	`wipefs -a` on a disk you have positively identified
Proxmox storage still shows the old size	`pvestatd` cache	Restart `pvestatd`, or wait a cycle

Escalation

Escalate when:

Any zpool add is about to be run.
The dry run shows a layout different from the plan.
An existing disk errors during a resilver.
A new disk fails SMART or errors while being added.
The pool is above 80 percent before the expansion starts.

Why this matters

ZFS gives you extremely strong guarantees and then hands you a command set that will let you throw them away without a confirmation prompt. zpool add tank /dev/sdc is a valid, sensible-looking command that converts a carefully designed redundant pool into a striped one, and the pool reports ONLINE afterwards.

That is why the discipline here is not “be careful” but a specific, mechanical habit: run it with -n, read the layout it prints, compare it to the layout you meant. It costs five seconds and it is the only check between a routine expansion and a restore-from-backup weekend.

The second idea worth carrying is that expansion is a stress test. A resilver reads every block on every remaining disk in the vdev, which is precisely the operation that finds the bad sector nobody knew about. That is a good reason to scrub regularly - so the bad sector is found on a Tuesday afternoon with full redundancy, rather than at hour six of a resilver with none.