ZFS pools, vdevs, and RAIDZ

Intermediate⏱ ~18 min

What you'll learn

Build ZFS pools with the right vdev layout
Choose between mirror, RAIDZ1, RAIDZ2, RAIDZ3 for production
Size vdevs for predictable resilver times
Recognise when a pool layout is unsafe

Prerequisites

Storage fundamentals: block, file, object

Verified against Proxmox VE 9.2.4 · Proxmox Backup Server 4.2.5 · Ceph Squid / Tentacle · Debian 13 (Trixie) · Linux kernel 7.0 (PVE 9.2 default) · 2026-08-12

Why this matters in production

A ZFS pool is close to forever. You cannot shrink a pool. You cannot remove a top-level vdev from a pool that contains any RAIDZ or dRAID vdev. And while current ZFS can widen a RAIDZ vdev by attaching a disk, it cannot change its parity level — a RAIDZ1 stays a RAIDZ1. Mistakes made at pool creation cost days of effort and risk data loss to fix.

Mental model

A ZFS pool is composed of vdevs (virtual devices). Each vdev is a redundancy group:

Mirror vdev: N-way mirror. Tolerates N−1 disk failures.
RAIDZ1 vdev: single-parity RAID. Tolerates 1 disk failure.
RAIDZ2 vdev: double-parity RAID. Tolerates 2 disk failures.
RAIDZ3 vdev: triple-parity RAID. Tolerates 3 disk failures.
Stripe: single disk, no redundancy.

flowchart TB
  P[Pool 'tank']
  P --> V1[vdev 1: mirror of 2 disks]
  P --> V2[vdev 2: RAIDZ2 of 6 disks]
  P --> V3[vdev 3: mirror of 2 disks]

The pool’s failure tolerance is the minimum of each vdev’s tolerance. A pool with three mirror vdevs can lose up to 1 disk per vdev and survive; losing 2 disks in the same vdev takes the whole pool down.

Choosing a vdev layout

Layout	Capacity efficiency	Failure tolerance	Performance	When to use
Mirror of 2	50 %	1 disk	Best random IOPS	Production VM storage
Mirror of 3	33 %	2 disks	Best random IOPS	Mission-critical
RAIDZ1 (3+1)	75 %	1 disk	Mediocre random IOPS	Bulk storage, small pools
RAIDZ2 (6+2)	75 %	2 disks	OK random IOPS	Large capacity, slow resilver acceptable
RAIDZ3 (7+3)	70 %	3 disks	OK	Very large arrays

Resilver time and vdev sizing

When a disk fails, ZFS rebuilds (“resilver”) the failed disk’s data onto a replacement. Resilver time is dominated by the total vdev size, not by how much data is in use.

Vdev size	Mirror resilver	RAIDZ2 resilver
1 TB	~30 min	~1 hour
4 TB	~2 hours	~4 hours
10 TB	~5 hours	~12 hours
50 TB	~24 hours	~3 days

A vdev that takes 24 hours to resilver is unsafe: another disk failure during the resilver takes the pool down. The OpenZFS community rule of thumb is to keep resilver under 12 hours; production deployments often target under 4 hours.

ashift

ashift is the sector-size hint for the pool. Modern SSDs are 4 KB sectors. Setting ashift=12 (or letting ZFS autodetect) ensures ZFS writes in 4 KB blocks that align with the SSD’s physical layout.

zpool create -o ashift=12 tank mirror /dev/disk/by-id/...

CLI walkthrough

zpool status tank

zpool list -v tank && zpool get fragmentation tank

zpool create -o ashift=12 -m /tank tank mirror /dev/disk/by-id/sda /dev/disk/by-id/sdb

zpool create -o ashift=12 -m /tank tank raidz2 /dev/disk/by-id/sda /dev/disk/by-id/sdb /dev/disk/by-id/sdc /dev/disk/by-id/sdd raidz2 /dev/disk/by-id/sde /dev/disk/by-id/sdf /dev/disk/by-id/sdg /dev/disk/by-id/sdh

Best practices

Use /dev/disk/by-id/ names, not /dev/sd* — stable across reboots.
Mirror for VM storage, RAIDZ for bulk.
Keep vdevs small (4–8 disks) to bound resilver time.
Set ashift=12 explicitly unless you know all disks are 512n.
Add a SLOG device for sync write performance.
Add an L2ARC device if read caching would help and you have RAM to spare.
Schedule scrubs to detect and repair silent corruption.

Production considerations

RAIDZ expansion exists now, and it does less than the name suggests

The rule “a RAIDZ vdev cannot change shape” was true for most of ZFS’s life and is the reason so much pool-planning advice treats geometry as permanent. OpenZFS added RAIDZ expansion, and PVE 9.2 ships ZFS 2.4, so on a current Proxmox node it is available.

zpool attach is the command. The upstream wording: “If the existing vdev is a RAID-Z device (e.g. specified as raidz2-0), the new device will become part of that RAID-Z group. A ‘raidz expansion’ will be initiated.”

Three limits decide whether it solves your problem:

The parity level does not change. “Expansion does not change the number of failures that can be tolerated without data loss (e.g. a RAID-Z2 is still a RAID-Z2 even after expansion).” A RAIDZ1 you regret is still a RAIDZ1 afterwards.

Existing data keeps its old ratio. “After the expansion completes, old blocks retain their old data-to-parity ratio… but distributed among the larger set of disks.” Only “New blocks will be written with the new data-to-parity ratio.” So a 5-wide RAIDZ2 expanded to 6-wide gives you the better efficiency on new writes and not on the data you already have — the capacity gain arrives gradually, as data is rewritten.

It requires the raidz_expansion pool feature, so a pool created under an older ZFS needs the feature enabled before it can be used.

What this changes in practice: adding a disk to a RAIDZ vdev is now a real answer to “we are running out of space”. It is still not an answer to “we chose the wrong parity level”, and it does not make the initial layout decision less important — it makes exactly one of its consequences reversible.

Common mistakes

Creating a RAIDZ1 with 8 disks. One failure begins a multi-day resilver during which a second failure is likely.
Using /dev/sda instead of /dev/disk/by-id/. Disk letters change; by-id names do not.
Forgetting ashift=12.
Assuming RAIDZ expansion fixes a parity mistake. It widens a vdev; it does not turn a RAIDZ1 into a RAIDZ2, and existing blocks keep their old data-to-parity ratio until they are rewritten.
Planning to remove a vdev later. zpool remove refuses on any pool containing a RAIDZ or dRAID top-level vdev.

Key takeaways

Pool = vdevs; vdev = redundancy group. Loss of any vdev loses the pool.
Mirror for VM storage; RAIDZ for bulk.
Keep vdevs small. Plan for fast resilver.

Knowledge check

Knowledge check · 3 questions

Q1. A pool has three RAIDZ1 vdevs of 6 disks each. How many disks can fail simultaneously without data loss?

1 3 6 18
Q2. Moving a pool from RAIDZ1 to RAIDZ2 means building a new pool and copying the data, not adding disks to the existing vdev.

True False
Q3. Why is ashift=12 recommended for SSDs?

Passing score: 75%. Answers are checked in this browser.