Special vdevs, metadata devices and native encryption · Proxmox VE

The previous lesson covered three devices you can add to a pool that all share a comforting property: losing them costs you performance, not data. L2ARC is a read cache and its contents are a copy. A SLOG holds sync writes that have not yet reached the pool, and its loss costs you at most the last few seconds of acknowledged writes.

A special vdev is not in that family, and treating it as if it were is the most expensive mistake in this lesson. It is a top-level vdev holding the only copy of data the pool cannot function without. Lose it and you lose the pool — every dataset, every zvol, every snapshot.

The payoff for that risk is large enough that it is still worth doing. A pool of spinning disks with a mirrored NVMe special vdev behaves like flash for directory traversal, snapshot listing and small-file I/O, because all of that is metadata and metadata now lives on NVMe.

What actually goes on it

The upstream description is precise about the contents: “metadata, the indirect blocks of user data, intent log (in absence of separate log device), and deduplication tables.” Optionally, small data blocks join them via the special_small_blocks property.

That list explains the performance effect. On a RAIDZ2 pool of spinning disks, a ls -R across a large dataset is thousands of small random reads of metadata blocks scattered across seven-millisecond seeks. Moving those to NVMe does not make the disks faster; it removes the disks from that path entirely.

The comparison that matters:

Device class	Holds	If it fails
`cache` (L2ARC)	A second copy of cached reads	Reads fall back to the pool. No data loss
`log` (SLOG)	Sync writes not yet committed	At most the last few seconds of sync writes
`special`	The only copy of metadata and small blocks	The pool is gone
`dedup`	The only copy of the dedup table	The pool is gone

There is one structural constraint to know before planning: “A pool must maintain at least one normal (non-dedup/-special) vdev before other devices can be assigned to the special class.” You cannot build a pool entirely out of special vdevs, which is occasionally what someone tries when they discover how much the class helps.

Adding one, and sizing it

Data-loss riskadd a mirrored special vdev to an existing pool— Adds a top-level vdev. This is not reversible on a pool containing RAIDZ - see the removal section below. Check the device list twice; zpool add with a typo attaches a single disk as a top-level vdev and there is no undo.

POOL=tank

# always dry-run first: -n prints what would be done and does nothing
zpool add -n "$POOL" special mirror \
/dev/disk/by-id/nvme-SPECIAL_A \
/dev/disk/by-id/nvme-SPECIAL_B \
/dev/disk/by-id/nvme-SPECIAL_C

zpool add "$POOL" special mirror \
/dev/disk/by-id/nvme-SPECIAL_A \
/dev/disk/by-id/nvme-SPECIAL_B \
/dev/disk/by-id/nvme-SPECIAL_C

zpool status "$POOL"

Sizing has two components, and only one of them is predictable.

Metadata alone is typically a fraction of a percent of the pool for VM-disk workloads, where the average block is large. A rule of thumb around 0.3% of pool capacity is the usual starting figure, and it is generous for zvols and stingy for a pool full of small files.

Small data blocks, if you enable them, are entirely workload-dependent and can be arbitrarily large. This is where sizing goes wrong.

Service impact possiblemeasure the actual block-size distribution before deciding— zdb walks the pool's block pointers. It is read-heavy and can take a long time on a large pool, so run it outside a busy period. The histogram tells you how much data sits below any candidate special_small_blocks value.

POOL=tank
zdb -Lbbbs "$POOL" 2>/dev/null | tail -40

An added special vdev only receives new allocations. Existing metadata stays where it was written, on the spinning disks, and it migrates only as those blocks are rewritten. On a pool holding VM disks that happens gradually; on a pool holding cold archive data it may never happen. The way to force it is a full rewrite — zfs send | zfs receive into a new dataset, or a restore from PBS — which is worth knowing before you conclude that the new NVMe made no difference.

`special_small_blocks`, and the way it goes wrong

The property is documented as “the threshold block size for including small file or zvol blocks into the special allocation class”, with a default of 0 — meaning no data blocks at all, metadata only. Valid values run “from 0 to maximum block size (16 MiB)”, and “before setting this property, a special class vdev must be added to the pool.”

Configuration changesend small blocks to the special vdev, on one dataset— Set it per dataset rather than at the pool root so the blast radius is bounded. It affects blocks written after the change; existing data is unaffected until rewritten.

DATASET=tank/smallfiles
zfs set special_small_blocks=32K "$DATASET"
zfs get -r special_small_blocks tank

Set special_small_blocks at or above recordsize and the special vdev becomes the pool

The threshold is compared against the block size after compression. A dataset with recordsize=128K and special_small_blocks=128K sends every block to the special class, because no block can exceed the record size.

What happens next is not an error. The special vdev fills up, and once it is full, “allocations intended for it revert to the standard class” — silently. So the sequence a team actually experiences is:

Excellent performance for a week.
Gradual, unexplained decline as the special vdev fills.
zpool list -v shows the special vdev at 99% and the data vdevs nearly empty.
No health warning, no error in zpool status, nothing in the logs.

The pool is not damaged and the data is safe. But you now have a pool whose small NVMe mirror is full, whose spinning disks are empty, and whose only correction is to rewrite the datasets after lowering the threshold.

Start at 0 — metadata only — and raise it deliberately, per dataset, after measuring with zdb. 32K against a 128K recordsize is a common, defensible starting point. 128K against a 128K recordsize is the mistake.

Read-only / Safethe check that catches it before the users do— Read-only. Compare the special vdev's utilisation against the data vdevs. A special vdev filling much faster than the pool as a whole is the signal, and it appears long before any performance symptom.

POOL=tank
zpool list -v "$POOL"
zpool status "$POOL"

Read-only / Safe— Illustrative. The pool is 20% full and the special vdev is 96% full. That divergence is the entire diagnosis, and nothing else in ZFS will report it.

$ zpool list -v tank

NAME                     SIZE  ALLOC   FREE  FRAG    CAP  HEALTH
tank                    43.7T  9.12T  34.6T    4%    20%  ONLINE
raidz2-0              21.8T  4.51T  17.3T    4%    20%  ONLINE
  ...
raidz2-1              21.8T  4.52T  17.3T    4%    20%  ONLINE
  ...
special                     -      -      -     -      -  -
mirror-2              1.86T  1.79T  73.4G   61%    96%  ONLINE
  nvme-SPECIAL_A          -      -      -     -      -  ONLINE
  nvme-SPECIAL_B          -      -      -     -      -  ONLINE

Illustrative output

Removing one: mostly, you cannot

This is the constraint that turns a special vdev from a tuning decision into an architecture decision.

The zpool remove documentation lists what can be removed: “hot spare, cache, log, and both mirrored and non-redundant primary top-level vdevs, including dedup and special vdevs”. Then it gives the condition: “Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz or draid vdev.”

Two further requirements apply: all top-level vdevs must have matching ashift, and all encrypted dataset keys must be loaded.

So:

Pool data vdevs	Can the special vdev be removed?
Mirrors	Yes, if `ashift` matches across all top-level vdevs
RAIDZ1/2/3	No
dRAID	No

The combination people most want — a large RAIDZ2 pool of spinning disks accelerated by a special vdev — is precisely the one where the decision is permanent. Adding the vdev is a one-way door on that pool, and the only exit is to destroy the pool and rebuild it.

Which makes the mirror width and the device quality decisions you make once, carefully, rather than something to improve later.

Under the hood

Why losing the special vdev loses everything, when it is “only” metadata.

ZFS is a tree. The uberblock points at the meta-object set, which points at per-dataset object sets, which point at dnodes, which point at indirect blocks, which finally point at the data blocks holding your VM’s disk contents. Every level above the leaves is metadata.

Normal ZFS behaviour is to store metadata redundantly beyond the vdev’s own redundancy — the copies mechanism, which by default writes two copies of metadata and three of the most critical metadata, spread across different vdevs where possible. That is why ZFS can often survive damage that would destroy another filesystem.

A special vdev changes where those copies go, not how many exist. When metadata is allocated to the special class, its copies live on the special vdev. Lose that vdev and you lose the indirect blocks — the pointers. The data blocks on the RAIDZ2 vdevs are physically intact and completely unreachable, because nothing remains that knows which sequence of blocks constitutes a given file. There is no scan that reconstructs it: ZFS blocks are not self-describing in a way that lets you rebuild the tree bottom-up.

This is what makes the redundancy requirement structural rather than cautious. The special vdev holds the index for the whole pool, so its redundancy is the pool’s redundancy, whatever the data vdevs say.

Two operational consequences follow from the same structure.

A resilver of a special vdev member is fast, because the vdev is small. That is a genuine advantage: replacing a failed NVMe in a three-way special mirror completes in minutes rather than the hours a data-vdev resilver takes.

Fragmentation on the special vdev matters more than elsewhere. It absorbs small, frequently-rewritten allocations, which is exactly the pattern that fragments an allocator. zpool list -v reports FRAG per vdev, and a special vdev whose fragmentation climbs while the data vdevs stay flat is behaving normally — but it is the number to watch if performance drifts without the capacity changing.

ZFS native encryption

Encryption is a per-dataset property, and like volblocksize it is fixed at creation: “encryption must be specified at dataset creation time and it cannot be changed afterwards.” It also “requires a keyformat to be set at dataset creation time.”

Configuration changecreate an encrypted dataset— Creates a new encryption root. Existing datasets cannot be encrypted in place; data has to be copied into a new encrypted dataset. The default cipher is aes-256-gcm.

POOL=tank
DATASET="$POOL/encrypted"

zfs create \
-o encryption=aes-256-gcm \
-o keyformat=passphrase \
-o keylocation=prompt \
"$DATASET"

zfs get encryption,keyformat,keylocation,keystatus "$DATASET"

keyformat accepts passphrase, raw or hex, with raw and hex keys required to be 32 bytes. keylocation accepts prompt (the default), a file:// path, or an HTTP/HTTPS URL.

Descendants inherit the key from their encryption root by default, and operations on the root apply implicitly to everything inheriting from it. A child can become its own encryption root by specifying a keyformat at creation time.

Read-only / Safekey state across the pool— Read-only. keystatus is available or unavailable per encryption root. A dataset whose key is unavailable cannot be mounted and its contents cannot be read - including by the VMs that were using it.

zfs get -r keystatus,encryptionroot,mounted tank

What happens if a node with encrypted datasets reboots unattended?

It comes back, and the VMs on the encrypted datasets do not.

This is the availability cost of native encryption and it is the part that gets discovered rather than planned. With keylocation=prompt, the key is not on the system. After a reboot, zfs get keystatus reports unavailable, the dataset is not mounted, its zvols do not appear, and every guest whose disk lives there fails to start. The node itself is perfectly healthy, which is what makes the alerting confusing.

Someone has to log in and run:

zfs load-key -r tank/encrypted
zfs mount -a

Then start the affected guests. zfs load-key -a does every encryption root in every imported pool, and -n performs a dry-run check that a key is correct without loading it — useful when you are not certain which passphrase you have.

The three ways out, and their honest trade-offs:

Accept it, and plan for it. A datacentre node that reboots unexpectedly gets a human. If your on-call already responds to an unplanned node reboot, adding “load the key” to that runbook costs nothing. This is the right answer for most estates, and it preserves the property that makes encryption worth having.

keylocation=file:///... on the root filesystem. The node boots unattended and everything starts. It also means the key sits on the same machine as the data, unencrypted, so an attacker who takes the server takes both. This defends against a disk being removed and against nothing else. Sometimes that is genuinely the threat model — a compliance requirement about disposed drives, for instance — and then it is a reasonable choice made with open eyes. It is not a reasonable choice made by accident because unattended reboots were annoying.

Fetch the key over the network at boot. keylocation accepts an HTTPS URL, so a key server that only answers requests from known addresses gives you unattended boot with the key held elsewhere. It moves the problem rather than solving it — now the key server is a dependency of every node’s ability to start guests — but it is a defensible architecture and it is the one that survives a stolen server.

What is not defensible is discovering the choice during an incident. Decide before the first encrypted dataset exists, write it in the runbook, and test it by rebooting a node on purpose.

Common mistakes

Treating a special vdev like L2ARC. L2ARC loss is free. Special vdev loss is the pool.
A single-device special vdev on a redundant pool. The pool’s failure tolerance becomes one device.
Adding it to a RAIDZ pool without realising it cannot be removed. zpool remove refuses on any pool containing a RAIDZ or dRAID top-level vdev.
special_small_blocks at or above recordsize. Everything goes to the special vdev, it fills, allocations silently revert, and performance decays with no warning.
Expecting existing metadata to migrate. Only new allocations go to the special class; a rewrite is what moves the rest.
Sizing from a rule of thumb after enabling small blocks. Measure with zdb -Lbbbs — the distribution is workload-specific.
keylocation=file:// on the same disk as the data, described in a compliance document as encryption at rest. It protects against a removed disk and nothing more.
Encrypting a dataset “later”. The property is create-time only; the migration is a copy.

Key takeaways

A special vdev holds metadata, indirect blocks and optionally small data blocks. It is part of the pool, not a cache: its loss loses everything.
Its redundancy must match the pool’s. Three-way NVMe mirror for a RAIDZ2 pool.
It cannot be removed from a pool containing RAIDZ or dRAID vdevs. On those pools, adding one is permanent.
special_small_blocks defaults to 0. Setting it at or above recordsize sends everything to the special class, which then fills silently and reverts.
Only new allocations use the special class; existing metadata migrates when rewritten.
Native encryption is create-time only, per dataset, inherited from an encryption root, and does not hide dataset names, sizes or snapshot structure.
An unloaded key means unmounted datasets and guests that will not start. Choose the key-availability model before the first encrypted dataset, not during the first unplanned reboot.

Knowledge check

Why this matters

Both halves of this lesson are about the same class of decision: things ZFS lets you set once and then enforces forever.

That is unusual for ZFS. Almost everything else about a pool is adjustable — you can add vdevs, replace disks with larger ones, change compression, retune ARC, add and remove cache and log devices freely. The design philosophy is that operational decisions should be reversible, and it mostly holds.

The exceptions cluster, and they are worth knowing as a set: pool geometry, ashift, volblocksize, whether a dataset is encrypted, and whether a RAIDZ pool has a special vdev. Every one of them is decided at creation, none of them can be changed by a command, and each one is corrected only by rewriting all the data.

What makes the special vdev the sharpest of these is that its downside is invisible until it fires. A pool with a single-device special vdev runs beautifully. zpool status says ONLINE. Scrubs pass. Every capacity and performance metric looks correct. The configuration is indistinguishable from a good one right up to the moment one NVMe drive fails and forty terabytes of intact RAIDZ2 data becomes unreadable.

So the discipline this asks for is not caution about using special vdevs — they are genuinely excellent, and a RAIDZ2 pool of spinning disks with a mirrored NVMe special vdev is one of the best price-performance configurations available to a Proxmox operator. It is a discipline about mirroring them properly the first time, setting special_small_blocks from a measurement rather than a guess, and writing down that this pool now has a one-way door in it.

The same sentence covers the encryption half: decide how the key becomes available at boot before there is anything encrypted, because the alternative is deciding it at 4 AM with guests down.