RAID5 and RAID6 - parity for capacity and fault tolerance · Linux

RAID5 and RAID6 are the workhorses of production data storage. They give the capacity advantage of striping plus fault tolerance through parity. The trade-off is the rebuild: a failed disk must be reconstructed from the parity, and the rebuild time scales with disk size.

How parity works

The write hole

Why a parity stripe update is not atomic

Updating one block in a RAID5 stripe takes two writes: the data chunk, and the recomputed parity chunk. They land on different disks, and nothing makes them atomic. Lose power between them and the stripe’s parity no longer matches its data.

md cannot detect that. Nothing was reported as failed, so the array comes back clean. The damage stays latent, sometimes for months. It surfaces only when a disk fails and md reconstructs the missing chunk by XORing the survivors with that stale parity - and hands the application a block of plausible, wrong data. No error, no log line, no I/O failure. This is the write hole, and it is the defining reliability difference between parity RAID and mirroring.

RAID6 narrows the window but does not close it. Mirroring (RAID1 and RAID10) has no write hole at all, because there is no parity to recompute: each write is one independent write per leg, and a torn pair is a mismatch between two copies rather than a silent reconstruction input.

Mitigations, in order of preference:

Use RAID10 for anything where silent corruption is unacceptable - database volumes above all.
Enable PPL on a RAID5 array. The Partial Parity Log lives in the metadata region of the members, so it needs no extra device: mdadm --create ... --consistency-policy=ppl. man 8 mdadm describes it as closing the write hole and eliminating resync.
Use a write journal on a small, power-safe SSD: mdadm --create ... --write-journal /dev/nvme0n1p1. md logs each stripe transaction and replays it after an unclean shutdown. Stronger than PPL, and available for RAID4/5/6, but the journal device becomes a dependency of the array.
A hardware controller with a healthy BBU or supercapacitor write cache does the same job in firmware - provided the battery is actually good. Monitor it.
A UPS with a tested automatic shutdown. Necessary, not sufficient: it does nothing for a PSU failure or a kernel panic.

Scrub monthly and alert on mismatch_cnt

Detection matters as much as prevention. A scrub reads every stripe and recomputes parity, recording disagreements in md/mismatch_cnt.

# Start a scrub (read-only check; it does not rewrite parity)
echo check | sudo tee /sys/block/md0/md/sync_action

# Watch progress, then read the result
cat /proc/mdstat
cat /sys/block/md0/md/mismatch_cnt

On a parity array a non-zero mismatch_cnt means a stripe is inconsistent - an already-open write hole. Alert on it. Most distributions ship a monthly mdcheck or raid-check timer that does this; confirm yours is enabled rather than assuming.

Note the caveat from man 4 md: on RAID1 and RAID10 a non-zero mismatch_cnt is expected and not reliably meaningful, because the kernel may write to a swap page while it is in flight. Treat the counter as an alarm on RAID4/5/6 only.

RAID5 chosen on capacity arithmetic alone

The selection tables in most documentation compare RAID levels on minimum disks, fault tolerance, capacity and speed. On those four columns RAID5 beats RAID10 for a general-purpose volume, so it gets chosen. None of those columns mention the write hole.

Picture the sequence: a database on mdadm RAID5, no UPS, no PPL, no journal. A power cut in March leaves one stripe inconsistent; the array reports clean and nobody knows. A disk fails in June. The rebuild reconstructs that stripe from stale parity and returns corrupt rows. Nothing errors, so the corruption is dumped into every nightly backup until someone notices the data is wrong - by which point the good backups have rotated out.

That is why “RAID5 versus RAID10” is a durability question, not an arithmetic one.

Creating a RAID5 array

Data-loss riskcreate RAID5— mdadm --create with --level=5 builds a RAID5 array. Three disks minimum for RAID5. Capacity is (N-1) * size, so 3 x 1 TB = 2 TB usable. The array is immediately available but degraded until the initial sync completes.

$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd

mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Creating a RAID6 array

Data-loss riskcreate RAID6— mdadm --create with --level=6 builds a RAID6 array. Four disks minimum for RAID6. Capacity is (N-2) * size, so 4 x 1 TB = 2 TB usable. RAID6 survives two simultaneous disk failures (rare but possible during rebuilds on large disks).

$ mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

The rebuild trade-off

Read-only / Saferebuild in progress— After a disk replacement, the array is rebuilding onto sde. Note that it still reads [4/3] [_UUU]: a rebuilding member is listed with an index above the slot count (sde[4]) and does not count as in-sync until recovery completes, so the array is degraded for the whole window. The counters are per-device. Rebuild speed depends on disk speed, controller bandwidth, and host load; for a 10 TB disk on a busy host it can take 24+ hours, and a second disk failure at any point in that window means data loss.

$ cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sde[4] sdd[3] sdc[2] sdb[1]
6287424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[==>..................]  recovery = 8.5% (178176/2095808) finish=2.3min speed=13896K/sec
bitmap: 1/1 pages [4KB], 65536KB chunk

Long rebuilds on large disks

A 10 TB disk rebuild at 200 MB/s takes 14 hours. During that time, the array is in a degraded state with reduced performance. A second disk failure during rebuild means data loss for RAID5 (two missing blocks), or RAID6 if a third disk fails.

For large production arrays, do not use RAID5. A second failure inside a 14-hour degraded window is not a rare event, and RAID5 has no cover for it. Above roughly 4 TB per disk, treat RAID5 as unsuitable.

What replaces it depends on which constraint binds.

RAID6 adds a second parity disk, so the array survives a failure during the rebuild, and it keeps (N-2) usable capacity. This is the choice when capacity per pound is the constraint.
RAID10 rebuilds by copying one surviving mirror partner: one disk read, one disk written, and only that pair degraded. A RAID6 rebuild instead reads every surviving member in full and recomputes from data plus parity, which takes far longer, loads the whole array while redundancy is lowest, and gives every one of those surfaces a chance to return an unrecoverable read error mid-rebuild. RAID10 is the choice when rebuild exposure is the constraint, and it also has no write hole.

Note which way that points. Large disks argue for RAID10 on rebuild grounds, not against it; the common claim that big disks mean you should prefer RAID6 is inverted. Choose RAID6 when you need the capacity and are accepting the longer rebuild, not because the disks are large. The RAID10 lesson works the same comparison from the other side.

Adding a hot spare

Data-loss riskadd hot spare— A hot spare is a disk that is part of the array but not used for data. When a primary disk fails, the array automatically rebuilds to the spare. The rebuild starts immediately - no operator intervention required. Use a hot spare for any production array.

$ mdadm /dev/md0 --add-spare /dev/sdf

mdadm: added /dev/sdf as spare