Degraded arrays and rebuilds - replacing a failed disk · Linux

A failed disk in a RAID array is a routine operational event. The array goes degraded, performance drops, and the operations team must replace the disk and rebuild. This lesson walks through the workflow.

Identifying a degraded array

Read-only / Safemdstat degraded— A 4-slot RAID5 running on 3 members. Slot 0 is empty and no replacement has been added yet, so there is no recovery line - the array is degraded and sitting there. This is the state a disk failure leaves you in until you act.

$ cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdd[3] sdc[2] sdb[1]
6287424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
bitmap: 1/1 pages [4KB], 65536KB chunk

A degraded array shows:

/proc/mdstat with [N/(N-1)] or similar.
A _ in the device layout line, at the position of the missing slot.
State : clean, degraded in mdadm --detail.
Performance is reduced (reads from one disk, writes to all).
The kernel log shows a mdadm event.

Reading the line properly

Both bracketed groups on the second line are per-slot, and they must agree with the device list above them. Learn to check that, because an array that does not add up means you have misread it.

md0 : active raid5 sdd[3] sdc[2] sdb[1]
      6287424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
                                                                 | |   |
      raid-devices (slots) --------------------------------------' |   |
      in-sync devices ---------------------------------------------'   |
      per-slot state, slot 0 first ------------------------------------'

[4/3] - 4 slots exist, 3 hold an in-sync device.
[_UUU] - slot 0 is the gap; slots 1, 2 and 3 are up. That matches the device list exactly: sdb[1], sdc[2], sdd[3], and nothing at 0.
The number in brackets after a device name is its slot, not its position in the list. A device with an index at or above the slot count - sde[4] in a 4-slot array - is a spare, or a device currently rebuilding into a slot. It does not become U until recovery finishes; when it does, md re-reports it under its new slot number (sde[0]).
(W) after a device marks a write-mostly member, (F) a faulty one, (S) a spare.

`recovery` only appears on a degraded array

The progress line names the operation, and the name tells you what md thinks it is doing:

Line	Meaning	Array state
`recovery`	Rebuilding a missing member onto a spare	Degraded
`resync`	Initial sync of a new array, or after an unclean shutdown	Not degraded
`check` / `repair`	A scrub you or a timer started	Not degraded
`reshape`	Changing geometry: growing, changing level or chunk	Not degraded

So [4/4] [UUUU] next to a recovery line is not a state md can be in - if you think you are looking at one, re-read the slot counts. A healthy array doing a consistency pass says check, not recovery.

Identifying the physical disk

This is the highest-stakes step of the whole procedure. Pull the wrong drive from a degraded RAID5 and the array is gone.

Read-only / Safemap device to serial— lsblk and smartctl read the identity the disk itself reports. udevadm info --query=property --name=/dev/sdc | grep -E 'ID_SERIAL|ID_WWN' gives the same data from the udev database. Write the serial down before you touch the chassis: hot-swap events renumber /dev/sdX, so the name you used five minutes ago may now belong to a different disk.

$ lsblk -o NAME,SERIAL,WWN,MODEL /dev/sdc; sudo smartctl -i /dev/sdc | grep -E 'Serial Number|Device Model'

NAME SERIAL       WWN                MODEL
sdc  WD-WCC4N1234567 0x50014ee2b1234567 WDC WD40EFRX-68N32N0
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC4N1234567

mdadm --examine does not show a disk serial

An older habit is to run mdadm --examine /dev/sdX | grep Serial. That does not work. --examine prints the md superblock, and a native 1.x superblock contains Array UUID, Device UUID, Device Role and Array State — there is no serial field in it. The only Serial output mdadm can produce comes from Intel IMSM (firmware RAID) metadata, which is not what super 1.2 arrays use.

So --examine is the right tool to confirm array membership and event counts, and the wrong tool to find a physical disk. For physical identification use lsblk -o NAME,SERIAL, smartctl -i, or udevadm info. If the enclosure supports it, light the bay:

sudo ledctl locate=/dev/sdc      # blink the failure LED on that bay
sudo ledctl locate_off=/dev/sdc  # turn it off again afterwards

Replacing a failed disk

Confirm the failure: cat /proc/mdstat; mdadm --detail /dev/md0
Map the md member to a physical disk BEFORE touching anything: lsblk -o NAME,SERIAL,WWN,MODEL /dev/sdX; sudo smartctl -i /dev/sdX
Mark the device faulty if md has not already done so: sudo mdadm /dev/md0 --fail /dev/sdX
Remove it logically: sudo mdadm /dev/md0 --remove /dev/sdX
Only now physically pull the drive and insert the replacement
Re-check the new device name, because hot-swap renumbers devices: lsblk
Add the replacement: sudo mdadm /dev/md0 --add /dev/sdY
Monitor the rebuild: cat /proc/mdstat
Verify completion: mdadm --detail /dev/md0 shows State : clean

Two ordering rules are load-bearing here.

Fail before remove. man mdadm says of --remove: “remove listed devices. They must not be active. i.e. they should be failed or spare devices.” In the common real case — a disk throwing media errors and SMART warnings that md has not yet kicked out — --remove on its own returns mdadm: hot remove failed for /dev/sdc: Device or resource busy and you are stuck mid-incident. --fail is what makes the device non-active so that --remove is legal.

Remove before you pull. Physically extracting a disk that md still considers active drops a healthy array straight to degraded (or a degraded array to failed), and the resulting hot-swap event renumbers /dev/sdX, so the commands you had queued now point at the wrong disk.

Data-loss riskreplace disk— --fail marks the member faulty so md stops using it. --remove then detaches it, which is only legal once it is non-active. --add attaches the replacement and the kernel starts the rebuild immediately. Confirm the serial of /dev/sdc with lsblk before the --fail, and re-read lsblk after the swap because /dev/sdd may not be the name you expected.

$ sudo mdadm /dev/md0 --fail /dev/sdc; sudo mdadm /dev/md0 --remove /dev/sdc; sudo mdadm /dev/md0 --add /dev/sdd

mdadm: set /dev/sdc faulty in /dev/md0
mdadm: hot removed /dev/sdc from /dev/md0
mdadm: added /dev/sdd

Prefer --replace when the disk is still readable

If the disk is failing but has not failed — SMART pre-fail attributes, pending or reallocated sectors, occasional read errors — do not throw its redundancy away:

sudo mdadm /dev/md0 --replace /dev/sdc --with /dev/sdd

man mdadm describes --replace as marking the device as requiring replacement: the array rebuilds onto the spare while the suspect disk stays in service. If another member throws a read error mid-rebuild, md can still satisfy it from the disk being replaced.

--fail does the opposite. It removes that redundancy the instant you run it, and on a RAID5 you then spend the whole rebuild window with zero fault tolerance. Use --fail when the disk is already dead or must come out now; use --replace when you have the luxury of planning.

Monitoring the rebuild

Read-only / Saferebuild progress, first sample— The replacement is in as sde[4] - an index above the slot count, because it is rebuilding into slot 0 rather than occupying it yet. The counters are per-device (2095808 KB, one member's worth), not per-array.

$ cat /proc/mdstat

md0 : active raid5 sde[4] sdd[3] sdc[2] sdb[1]
6287424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[==>..................]  recovery = 12.5% (262144/2095808) finish=2.0min speed=14553K/sec
bitmap: 1/1 pages [4KB], 65536KB chunk

Sixty seconds later:

Read-only / Saferebuild progress, second sample— The percentage has moved and the finish estimate has shrunk, but note what has NOT changed: the array still reads [4/3] [_UUU]. It stays degraded for the whole rebuild - no redundancy, and a second failure now is fatal. Only when recovery completes does md re-report the device under its slot, giving sde[0] and [4/4] [UUUU]. Avoid heavy IO during the rebuild; it slows recovery and lengthens exactly this window.

$ sleep 60; cat /proc/mdstat

md0 : active raid5 sde[4] sdd[3] sdc[2] sdb[1]
6287424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[=====>...............]  recovery = 28.4% (595200/2095808) finish=1.5min speed=16673K/sec
bitmap: 1/1 pages [4KB], 65536KB chunk

When rebuilds fail

Configuration changemdadm detail— mdadm --detail shows the array state, rebuild progress, and device-level status. Use this to identify which md member failed and why. The output includes events, spares, and the active set. To go from the md member to a physical disk, follow up with lsblk -o NAME,SERIAL.

$ mdadm --detail /dev/md0 | head -10

---

Read-only / Safeexamine superblock— This is what --examine is for. Array UUID confirms the disk belongs to this array. Device Role and Array State show where it sits and what md thinks is missing. Events is the write counter: a disk whose event count is far behind the others was ejected earlier and must not be force-assembled back in. None of these fields is a hardware serial.

$ sudo mdadm --examine /dev/sdb | grep -E 'Array UUID|Device UUID|Device Role|Array State|Events'

        Array UUID : 9d2c1f5a:44b7e021:6c3f9a18:0e5d7b42
Device UUID : 1c7a4e93:2f80b5d6:a4917c30:88bde215
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
Events : 18402

If a rebuild fails (bad sectors on the replacement disk, second failure during rebuild), the array remains degraded. The operations team has three options: replace the new disk and retry, restore from backup, or live with the degraded array while planning a replacement.