md RAID monitoring and failure detection

Building the array is the easy part. A RAID1 that loses a leg keeps serving reads and writes exactly as before: no error, no slowdown a user would notice, nothing in the application log. The array is now a single disk, and it will stay that way until someone looks.

This lesson closes that gap. Four things have to be true before an array counts as production-ready: it assembles at boot, it tells you when a member fails, it finds bad sectors before a rebuild needs them, and its health is a metric you can alert on.

Persist the array before you do anything else

mdadm --create builds an array in the running kernel. It does not write anything that survives a reboot. Without the two commands below, the array may come back with a different name (/dev/md127 is the classic symptom) or fail to assemble at all — and if it holds the root filesystem, the host does not boot.

$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

ARRAY /dev/md0 metadata=1.2 name=host:0 UUID=8f3a1c2e:9b4d5e6f:1a2b3c4d:5e6f7a8b

$ sudo update-initramfs -u        # Debian-family. RHEL-family: sudo dracut -f

update-initramfs: Generating /boot/initrd.img-6.8.0-40-generic

Alerting: mdadm —monitor

mdadm --monitor is a daemon that polls arrays and fires an event on Fail, FailSpare, DegradedArray, SparesMissing and RebuildFinished. The distribution ships a unit for it, but it does nothing useful until you tell it where to send the event.

# /etc/mdadm/mdadm.conf   (Debian-family; /etc/mdadm.conf on RHEL-family)
MAILADDR storage-oncall@example.com
PROGRAM  /usr/local/sbin/md-alert

MAILADDR needs a working local MTA. In a fleet that has no mail path — most modern fleets — PROGRAM is the one that matters: mdadm runs it with the event name, the array and, when relevant, the device as arguments, so it can push straight into your alerting system.

$ sudo systemctl enable --now mdmonitor.service

Created symlink /etc/systemd/system/multi-user.target.wants/mdmonitor.service -> /usr/lib/systemd/system/mdmonitor.service.

Scrubbing: find bad sectors before the rebuild does

This is the mechanism that decides whether a rebuild succeeds.

A RAID5 rebuild has to read every sector of every surviving member. Sectors that were written years ago and never read since may have decayed. Nobody has noticed, because nobody has read them. The rebuild reads them, hits an unrecoverable read error on a surviving disk, and the array — already down one member — cannot reconstruct that stripe. That is the mechanism behind “the rebuild failed and we lost the array”.

A scrub reads the whole array on a schedule, while it is still redundant, so a decayed sector is found at a moment when the parity or mirror can still rewrite it.

$ cat /sys/block/md0/md/sync_action; echo check | sudo tee /sys/block/md0/md/sync_action

idle
check

$ cat /sys/block/md0/md/mismatch_cnt

0

Distributions already ship a schedule; enable it rather than writing your own cron entry:

# Debian-family: /etc/cron.d/mdadm runs checkarray on the first Sunday monthly
sudo /usr/share/mdadm/checkarray --cron --all --idle --quiet

# RHEL-family: systemd timers
sudo systemctl enable --now mdcheck_start.timer
sudo systemctl enable --now mdcheck_continue.timer

1. Confirm a schedule exists: systemctl list-timers | grep -i mdcheck on RHEL-family, or cat /etc/cron.d/mdadm on Debian-family
2. Run one scrub by hand during a maintenance window and time it, so you know the real duration on your disks
3. Read mismatch_cnt after every scrub and record it - a count that grows month on month is a failing member, not noise
4. Throttle the scrub if it hurts production: sysctl dev.raid.speed_limit_max caps rebuild and scrub speed in KB/s per device
5. Alert on a scrub that has not completed in over 35 days as well as on a non-zero mismatch count

Write-intent bitmaps

mdadm --detail prints an Intent Bitmap field. It is worth understanding because it changes the cost of an unclean shutdown by orders of magnitude.

Without a bitmap, an unclean shutdown means the kernel does not know which regions were mid-write, so it resyncs the entire array: hours on large disks, with degraded performance throughout. With an internal bitmap, the array records which regions have writes in flight, so recovery touches only those regions — seconds instead of hours.

# Add a bitmap to an existing array
sudo mdadm --grow /dev/md0 --bitmap=internal

# Confirm it
sudo mdadm --detail /dev/md0 | grep -i 'intent bitmap'

The cost is a small write amplification, since the bitmap is updated as writes are issued. For arrays large enough for a full resync to matter — which is most of them — the trade is worth it. Latency-critical small arrays are the exception.

Metrics and alerts

node_exporter’s mdadm collector reads /proc/mdstat and exposes it, which turns “someone must remember to look” into a normal alerting rule.

# A member has failed
- alert: MdArrayDegraded
  expr: node_md_disks{state="failed"} > 0
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "md array {{ $labels.device }} on {{ $labels.instance }} has a failed member"

# Fewer active disks than the array requires - degraded even with no hard failure
- alert: MdArrayMissingDisks
  expr: (node_md_disks_required - node_md_disks{state="active"}) > 0
  for: 15m
  labels: { severity: critical }

# The array is not fully in sync
- alert: MdArrayNotSynced
  expr: node_md_state{state="active"} == 0
  for: 15m
  labels: { severity: warning }

Knowledge check