Lab: Build a RAID array, simulate a failure, and rebuild

This lab walks through the full RAID lifecycle: create, use, simulate a failure, rebuild. By the end you will have built and recovered a RAID1 array.

Objective

By the end of this lab, you can:

• Create a RAID1 array from loop devices.
• Mount a filesystem on the array.
• Simulate a disk failure and verify the array is degraded.
• Replace the failed disk and observe the rebuild.
• Verify the array is healthy after the rebuild.

Tasks

Task 1: Create the array

Create three 200 MB files, attach each one to exactly one loop device, and build a RAID1 array with two members and a hot spare.

$ for i in 1 2 3; do truncate -s 200M /tmp/raid-lab-$i.img; done

LOOP1=$(sudo losetup --find --show --nooverlap /tmp/raid-lab-1.img)
LOOP2=$(sudo losetup --find --show --nooverlap /tmp/raid-lab-2.img)
LOOP3=$(sudo losetup --find --show --nooverlap /tmp/raid-lab-3.img)
echo "$LOOP1 $LOOP2 $LOOP3"

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 $LOOP1 $LOOP2
sudo mdadm /dev/md0 --add-spare $LOOP3

sudo mkfs.ext4 -q /dev/md0
sudo mkdir -p /mnt/raid
sudo mount /dev/md0 /mnt/raid
df -h /mnt/raid

Confirm you have exactly three loop devices, one per image.

$ losetup --list | grep raid-lab

/dev/loop10  0  0  0  0  /tmp/raid-lab-1.img  0  512
/dev/loop11  0  0  0  0  /tmp/raid-lab-2.img  0  512
/dev/loop12  0  0  0  0  /tmp/raid-lab-3.img  0  512

Verify the array with cat /proc/mdstat and mdadm --detail /dev/md0. You should see two active members and one spare.

Real arrays also need to survive a reboot. An array that assembles only because it happens to be running is not configured.

$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

Task 2: Use the array

echo "raid test data" > /mnt/raid/test.txt
cat /mnt/raid/test.txt

Task 3: Simulate a failure

Mark the first loop device as failed. Use the variable you already have. Do not run losetup --find again - that would attach a second device to the same image, and mdadm --fail would then be pointed at something that is not a member of the array. mdadm would report “device not found” or simply do nothing useful, and the array would stay clean.

$ sudo mdadm /dev/md0 --fail $LOOP1
cat /proc/mdstat

md0 : active raid1 loop12[2] loop11[1] loop10[0](F)
    204736 blocks super 1.2 [2/1] [_U]
    [====>................]  recovery = 24.3% (49920/204736)

Read that output carefully. loop10[0](F) is the member you just failed. [2/1] [_U] means one of two active slots is populated, so the array is degraded. But recovery has already started: the hot spare added in Task 1 was promoted automatically the moment a slot went faulty. That automatic promotion is the entire value of a hot spare - the rebuild window does not wait for a human.

The data on the array remains accessible throughout.

Task 4: Verify the array is still functional

cat /mnt/raid/test.txt

The data should still be readable from the surviving disk.

Task 5: Replace the failed disk

The spare has already taken over, so the array is rebuilding without you. What is still outstanding is the failed member: it is attached to the array as a faulty device and there is no spare left to cover the next failure.

Wait for the rebuild to finish, evict the failed member, then re-add that device as the new hot spare. On real hardware this is the moment you physically pull the disk and slot in a replacement.

$ # wait until /proc/mdstat shows no recovery in progress
sudo mdadm --wait /dev/md0

sudo mdadm /dev/md0 --remove $LOOP1
sudo mdadm --detail /dev/md0 | grep -E 'State|Devices'

# the replacement disk: here, the same device re-added as a spare
sudo mdadm /dev/md0 --add-spare $LOOP1
cat /proc/mdstat

Task 6: Verify array integrity

$ sudo mdadm --detail /dev/md0 | grep -E 'State :|Active Devices|Spare Devices|Failed Devices'
cat /mnt/raid/test.txt

         State : clean
Active Devices : 2
Spare Devices : 1
Failed Devices : 0
raid test data

State : clean, two active devices, one spare, zero failed, and the file still readable. Anything else means the rebuild has not finished or the membership is wrong. Do not proceed to cleanup until this matches.

Task 7: Clean up

Order matters. Stopping the array while the filesystem is still mounted leaves the mount pointing at a device that no longer exists; unmount first.

$ sudo umount /mnt/raid
sudo mdadm --stop /dev/md0
sudo mdadm --zero-superblock $LOOP1 $LOOP2 $LOOP3

sudo losetup -d $LOOP1 $LOOP2 $LOOP3
rm -f /tmp/raid-lab-*.img
sudo rmdir /mnt/raid

# remove the persistence line added in Task 1
sudo sed -i '/md0/d' /etc/mdadm/mdadm.conf
sudo update-initramfs -u

Confirm the host is back to its starting state:

$ cat /proc/mdstat            # no md0
losetup --list | grep raid-lab || echo 'no lab loop devices'
findmnt /mnt/raid || echo 'not mounted'

Validation

• A RAID1 array was created with 2 active members and 1 spare.
• A simulated disk failure caused the array to become degraded but functional.
• A replacement disk was added and the array rebuilt.
• The data survived the simulated failure.

Cleanup

The Task 7 cleanup removes all RAID and loop device artifacts.

What you learned

• Creating a RAID array with mdadm is straightforward - and it is not finished until the definition is persisted to mdadm.conf and the initramfs.
• Attach each backing file to exactly one loop device. losetup --find called twice on the same image creates a second device, and mdadm operations then target the wrong one.
• A hot spare is promoted the instant a member is marked faulty, so the rebuild begins with no human in the loop. That is the window a spare buys you.
• The failed member is evicted with --remove and spare coverage is restored by adding the freed device back with --add-spare. Check membership with --detail before adding anything.
• Unmount before --stop, and detach loop devices by name. losetup -D detaches every loop device on the host, including the ones backing snap packages.