Common Linux failure labs - the diagnostic drills

This lab covers four common Linux failure scenarios. The discipline is to follow the methodology for each.

Objective

You will inject four faults into a disposable host — a full filesystem, an OOM kill, an unreachable destination, and a service that will not start — diagnose each from first principles, and return the host to its exact pre-lab state.

Requirements

• A disposable VM or container you can snapshot. Not a workstation, not a shared host, not anything running a service someone else depends on.
• Root or sudo.
• nginx, iptables, curl and python3 installed.
• stress-ng for Task 2 (optional; the task gives an alternative).

Tasks

Task 0: Snapshot and record the baseline

Snapshot first. Then record the state each later task will change, because you cannot restore to a value you never wrote down.

# 1. VM snapshot (Proxmox / libvirt / cloud provider equivalent)
#    e.g. Proxmox:  qm snapshot 120 pre-failure-lab
#         libvirt:  virsh snapshot-create-as failure-lab pre-lab

# 2. Record what the lab will change
mkdir -p /tmp/lab-baseline
stat -c '%a %U:%G %n' /etc/nginx/nginx.conf | tee /tmp/lab-baseline/nginx-mode.txt
sudo cp -a /etc/nginx/nginx.conf /tmp/lab-baseline/nginx.conf
sudo iptables-save    | sudo tee /tmp/lab-baseline/iptables.rules >/dev/null
ip route show         | tee /tmp/lab-baseline/routes.txt
systemctl is-active nginx | tee /tmp/lab-baseline/nginx-state.txt
mount | grep /tmp/lab || echo 'no /tmp/lab mount' | tee /tmp/lab-baseline/mounts.txt

Expected: nginx-mode.txt reads 644 root:root /etc/nginx/nginx.conf on a stock install. That number is what you restore to in Cleanup — not a guess.

Task 1: Filesystem full

Simulate a filesystem full by filling a tmpfs mount.

A tmpfs is used deliberately: it is a filesystem you can fill and then throw away without touching a real one.

sudo mkdir -p /tmp/lab
sudo mount -t tmpfs tmpfs /tmp/lab -o size=100M
sudo dd if=/dev/zero of=/tmp/lab/bigfile bs=1M count=200
# This will fail: "No space left on device"
df -h /tmp/lab

Diagnose:

df -h
sudo du -sh /tmp/lab/* | sort -h

Document the findings. Then restore:

sudo rm -f /tmp/lab/bigfile
sudo umount /tmp/lab
sudo rmdir /tmp/lab
df -h | grep /tmp/lab || echo 'restored: no lab mount'

Task 2: OOM

Simulate an OOM:

# A transient scope: the cgroup and its limit disappear when the
# process exits, so this fault cleans itself up.
sudo systemd-run --scope -p MemoryMax=200M \
  stress-ng --vm 1 --vm-bytes 500M --timeout 20s

# No stress-ng? Any allocator inside the same scope will do:
sudo systemd-run --scope -p MemoryMax=200M \
  python3 -c "b=bytearray(); [b.extend(bytes(10**7)) for _ in range(200)]"

Diagnose:

journalctl -k | grep -i oom
dmesg | grep -i oom
systemctl status "run-*.scope" 2>/dev/null | head

Document the findings and the fix (raise the limit, or fix the leak — decide which from the evidence, not by reflex).

Restore: nothing to undo. The scope is transient, so the cgroup and its MemoryMax vanish with the process. Confirm with systemd-cgls | grep run- returning no lab scope.

Task 3: Network unreachable

Two different faults produce the same user-visible symptom — “it will not connect” — and the diagnosis is what tells them apart. Inject them one at a time.

Stand up a target you own, so the fault is scoped to it:

python3 -m http.server 8080 --bind 127.0.0.1 &
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/   # 200

Fault A — a firewall drop. Put it in a dedicated chain, and match only the lab port. Cleanup is then a single chain removal, with no chance of deleting a rule that was already there.

sudo iptables -N LABDROP
sudo iptables -A LABDROP -j DROP
sudo iptables -I OUTPUT -p tcp -d 127.0.0.1 --dport 8080 -j LABDROP

curl --max-time 5 http://127.0.0.1:8080/      # hangs, then times out

Fault B — a routing black hole. Use 192.0.2.10, from TEST-NET-1, a documentation range that is never a real destination.

sudo ip route add blackhole 192.0.2.10/32
ip route get 192.0.2.10                        # "RTNETLINK answers: ..."

Diagnose:

ip route get 192.0.2.10          # is there a route at all?
sudo iptables -L OUTPUT -n -v --line-numbers   # is a rule counting packets?
sudo iptables -L LABDROP -n -v
ss -tlnp                         # is anything even listening?
sudo traceroute -T -p 8080 127.0.0.1

The distinguishing signal: a routing fault fails immediately with “Network is unreachable”; a DROP rule fails by timing out, because nothing is sent back. A REJECT would fail immediately too — which is why you always read the rule, not just the symptom.

Restore both faults now, before moving on:

sudo iptables -D OUTPUT -p tcp -d 127.0.0.1 --dport 8080 -j LABDROP
sudo iptables -F LABDROP
sudo iptables -X LABDROP
sudo ip route del blackhole 192.0.2.10/32
kill %1                                       # stop the test server

sudo iptables -S OUTPUT | grep LABDROP || echo 'restored: no lab rule'
ip route get 192.0.2.10

Task 4: Service will not start

Confirm you recorded the original mode in Task 0 — you cannot restore a permission you never read.

cat /tmp/lab-baseline/nginx-mode.txt      # expect: 644 root:root /etc/nginx/nginx.conf

sudo systemctl stop nginx
sudo chmod 000 /etc/nginx/nginx.conf
sudo systemctl start nginx                # fails

Diagnose:

sudo systemctl status nginx
sudo journalctl -u nginx -n 50
sudo nginx -t
ls -l /etc/nginx/nginx.conf

Document the findings. Then restore to the recorded mode:

sudo chmod 644 /etc/nginx/nginx.conf
sudo chown root:root /etc/nginx/nginx.conf
sudo nginx -t
sudo systemctl start nginx
systemctl is-active nginx                 # expect: active

Task 5: Document

For each scenario, write a brief report:

• Symptom.
• Impact.
• Evidence.
• Hypothesis.
• Test.
• Fix.
• Verification.

Validation

Every one of these must pass before you call the lab done:

mount | grep -q ' /tmp/lab '  && echo 'FAIL: tmpfs still mounted'  || echo 'ok: no tmpfs'
sudo iptables -S | grep -q LABDROP && echo 'FAIL: lab chain remains' || echo 'ok: no lab chain'
ip route show | grep -q 192.0.2.10 && echo 'FAIL: blackhole route remains' || echo 'ok: no lab route'
[ "$(stat -c '%a %U:%G' /etc/nginx/nginx.conf)" = '644 root:root' ] \
  && echo 'ok: nginx.conf mode restored' || echo 'FAIL: nginx.conf mode wrong'
systemctl is-active nginx
curl -sSf -o /dev/null https://example.com && echo 'ok: outbound 443 works'

Expected outcome

Four written diagnostic reports, and a host whose iptables -S, ip route show, mount and stat /etc/nginx/nginx.conf output matches the Task 0 baseline byte for byte. A break/fix drill that leaves residue has taught you half the job.

Troubleshooting

• iptables -D says “No chain/target/match by that name” — the rule was already removed, or the arguments do not match the rule exactly. iptables -D OUTPUT 1 deletes by line number instead; find the number with iptables -L OUTPUT -n --line-numbers.
• umount /tmp/lab says “target is busy” — a shell is sitting in the directory. lsof +D /tmp/lab or fuser -vm /tmp/lab names the process. umount -l is a last resort, not a first move.
• nginx still fails after chmod 644 — the fault may not be the one you injected. nginx -t names the file and line; compare against /tmp/lab-baseline/nginx.conf.
• ip route del says “No such process” — the route is already gone. Confirm with ip route get 192.0.2.10.
• You lost the baseline or skipped Task 0 — restore the snapshot. Do not guess at file modes.

Cleanup

Run this whether or not every task succeeded. Each line is idempotent, so it is safe to run twice.

# Task 1
sudo umount /tmp/lab 2>/dev/null; sudo rmdir /tmp/lab 2>/dev/null

# Task 3
sudo iptables -D OUTPUT -p tcp -d 127.0.0.1 --dport 8080 -j LABDROP 2>/dev/null
sudo iptables -F LABDROP 2>/dev/null; sudo iptables -X LABDROP 2>/dev/null
sudo ip route del blackhole 192.0.2.10/32 2>/dev/null
pkill -f 'http.server 8080' 2>/dev/null

# Task 4
sudo cp -a /tmp/lab-baseline/nginx.conf /etc/nginx/nginx.conf
sudo chmod 644 /etc/nginx/nginx.conf && sudo chown root:root /etc/nginx/nginx.conf
sudo nginx -t && sudo systemctl start nginx

# Prove it
sudo diff <(sudo iptables-save) /tmp/lab-baseline/iptables.rules && echo 'ruleset matches baseline'
diff <(ip route show) /tmp/lab-baseline/routes.txt && echo 'routes match baseline'
rm -rf /tmp/lab-baseline

If any check still disagrees with the baseline, roll back to the Task 0 snapshot. That is what it is for.

What you learned

• A fault you cannot reverse is not a drill.
• Record state before you change it; a restore target you guessed is not a restore.
• Scope the blast radius of an injected fault: a dedicated iptables chain and a documentation-range address, not a global DROP and a real public IP.
• Prefer faults that clean themselves up, such as a transient systemd-run --scope.
• The same symptom has different causes, and only the evidence separates them: routing failures are immediate, DROP failures time out.