Testing dependency failures - injecting the outage safely

Every mitigation in linux-designing-around-dependency-failures is a hypothesis. /etc/hosts might have a stale address. cache_credentials might be set on two nodes out of three. The break-glass account might have expired.

None of that shows up in a configuration review, because the configuration says what you intended. Only injecting the failure says what happens.

The safety envelope, before anything else

A dependency test is a deliberate outage. It is safe only inside an envelope you built first:

• One node, then the fleet. A fault on one node tests whether the mitigation works. A fault on all nodes tests your incident response. Do them in that order, on different days. Most teams skip straight to the second and learn nothing about the first.
• A timebox with an armed rollback, described below. Not a reminder, not a person watching a clock.
• Written abort criteria. “Abort if user-facing error rate exceeds X” or “abort if the cluster loses quorum”. Decided before, not during.
• An observer who is not the operator. The person typing is looking at the fault. Somebody else has to be looking at the service.
• A change record. Half the value of a game day is that the next unexplained alert is correctly attributed within ten seconds.

Arm the rollback before you inject

This is the single most important mechanic in the lesson. Schedule the undo first, so that losing your session, your network or your nerve still restores the system.

# systemd-run --on-active=600 --unit=depfail-rollback /usr/sbin/nft delete table inet depfail

Running timer as unit: depfail-rollback.timer
Will run service as unit: depfail-rollback.service

Ten minutes from now, the fault is removed whether or not anybody is still connected. Then inject:

# All injection lives in one table, so removal is one atomic operation
sudo nft add table inet depfail
sudo nft add chain inet depfail out '{ type filter hook output priority 0; policy accept; }'
sudo nft add rule inet depfail out udp dport 53 drop
sudo nft add rule inet depfail out tcp dport 53 drop

# Confirm exactly what is in force
sudo nft list table inet depfail

Everything about that shape is deliberate. A dedicated table means the rollback is nft delete table inet depfail - one command that cannot partially succeed and cannot touch the production ruleset. Compare it with adding rules to the main filter table, where the undo is a list of handles you have to find again while the system is misbehaving.

To finish early:

sudo nft delete table inet depfail
sudo systemctl stop depfail-rollback.timer

Hard failures are the easy half

Latency and loss are injected with tc netem. The hazard from the callout above applies directly - a root qdisc affects everything leaving that interface, corosync included:

# Safe only if the dependency is reached over an interface the
# cluster links do NOT use. Verify first.
ip route get 192.0.2.30
sudo corosync-cfgtool -s

# 400 ms of added latency towards that dependency
sudo tc qdisc add dev eth1 root netem delay 400ms

# Remove
sudo tc qdisc del dev eth1 root

Where the dependency shares an interface with the cluster links, do not use a root qdisc. Inject on the dependency’s own server, on a test client, or in a lab.

A test per dependency

Two entries there are honest limits rather than gaps. You cannot safely expire a production certificate, and you cannot safely skew a production cluster’s clock. What you can test in production is the detection: does the expiry alert fire at the right lead time, does the offset alert fire at the right threshold. Test the mitigation in a lab and the alarm in production.

What to measure

A test that produces “it worked” has produced nothing. Record numbers:

1. Time from injection to the first observable symptom anywhere - including "no symptom for the full ten minutes", which is a result.
2. Whether the alert on the dependency itself fired, and how long it took. A cache that hid the outage completely is a finding, not a pass.
3. Whether an operator could still log in, run sudo, and read logs on the affected node.
4. What the service did: served normally, degraded as designed, or degraded in a way nobody had described.
5. Time from removing the fault to full recovery, with no human intervention.
6. Anything that did not recover by itself.

The last two matter more than people expect. Recovery is where latent bugs live: a daemon that caches a negative result forever, a connection pool that never re-resolves a name, a service that needed a restart nobody realised was required. If removing the fault does not restore the system on its own, you have found a second outage hiding inside the first - one that would have doubled the length of the real incident.

Recording the finding

A finding is a sentence with a subject and a change:

With DNS unreachable, node2 could not be fenced: the fence device is configured with bmc-node2.example.com. Fencing is the last defence against corruption and it depends on a resolver. Change: put BMC addresses in /etc/hosts on every node, managed by configuration management, with a weekly check that they match the inventory.

Compare with “DNS test passed, no issues found”, which is what gets written when nobody measured anything.

Knowledge check