Stabilisation - restoring service before you understand it

There are two clocks running during an incident and they measure different things.

• Time to mitigate — how long until users stop being affected.
• Time to resolve — how long until the fault is actually fixed and cannot recur.

The incident is driven on the first clock. linux-the-troubleshooting-loop is how you get to the second, and it is the right discipline once service is restored. During the outage, an investigation that would take forty minutes is worth less than a failover that takes four, even though the failover teaches you nothing.

This is genuinely uncomfortable for good engineers, because the mitigation often destroys the evidence that would explain the fault. That tension is real, and linux-incident-evidence-preservation is how it is resolved — not by delaying the mitigation, but by taking sixty seconds of capture before it.

The mitigation catalogue

Improvising under pressure produces creative mitigations with unpredictable side effects. Choose from a list instead. Almost every stabilisation is one of these:

The drain option deserves more use than it gets. Removing a backend from the pool restores the service and leaves the failing host running, untouched, for investigation:

# Substitute your own values before running:
NODE=node2

# Cluster-managed service: move resources off, leave the host up
sudo pcs node standby "$NODE"
sudo pcs status nodes

# Substitute your own values before running:
UNIT=checkout

# Behind a load balancer with a file-based health check: fail the
# check without stopping the process, so the LB drains it
sudo touch /etc/checkout/drain
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health
systemctl is-active "$UNIT"

That leaves you with a live, broken host and a working service — which is the best position an incident can be in.

One change at a time

The rule has one purpose: making recovery attributable. When three people apply three mitigations in the same two minutes and the service recovers, nobody knows which one worked, the review has no causal chain, and the fault is not actually understood.

Worse, the changes that did nothing are still in place. Two of those three mitigations are now undocumented configuration on a production host — which is precisely how a stabilisation becomes next month’s configuration drift.

1. Say what you are about to do, in the channel, before you do it. "I am going to standby node2. I expect the VIP to move to node1 and the 5xx rate to drop within 30 seconds."
2. Say what you expect to happen. Stating the expectation beforehand is what lets you tell success from coincidence afterwards.
3. Make the change. One change.
4. Wait for the stated effect, on the measurement you named. Not on a feeling that things look better.
5. Report the outcome, including when nothing happened. A mitigation that had no effect is a finding.
6. Record the change in the mitigations register, whether or not it helped, so it can be unwound later.

The mitigations register

Every mitigation is a production change made without the normal controls. That is justified during the incident and not afterwards, so each one has to be tracked from the moment it is applied:

MITIGATIONS - INC-4820
| Time  | Host   | Change                                   | By        | Unwind |
|-------|--------|------------------------------------------|-----------|--------|
| 06:19 | node2  | pcs node standby                         | e.brandi  | open   |
| 06:24 | edge01 | rate limit 50r/s on /api/search          | s.okonkwo | open   |
| 06:31 | all    | feature flag checkout_v2 -> off          | e.brandi  | open   |
| 06:47 | node2  | (none - rolled back the 06:19 standby)   | e.brandi  | done   |

Anything still marked open when the incident closes is an action item with an owner, not a completed incident. linux-incident-recovery-and-standdown covers unwinding them, and linux-drift-remediation covers what happens when they are never unwound: they become drift, they are found months later by a detection run, and by then nobody remembers whether they are load-bearing.

When stabilising first is wrong

The rule has three exceptions, and each of them is a case where restoring service destroys something more valuable than the downtime costs.

• Confirmed security compromise. The goal changes from restoration to containment. Rebooting evicts the attacker from memory and destroys the evidence of how they got in; restoring from a backup taken after the compromise restores the compromise. Isolate the host at the network, preserve it, and escalate — do not reboot, do not reimage, do not “clean it up”.
• Active data corruption. If the system is writing bad data, keeping it up makes the eventual recovery larger. Stopping the writes is the mitigation, even though it looks like extending the outage.
• A mitigation whose failure mode is worse than the incident. A failover to a secondary that has never been tested, during a SEV2 where the service is degraded but working, can convert a partial outage into a total one. Weigh the mitigation risk against the current impact, and say so out loud.

Knowledge check