Recovery and stand-down - closing an incident properly

The alert cleared. Everyone is tired, the channel has gone quiet, and somebody types “looks like we are good”. That moment is where incidents are most often closed badly, and where the second incident of the day is created.

Recovery has three parts, and only the first one is technical: verify that users are actually served, watch for long enough to be confident it holds, and unwind the emergency changes that got you here.

Verify from where the users are

The alert clearing means the check that fired has stopped firing. That is a statement about the check, not about the service. linux-incident-impact-assessment established the same principle at the start of the incident, and it matters at least as much at the end: the recovery must be measured on the same terms the impact was.

1. Measure the same thing you measured at declaration. If the impact was recorded as an edge 5xx rate, the recovery is that rate returning to baseline - not a green dashboard tile.
2. Check success volume, not just success rate. A 100% success rate on 4% of normal traffic means the users have gone, and that is the shape a serious incident takes immediately before it is wrongly closed.
3. Exercise the whole path, not the health endpoint. Health checks are designed to be cheap, which usually means they do not touch the database, the queue, or the third party that broke.
4. Check the paths you did not think about. A queue that stopped being consumed is now deep; a retry backlog is about to arrive all at once; a cache is cold and the database behind it is about to see traffic it has not seen for an hour.
5. Verify the reversibility question from the impact assessment. If you recorded that writes might have been dropped, the recovery is not complete until somebody has looked.

# Substitute your own values before running:
URL=https://shop.example.com/checkout

# The same 30-sample probe used at declaration, so the numbers compare
for i in $(seq 1 30); do
  curl -sS -o /dev/null -w '%{http_code}\n' --max-time 5 "$URL" || echo "timeout"
  sleep 1
done | sort | uniq -c | sort -rn

The observation period

Do not stand down at the moment of recovery. Stand down after watching it hold.

A workable default is to observe for the length of the update cadence the severity requires — 15 to 30 minutes for a SEV1 — and to observe actively: someone is assigned to watch, and “recovery confirmed” is a statement they make, not the absence of a new alert.

The period exists to catch four specific things:

Unwind the emergency changes

Every entry in the mitigations register from linux-incident-stabilisation is an unreviewed production change. Before the incident closes, each one has exactly three possible end states:

There is no fourth state, and “we will tidy it up later” is not one of the three. An emergency change with no end state recorded becomes configuration drift: it is discovered months later by a drift detection run, and by then nobody remembers whether it is load-bearing. linux-drift-remediation describes the resulting investigation from the other end — and the reason its first rule is never to revert before you know why the file changed is precisely this incident, closed without a register.

# Substitute your own values before running:
NODE=node2

# Unwind: return the drained node to service and confirm it took work
sudo pcs node unstandby "$NODE"
sudo pcs status nodes
sudo pcs status resources

# Substitute your own values before running:
UNIT=checkout

# Unwind a file-based drain, and confirm the health check recovers
sudo rm -f /etc/checkout/drain
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health
systemctl is-active "$UNIT"

Unwind one at a time and confirm each one, for the same reason you applied them one at a time. Removing three mitigations simultaneously and watching the service degrade tells you nothing about which one was load-bearing.

Stand-down and handover to the review

The stand-down message closes the incident for everyone who was watching it, so it has to say what was true, not what was hoped:

INCIDENT CLOSED - INC-4820
Resolved:    2026-08-11T07:14Z    Closed: 2026-08-11T07:46Z
Duration:    05:58Z - 07:14Z (76 min, measured from first customer-visible
             5xx at the edge, not from the first alert at 06:07Z)
Impact:      Checkout unavailable for EU customers, approx 40% of traffic.
             No data loss on the payment path (verified). Order-history
             write path verified clean at 07:31Z.
Resolved by: Rollback of deploy 4471 plus rate limit on /api/search.
Still open:  Rate limit on edge01 remains in place (DEFERRED, owner
             s.okonkwo, ticket OPS-9902, review by 2026-08-15).
Review:      PIR scheduled 2026-08-13 14:00Z. Timeline, impact assessment
             and mitigations register attached to INC-4820.

Two details in that message carry most of its value. The duration is measured from the first customer-visible symptom rather than the first alert — the gap between 05:58 and 06:07 is a detection finding, and hiding it inside the duration makes it disappear. And Still open names a mitigation that has not been unwound, with an owner and a date, so it cannot quietly become permanent.

What the review needs is then already written, because it was written during the incident:

• The timeline, with the ordering uncertainties marked.
• The impact assessment, including the confidence line.
• The mitigations register, with every entry closed out.
• The detection gap: symptom time versus alert time versus human-response time.

linux-incident-roles-and-communication covers running the review itself, and linux-avoiding-human-error-as-cause covers where its conclusions tend to go wrong.

Knowledge check