Incident command and human factors during recovery

An exercise programme that climbs from tabletop to full failover, which is where the previous lesson ended, tests the plan. It does not test the people, and the plan is not what fails first at three in the morning. What fails is one person holding four incompatible jobs, a record that exists only in somebody’s short-term memory, a ninth hour nobody scheduled, and a conversation with an executive who wants a number the responder does not have yet. None of this is soft material: each item has a mechanical consequence later, and the sharpest is that a recovery nobody wrote down cannot be failed back.

Four jobs, and the person who ends up holding all of them

A recovery of any size contains four distinct jobs. Someone decides: which recovery point to use, whether to proceed past a failed check, when to promote, what to accept as good enough. Someone executes: runs the commands, watches the output, reports what actually happened rather than what was expected. Someone communicates: with the business, with the affected teams, with vendors and with whoever is asking when it will be back. Someone records: what was changed, where, at what time, and on whose authority.

The jobs are separable because they use different faculties and run on different clocks. Execution is detailed and continuous. Decision is intermittent and requires a view of the whole. Communication is interruption-driven and paced by people outside the incident. Recording is continuous, requires no domain knowledge, and produces nothing the incident needs in the next ten minutes.

The default is that one person holds all four, and it arrives honestly. The person paged is the one who knows the system; escalating adds people who must be briefed before they can help, and briefing costs the minutes the incident is already consuming; the executive channel is answered by whoever is in it; and recording feels catchable-up later, because nothing visibly breaks when it is deferred.

Each collapses differently under that arrangement. The decider who is also the executor has already committed by the time a decision is due: a person twenty commands into a sequence evaluates whether to continue from inside it, with the sunk cost that implies. The executor who is also the communicator loses the output of the running command while typing an answer, and the recovery serialises against a queue of questions. The recorder is dropped first, because it is the only one of the four with no immediate consequence — which is why its absence is discovered days later, during failback, when someone has to establish what changed.

Small teams do not escape this by declaring it impractical. Two people split it as decide-and-communicate against execute-and-record, which keeps the decision outside the terminal and still produces a record. The recording job can be staffed by anyone who can type and read a clock, including someone with no knowledge of the estate: the cheapest role to fill and the most expensive to have skipped.

The decision log is what makes failback possible

The decision log is not the shell history and not the tool audit trail; those record what was typed on hosts you still have. It records intent, and it records the actions that happen outside every system that logs anything: a TTL changed in a provider’s console, a vendor call agreeing to rebind a licence, a ticket raised with a partner, a check skipped and the name of whoever accepted that risk. A decision not to act leaves no artefact in the estate at all. It exists only if somebody writes it down.

A usable entry carries six fields — timestamp in UTC, actor, action, target, effect expected, effect observed — and a seventh that earns its place in a disaster: whether the action is reversible, and if not, what it forecloses. The format matters far less than writing at the moment of the action rather than at the end of the shift.

LOGDIR=/srv/dr/incident/2026-08-28-primary-loss
DECISIONS="${LOGDIR}/decisions.log"
OPERATOR=$(id -un)

mkdir -p "$LOGDIR"

record() {
  printf '%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$OPERATOR" "$*" \
    >> "$DECISIONS"
}

record "promoted dr-db-01; chose recovery point 02:15Z over 04:05Z, checksum failed on the later one"
record "lowered zone TTL to 60s in provider console (out of band, no estate record)"

Two categories of entry pay for the whole practice. The first is the promotion moment for each service: the wall-clock instant at which the recovery site began accepting writes. That instant is the boundary between data belonging to the primary and data belonging to the recovery site, and failback is the reconciliation of exactly that boundary. The second is the emergency change — the firewall rule opened by hand, the quorum check disabled to get a cluster started, the elevated credential issued to an operator, the retention hold placed on a bucket, the replication link deliberately broken. Each is a temporary exception whose removal is now owed, and an exception nobody recorded becomes permanent: not by decision, but because nothing remembers it exists.

Fatigue, and a handover that transfers state rather than tasks

A recovery of a real estate runs for hours and sometimes days, and it degrades the faculties it depends on most. Fatigue removes the habit of reading output first: commands still get typed and their results stop being examined. Then calibration goes — the tired estimate of the next step is systematically short. Last and most expensive is the willingness to abandon a plan whose premise has expired, because the person who has spent four hours on an approach is the worst placed to judge it.

Rotation is therefore a control, not a courtesy. It has to be scheduled before the incident, because at hour nine nobody believes they are impaired; impairment that could self-assess would not be impairment. Set the shift length in the plan and let the incident enforce it rather than negotiate with it.

Handover is where recoveries silently lose their history. A handover that transfers a task list — “next, promote the second replica, then re-enable the job” — transfers the least valuable part of the state. The incoming shift needs what is running where and in what mode, the decisions already made and their reasons, the approaches already ruled out, the emergency changes outstanding, the open unknowns, and what has been said to whom. Without the ruled-out list, the incoming shift re-tries what the outgoing shift discarded, believing it is making progress.

LOGDIR=/srv/dr/incident/2026-08-28-primary-loss
HANDOVER="${LOGDIR}/handover-$(date -u +%H%MZ).md"

{
  printf '# Handover at %s\n\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf '## Decisions and actions so far\n\n'
  cat "${LOGDIR}/decisions.log"
} > "$HANDOVER"

Two properties make a handover hold. It is written, because a verbal handover is recalled differently by the two people involved and neither knows it. And it overlaps: the outgoing shift stays for a period after the incoming shift takes over, and the incoming decider states the current plan back in their own words before the outgoing one leaves. That restatement is the only cheap test of whether the transfer actually occurred.

Refusing to skip verification without becoming the obstacle

At some point in a long recovery someone with authority asks for the service to be opened now and checked afterwards. The request is reasonable from where they sit: they can price every minute of downtime and cannot price the check.

The argument that answers it is not about policy. It is about which options remain. Verification delays recovery by a measurable number of minutes; skipping it and taking writes on an unverified restore destroys the ability to restore again cheaply, because from the first customer transaction onward going back to another recovery point means discarding real work. The loss window the business eventually pays therefore grows with every minute the service stays up, and a twenty-minute check becomes hours of reconciliation if it turns out to have been necessary. That is a comparison the person asking can make, which is the only form in which a refusal is information rather than obstruction.

A credible refusal has a shape, and it is not the word no. Name the check. State its cost in minutes, measured rather than estimated where an exercise has measured it. State what it prevents, in terms of the loss window. Offer the non-binary alternatives — read-only, internal users first, one region — because the real answer is usually a schedule rather than a yes or a no. Then hand the decision back to the person who owns the risk and record their answer with their name against it. Recording an override is not blame-collection: failback needs to know the check did not run, and so does the review.

Saying what it cost, to people who do not want the mechanism

The audience outside the incident wants four things: whether it is working, when it will work, what was lost, and what they should do now. Everything else arrives as noise, and a responder who answers with the mechanism is answering the question they wish had been asked.

An update that works has four fields: one sentence of status, the impact stated in the business’s own terms rather than in components, the time of the next update, and the ask if there is one. The next-update time does the most work, because a missing update is read as an absence of control, and each person reading it that way generates an inbound question that consumes the attention the recovery needs. Publish on the interval even when nothing has changed; “no change, next update at 05:30 UTC” is a complete message.

The RPO statement is where honesty is hardest and most valuable. The recovered system is missing a window of data, and the window has edges that can be named: the last recovery point that was used, and the moment service was lost. So the statement is that transactions committed between 02:15 and 04:40 UTC are not present in the recovered system — not that data loss was minimal. Three categories belong in it separately: what is definitely lost, what is uncertain and still being established, and what is confirmed intact. State the uncertain category as uncertain, with a time by which it will be resolved — otherwise finance, customer support and whoever owns the regulatory clock all begin work against a number that later moves.

And state the figure as what it is: a measurement of what this recovery paid, under this architecture, on this occasion. It is not the RPO target, and if it exceeds the target that gap is the most important thing the incident produced.

The review, and the difference between a finding and a blame

Hold the review within a few days, while the people who were there still remember what they believed at the time. Its primary input is the decision log, which turns the meeting into an examination of a sequence rather than a collection of impressions.

The distinction that decides whether the review is worth holding is between a finding and a blame. A blame names a person and terminates: the operator ran the wrong command. A finding names the condition that made the wrong action reasonable and can be changed: the two hostnames differ by one character, both were in the operator’s history, and the destructive command asks for no confirmation. There is a serviceable test: if the proposed remedy is that this person should be more careful, or be replaced, it is a blame, because it predicts the next person will not make the error and nothing supports that prediction. If the remedy changes what the system makes easy, what information was available when the decision was made, or who was there to challenge it, it is a finding.

The question that produces findings is not what should have been done, but what the person knew, what the interface showed, what the runbook said, and what the pressure was. An action that looks obviously wrong afterwards was chosen by a competent person from the options they could see, and the review’s job is to reconstruct that set.

Reviews produce three things or they are theatre: actions with a named owner and a date, the emergency changes still to be reversed with a deadline against each, and new scenarios for the exercise programme drawn from whatever surprised the team. A review whose only conclusion is that communication could have been better has produced none of the three.

Production discipline

1. Name the four jobs and staff at least three of them before the incident. Deciding, executing, communicating and recording use different faculties and run on different clocks; the recording job needs no domain knowledge, is the cheapest to fill, and is the one whose absence is discovered during failback.
2. Write the decision log at the moment of the action, in UTC, and include what happened outside the estate. Console edits, vendor calls, partner tickets and checks deliberately skipped leave no trace in any system, and the promotion instant for each service is the cut that failback reconciles against.
3. Fix the shift length before anyone is tired, and hand over state rather than a task list. The incoming shift needs the decisions and their reasons, the approaches already ruled out, the outstanding emergency changes and the open unknowns; overlap the shifts and have the incoming decider restate the plan.
4. Answer the pressure to skip verification with minutes, with the loss window it protects, and with a partial-release option. Then put the decision back to the person who owns the risk and record their answer with their name against it, because failback and the review both need to know the check did not run.
5. Report the RPO as a measured window with named edges, and run the review against the log. Separate what is lost from what is uncertain and what is intact, publish on a fixed interval even when nothing changed, and accept only remedies that change what the system makes easy rather than who was on shift.

Cross-course references

• Linux for Production Sysadmins — Part LXXXI (Incident Command) builds the role structure this lesson applies to a recovery, and Part LXXXII (Root Cause Analysis) develops the finding-versus-blame distinction in general form; this lesson specialises both to an incident whose defining feature is that data is missing, so the record of what changed decides whether it can be failed back.
• Observability for Production Sysadmins — Part CIX (Incident Investigation Workflows) and Part CX (Observability During Major Incidents) supply the signals a decider reads before choosing to proceed, and Part CXIII (Documentation and Runbooks) is where the update cadence assumed here is designed; this lesson depends on those signals existing, because a decision log records a judgement and not the evidence that informed it.
• Terraform for Production Sysadmins — Part XXIX (Incident Response: The 3 AM Test) examines the same human conditions from the infrastructure-as-code side, where the emergency change made by hand is exactly the entry this lesson insists be logged and later reversed, since otherwise it becomes drift.

Quiz