DR exercises from tabletop to full

Failback is a second outage arriving after the emergency has passed, which is the previous lesson’s conclusion, and none of what it asks for is knowledge a team acquires during the incident that needs it. It is acquired by rehearsal or not at all. That leaves one question open: what a rehearsal must contain before its result carries information, and why an organisation can run one every year and end the decade knowing what it knew at the start.

Five stages, and the boundary each one crosses

Part XVI graded recovery tests by how much of a recovery was performed. An exercise is the same estate seen as an event rather than a job: a window, a declared scope, participants who are not the author of the procedure, someone watching who is not participating, and a report that outlives everyone in the room. The stages below are ordered by the boundary each crosses, and none retires the one beneath it: a monthly component recovery still catches the repository that stopped opening in March.

A tabletop touches nothing. Named people walk a named scenario against the actual runbooks while a facilitator injects conditions — the site is gone, and the person holding that credential is on a flight. It proves the plan exists in a form a human can follow, that each decision has an owner, and that the participants disagree about who may declare a disaster, which is the finding it produces most reliably. It also finds unreachable contacts and steps written for a system since replaced. It proves nothing technical: every assertion made at a tabletop is a claim by the person making it, and no tabletop discovers an expired credential, a repository that will not open, or a transfer running six times longer than the plan estimates.

A component recovery restores one element in isolation. A database onto a scratch host, one virtual machine from its image, a copy of the secret manager unsealed from escrowed material. It proves that artefact is readable today, that its procedure is accurate, and it produces a real duration for a real volume of data rather than an estimate. What it cannot reach is ordering: a component recovery runs while the rest of the estate is alive, so the component’s dependencies are satisfied by production — names resolve, the identity provider answers, the registry serves the image. Nor does it reach contention, since recovering one component over an idle link is a different measurement from recovering forty at once.

An isolated site bring-up assembles the stack from recovery material. Build the recovery site’s services into a network of their own, in the order the dependency graph demands, using only what recovery material provides. This is the first stage that exercises sequence, and therefore the first to meet the bootstrap cycle from earlier in this part: the secret manager comes up sealed, and the identity provider that would authenticate the holders of the unseal material is itself waiting to be started. It proves the material is complete enough to assemble, and nothing about clients, because nothing real is pointed at it — names are not moved, certificates may be locally issued substitutes, and no third party sees the traffic.

A live failover of one service moves real users. One production service is served from the recovery site for a defined period, with its clients following. This is the first stage to test the half of the problem living outside the estate: the names, the certificate that has to validate for a browser that never read your runbook, session behaviour across the cutover, and the partners whose allowlists were written against the primary’s addresses. It also generates real state at the recovery site, so the failback material is genuine. What it cannot prove is the estate: one service moved while its dependencies stayed behind establishes that the estate can run split, not that it can run elsewhere. Which service to pick is a design decision, and the one with the most external dependencies teaches more in an afternoon than the quiet internal one does in a year.

A full failover with failback moves everything in scope and brings it home. The declared scope runs at the recovery site with the primary unavailable, real work accumulates there, and the estate then returns carrying the state the exercise created. This is the only stage that tests the return, the aggregate bandwidth, the contention between simultaneous recoveries, and the people doing all of it at once. Even here the limits belong in the report: a planned exercise starts with a healthy primary, complete recovery material, a rested team and an hour somebody chose, and a real disaster supplies none of those.

The two conditions that separate an exercise from a demonstration

Two properties decide whether any of the last three stages produced information. Both are usually the first thing traded away for convenience, and an exercise lacking either is a demonstration.

The primary must be genuinely unavailable to the exercise. Not agreed to be off limits. Agreement is a statement of intent held by tired people under time pressure who know exactly where the value they need is, and the path of least resistance is to read it once from production and carry on. The block belongs at the network boundary, applied before the window opens and verified rather than assumed. Unavailable is not destroyed: the primary keeps serving everything outside the exercise scope, and the isolation is revertible in a single action, because that action is also the abort path.

Acceptance must compare data against a reference the recovered system did not produce. Decide before the window which invariant will be checked and where its reference value comes from: a row count, a sum over a business column, a manifest of paths and digests. The reference is recorded from the source and held where neither site can overwrite it. Then add the one question only data can answer — what is the newest record present — because that value, set against the last work the primary committed, is the exercise’s measurement of the loss window. A process list cannot answer it.

The objective, the scope and the abort criteria are written before the window

One page, circulated before the window opens, is the difference between an exercise and an outage everyone had been warned about. It is short enough to be read and specific enough to be failed.

exercise      EX24 - full failover with failback, tier-one order path
scenario      primary site unreachable, its storage assumed unrecoverable
in scope      orders-api, orders-db, payments-gw, the identity provider
out of scope  analytics warehouse, print service, everything tier three
isolation     deny site B to 10.10.0.0/16 at the site B border, one rule,
              reverted by one rule change, verified before 10:00
acceptance    orders-db row count and order-total sum equal the reference
              taken at 09:00 and held at site C; newest order id present is
              compared against the last id the primary committed
window        10:00 open, 16:00 latest end, failback complete by 18:00
abort         impact outside scope, any write to production from site B,
              or 16:00 with failback not started; called by the duty
              incident commander, no second opinion required

The scope statement earns its place by naming what is out. An unbounded exercise is refused by everyone who could authorise it, and rightly: nobody can consent to a risk nobody bounded. Writing the exclusions down converts an argument about whether to exercise at all into a negotiation about which services are in this one.

Acceptance is agreed beforehand for a reason that has nothing to do with process hygiene. After a run, acceptance drifts toward what happened: the database came up, the row count was never captured, and the report says the service was verified operational — true, unfalsifiable and worth nothing. Criteria fixed in advance are the only ones capable of producing a failure.

Observers record; they do not rescue

An observer has no task in the runbook. The job is to write down what happened: the timestamp of each decision, the questions asked, the step where the runbook was wrong and what was done instead, and the elapsed time between “we should fail over” and anyone touching a keyboard. Two records come out of an exercise and they are kept apart — the procedure as written, and what people did. The gap between them is the next revision of the runbook.

The rule that makes the role work is that an observer does not answer. The moment somebody supplies the missing password, names the host the runbook got wrong, or points at the skipped step, the finding is gone and the exercise has become a demonstration. This is genuinely difficult: the observer is usually the person who knows most, watching people struggle with something they could resolve in seconds. That struggle is the measurement.

For the same reason the participants are whoever is on shift, not the author of the procedure. A runbook validated by its author has been tested against the one person who does not need it.

A finding is a dated work item with an owner

Findings are the product. An exercise that completes with none has more likely tested less than it claimed — a narrow scope, a reachable primary, acceptance that watched services start — so a clean result deserves the same scrutiny as a failure.

id       exercise    finding                                     sev   owner     due         state
EX24-01  2026-05-14  site B egress address not on partner allow  high  n.okafor  2026-06-13  closed
EX24-02  2026-05-14  runbook step 7 names a decommissioned host  med   a.silva   2026-07-01  open
EX24-03  2026-05-14  two of five unseal holders unreachable      high  t.mbeki   2026-06-06  open
EX24-04  2026-05-14  payroll SaaS has no second region           low   f.dubois  2027-05-01  accepted

Severity is what the finding would have cost in a real recovery, not how hard it is to fix. EX24-03 describes a condition under which nothing decrypts, so it outranks a wrong hostname even though the hostname is fixable this afternoon. And some findings are constraints rather than defects — a SaaS platform with no second region is a fact about a supplier, so it becomes an accepted risk with a named accepter and a review date rather than an item that stays open forever and teaches everyone to ignore the list.

Production discipline

1. Name the stage in every report, and name what it could not reach. “A DR test was performed” says nothing until it says whether people sat in a room, one component was restored, or real clients were served from the recovery site with the primary unreachable.
2. Make the primary unavailable at the network boundary and verify it before the window opens. An agreement not to use production is not a control; the deny rule is, and a list of flows expected to be dead is what tells you the rule went the right way.
3. Fix acceptance in writing beforehand, as a data comparison. The reference value is recorded from the source and held where neither site can overwrite it, because criteria settled afterwards only ever describe what happened.
4. Give observers no task and no permission to answer. The moment the person who knows supplies the missing value, the finding that it was undocumented is destroyed and the exercise has become a demonstration.
5. Convert every finding into a tracked item with an owner and a due date before the room empties. A finding that exists only in a report is rediscovered at the next exercise, and the programme becomes an annual re-enactment of the same eleven problems.

Cross-course references

• Ansible for Production Sysadmins — Part XXIV (Assertions and Guardrails) is where this lesson’s acceptance criteria stop being a sentence in a document and become assertions that fail the run: the row count, the business-column sum and the newest-record check written as tasks that halt the play, so an exercise cannot be recorded as passed by a human deciding it looked fine.
• Observability for Production Sysadmins — Part LXIII (Synthetic Monitoring) builds the exact probe that produces the theatre this lesson warns about. A synthetic check confirming a login page renders is a valid availability signal and a worthless recovery acceptance test, because the page renders against an empty database; read that part for the probe and this one for why it cannot be the criterion an exercise is judged on.
• Observability for Production Sysadmins — Part CX (Observability During Major Incidents) covers the telemetry an observer reads while the exercise runs, which turns abort criteria from a judgement call into a condition somebody watches arriving before it is breached.

Quiz