critical riskservice affecting~180 min

Execute a DR failover

1 · Prerequisites

Confirm every item is in place before any state change.

A named declaration authority for this estate, with a named deputy, both reachable now. The role is written down in advance; an incident is not the moment to work out whose decision this is.
The declaration criteria agreed with the business before today, as observable signals and thresholds rather than as judgement — what state of the primary, sustained for how long, with what estimate to repair in place, makes failover the chosen path.
A dependency graph for the estate that names what must be running before what, produced while the estate was healthy. Not an architecture diagram: an ordered list of pairs that a machine can sort.
A recovery site that already holds the backup catalogue, the key material and an identity bootstrap path that does not traverse the primary. All three are pre-flight checks below, because all three are the ones estates discover they lack at the worst moment.
Fencing capability against the primary that works when the primary is unreachable by its normal management path: out-of-band console or power, an upstream route withdrawal, or credential revocation at the storage and database layers.
Control of the cutover mechanism — authoritative DNS, the global load balancer, or the routing policy — plus the record of what the current time-to-live values are and who else caches the answer.
A client position outside the estate from which the service can be exercised the way a customer exercises it: an external host, an external resolver, and a credential that belongs to a real business role.
A place to record declarations, decisions, recovery points and timings that survives the loss of the primary site, plus the failback procedure on hand — this runbook's exit condition is the point at which that other procedure becomes the only way home.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm who is declaring, and record the criterion they matched. A failover with no named declarer is a failover nobody authorised, and it will be re-litigated afterwards by people who were not there. Write the name, the time in UTC, and the specific signal that met the threshold.
· Establish the time since onset and the estimate to repair in place, and compare both against the agreed thresholds. The commonest failure in this whole procedure is a late declaration: an hour spent on "it might come back" is an hour subtracted from the recovery window, and it is spent before the sequence has even started.
· Confirm the recovery site holds the backup catalogue, and that it can be read there without reaching the primary. A catalogue that lives only at the failed site turns a rehearsed failover into an archaeology exercise across repositories nobody has an index for.
· Confirm the key material at the recovery site opens the repositories, by opening one. Not that the escrow exists — that it works, today, with the credential the recovery team holds. Repository bytes that cannot be decrypted are indistinguishable from repository bytes that are gone.
· Confirm the identity bootstrap path. Establish which of directory, secrets platform, certificate authority and time source the recovery site can reach or start on its own, and whether any of them silently depends on the primary. An estate that can start everything except the thing that authenticates everything has not failed over.
· Confirm you can fence the primary before you need to. Test the out-of-band path, the route withdrawal or the credential revocation now, while the decision is still reversible. Fencing discovered to be unavailable after the recovery site is live is the split-brain scenario, arriving late.
· Record the current time-to-live on every name that will be moved, and the name of every party that caches those answers. The cutover is not complete when the record changes; it is complete when the last cache that matters has expired or been repointed.
· Confirm the monitoring for the recovery site is live and not silenced, and that alerts from it will reach the bridge. Failing over into a site nobody is watching converts one incident into two.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare, by name, and start the clock. The declaration is the first record in the log: who declared, at what time in UTC, and which criterion was met. Everything after this is executed against a decision that exists rather than against a mood on the bridge.
2Announce the declaration to the business before the first technical step. The people who will validate the recovery need to know it is happening, and the people who own the data need to know a recovery point is about to be chosen on their behalf.
3Fence the primary so it cannot resume serving and cannot accept writes. This is the step that makes the rest safe. A primary that returns while the recovery site is live will accept writes that are never merged and never missed until an audit finds them. Fence at more than one layer: stop and mask the service units, withdraw the route or the virtual address, and revoke the credentials the primary uses against shared storage and databases.
4Prove the fencing held before continuing. Confirm from outside the primary that it no longer answers on its service address and no longer authenticates against shared state. "It was already down" is an observation, not a fence — a site that is down can come back.
5Select the recovery point for each tier explicitly, and never take "the latest". Name each by repository or snapshot identifier and timestamp, and state the gap it leaves. The set has to be mutually consistent: a database restored past the point at which the message queue was captured produces an estate that disagrees with itself about what happened.
6Get the recovery point selections signed off by the data owners before restoring anything. They are the only people who can say what the gap costs, and it is far cheaper to hear their answer now than after the point of no return.
7Establish time first. Clock disagreement breaks certificate validation, token lifetimes, log correlation and consensus, and it does so with error messages that point somewhere else. Confirm the recovery site's time source before the first service starts.
8Start in dependency order taken from the graph, not in order of business priority. Business priority says the revenue-bearing service goes first; the graph says that service cannot authenticate, cannot resolve, cannot decrypt and cannot reach its database yet. Sort the graph and follow the sort.
9Bring up identity, DNS and the secrets and certificate path as one tier, and validate them before anything depends on them. Most failed failovers stall here, and they stall in a way that looks like an application problem three tiers later.
10Restore storage and databases to their selected recovery points, and validate each before starting anything above it. A database that starts is not a database that is consistent; check the recovery target it actually reached, and read a row somebody can vouch for.
11Validate every tier before proceeding to the next, against a check that belongs to that tier. The temptation under time pressure is to start everything and see what breaks. That produces a cascade of failures whose first cause is four tiers down and invisible.
12Decision point — cut over now, or hold the recovery site warm and keep working. The criteria are in the body. Holding is a legitimate outcome: a recovery site that is running but not yet serving is a stable, reversible position, and it is the last one that is.
13Execute the cutover through the mechanism the estate owns, and account for what caches the old answer. Changing an authoritative record does not move traffic; it starts a timer. Long-lived connections, connection pools and resolvers that ignore short time-to-live values will keep arriving at a site you have just fenced.
14Record the point of no return the moment the recovery site accepts its first write. After that instant, returning to the primary is a failback with deltas to move, not a rollback. Say so on the bridge, in those words, so that everybody knows which procedure they are now inside.
15Validate the business from a client position outside the estate. Not that the process started and not that the health endpoint is green: a real transaction, exercised by a real business role, from outside, and confirmed by the person who owns that transaction.
16Re-establish protection at the recovery site before standing down. The recovery site is now production and is running on whatever backup schedule it inherited, which is frequently none. An estate that survives the disaster and then loses a day of work to an unprotected week has not finished this runbook.
17Record the declaration, the criteria, every recovery point, both decision points, the point of no return and the sign-off — then open the failback as a planned change. Failback is never an extension of the incident; it is a separate, scheduled procedure with its own recovery points.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The declaration is recorded with the declarer's name, the UTC time and the specific criterion that was met, and the elapsed time between onset and declaration is written down even where it is uncomfortable.
✓The primary is fenced and the fencing was proven from outside it: no answer on the service address, and no successful authentication against shared storage or the database from the primary's identities.
✓Every tier's recovery point is named by identifier and timestamp in the record, the set is stated to be mutually consistent, and each data owner's acceptance of the gap is recorded against their name.
✓The startup order executed matches the sorted dependency graph, and any deviation is recorded with the reason and the person who authorised it.
✓Each tier passed its own validation before the tier above it was started, and the evidence for each is in the log rather than summarised as "tier up".
✓The cutover mechanism reports the new target authoritatively, and the caches that matter have been observed to follow — measured from outside, not inferred from the time-to-live value.
✓The point of no return is recorded as a timestamp: the first write accepted at the recovery site, identified by what that write was.
✓A business transaction has been completed end to end from a client position outside the estate, using a real business role, and the transaction owner has confirmed by name that the result is correct.
✓Backups at the recovery site are running and have produced at least one recovery point that has been listed, so the new production site is protected before the incident is closed.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Rollback exists only before the recovery site accepts its first write. After that, the way home is the failback procedure, and treating it as a rollback is how the writes taken at the recovery site are lost.
↶To roll back before that point: stop the recovery site's services in reverse dependency order, revert the cutover mechanism, unfence the primary, and confirm the primary is serving before standing anybody down.
↶Reverse the fencing deliberately and in the record: unmask and re-enable every unit that was masked, restore every route or virtual address that was withdrawn, and re-issue every credential that was revoked. A masked service unit is invisible until the next reboot fails to bring it back.
↶Remove the monitoring silences created during the failover. A silence taken to quiet the primary's alarms is the commonest mutation left behind by this procedure, and it hides the next failure of the same component.
↶Revert speculative changes made while working the incident: firewall rules opened to reach the recovery site, resolver overrides, hosts-file entries, temporarily relaxed certificate checks and connection-pool settings changed to force a re-resolve.
↶Revoke every temporary identity created during the failover — recovery-team credentials, break-glass accounts, service identities issued at the recovery site — and record the revocation time for each.
↶Restore the original time-to-live values on any name whose TTL was lowered for the cutover, and record that they were restored. A production estate left on a thirty-second TTL carries a resolver load nobody budgeted for.

6 · Escalation

When the runbook isn't enough, contact:

· Nobody with declaration authority is reachable: escalate immediately to the incident owner and the business continuity owner. Proceeding without a declaration and delaying indefinitely are both bad outcomes, and choosing between them is not the responder's decision to make alone.
· The primary cannot be fenced: escalate to the network and platform owners before the recovery site is allowed to serve. Two sites serving the same names and writing to the same logical dataset is the worst outcome available in this runbook, and it is worse than a longer outage.
· The key material at the recovery site does not open the repositories: escalate to the key custodian and the incident owner in parallel. No amount of sequence work addresses this, and the estate needs to know within minutes rather than hours.
· The identity tier cannot be started at the recovery site: escalate to the identity owner. Everything above it will fail in ways that look like application faults, and responders will spend the recovery window debugging the wrong layer.
· Restores are running but the estate will miss the agreed recovery time: escalate to the incident owner with measured remaining work, so that continuing, degrading scope or holding becomes an explicit business decision rather than a default.
· The primary's failure has characteristics of a deliberate act: escalate to security before restoring anything into the recovery site. Restoring from a recovery point taken after a compromise reproduces the compromise at the site you just failed over to, and choosing the pre-incident recovery point is a decision security owns.

A failover is not something that happens to an estate. It is a decision somebody makes, followed by a sequence somebody rehearsed. This runbook covers both, for the case where the primary site is unavailable or untrustworthy and the recovery estate already exists.

Who declares, and on what criteria

Authority to declare belongs to a named role with a named deputy, and the criteria are agreed before the incident so that declaring is a lookup rather than an argument.

Observation at the primary	Estimate to repair in place	Declaration
Unreachable, cause unknown, past the agreed window	none available	Declare
Storage or site confirmed lost	irrelevant	Declare
Partial outage, cause understood	inside the agreed recovery time	Do not declare; keep repairing
Suspected compromise at the primary	irrelevant	Declare, and fence hard

The characteristic failure of this runbook is not a wrong step. It is a late declaration. An hour of “it might come back”, spent by people who each assumed somebody else was authorised to decide, is subtracted from the recovery window before the sequence has started — and it is then spent again as haste.

RBDR_INCIDENT=rbdr-dr-2026-08-28-01
RBDR_LOG=/srv/rbdr-dr/"$RBDR_INCIDENT".log
RBDR_DECLARER='a.tanaka (duty DR authority)'
RBDR_CRITERION='primary unreachable 42 min, no repair estimate'
mkdir -p /srv/rbdr-dr
printf '%s declared_by=%s criterion=%s\n' "$(date -u +%FT%TZ)" "$RBDR_DECLARER" "$RBDR_CRITERION" | tee -a "$RBDR_LOG"

Pre-flight: three things the recovery site must already hold

The catalogue, so recovery points can be found without the primary. The key material, proven by opening a repository rather than by confirming an escrow exists. And an identity bootstrap path — directory, secrets platform, certificate authority, time source — that does not route through the site that is gone.

RBDR_REPO=sftp:rbdr-recovery@vault.dr.example.net:/srv/rbdr-repo
RBDR_PASS=/mnt/rbdr-escrow/rbdr-recovery.pass
restic -r "$RBDR_REPO" --password-file "$RBDR_PASS" snapshots
getent hosts idp.dr.example.net
chronyc tracking

Fence the primary before anything else starts

Service impact possiblestop it serving, and stop it writing— Documented procedure; command form only. No transcript in this course covers a two-site estate, so no output is shown.

$ ssh rbdr-oob@primary-app-01.example.net 'systemctl stop rbdr-app.service && systemctl mask rbdr-app.service'

Masking matters because stopping alone leaves the unit startable by a dependency or a reboot, and the primary rebooting into service is precisely the event being fenced against. Fence at more than one layer: units, the route or virtual address, and the credentials the primary uses against shared storage and the database. Then prove it from outside.

Recovery point selection, tier by tier

Start in dependency order, not business priority

Business priority nominates the revenue-bearing service first. The graph points out that it cannot resolve, authenticate, decrypt or reach its database yet. Sort the graph and follow the sort.

Read-only / Safethe order comes from the graph, not from the bridge— Documented procedure; command form only. tsort orders the pairs; the file is the estate's own dependency list.

$ tsort /srv/rbdr-dr/rbdr-dependency-graph.txt

Each tier is validated with a check that belongs to that tier before the tier above it starts. Under time pressure the temptation is to start everything and see what breaks; that produces a cascade whose first cause is four tiers down and invisible from the top.

Decision point: cut over, or hold the site warm

Condition	Cut over	Hold and keep working
Every tier validated, identity included	Chosen	Unnecessary delay
A tier is up but unvalidated	Premature	Chosen
Data owner has not accepted the recovery point gap	Blocked	Chosen
Primary confirmed fenced	Required	Required either way
Primary may return and cannot be fenced	Unsafe	Chosen, and escalate

Holding is a real outcome and the last reversible one. A recovery site that is running but not yet serving costs nothing to unwind.

The cutover, and what caches the old answer

Changing an authoritative record does not move traffic; it starts a timer. Resolvers hold the previous answer for its remaining time-to-live, some ignore short values, and long-lived connections and pooled clients may not re-resolve at all until they are restarted.

RBDR_FQDN=app.example.net
RBDR_AUTH_NS=ns1.example.net
RBDR_PUBLIC_RESOLVER=9.9.9.9
dig +noall +answer "$RBDR_FQDN" @"$RBDR_AUTH_NS"
dig +noall +answer "$RBDR_FQDN" @"$RBDR_PUBLIC_RESOLVER"

The authoritative answer tells you what you published. The second tells you what a client is still being given, and the difference between them is the cutover in progress. Where the estate owns its routing, moving the address is faster than moving the name — and it has its own convergence to measure rather than assume.

The point of no return

The recovery site accepting its first write is the boundary. Before it, this procedure is reversible. After it, the primary holds an older dataset and the recovery site holds newer work, and coming home is a failback: a separate, scheduled procedure that moves deltas in the other direction. Record the timestamp and say the word failback on the bridge, so everybody knows which procedure they are now inside.

Abort criteria

Stop and hold, rather than continuing, when any of these is true:

The primary cannot be fenced and may return. Two sites serving the same names and writing the same logical dataset is worse than a longer outage.
The identity tier will not start at the recovery site. Everything above it fails in ways that look like application faults.
The key material does not open the repositories, so the selected recovery points cannot be restored.
A tier’s validation fails and the cause is unknown. Cutting over onto an unexplained fault moves the incident rather than resolving it.
The primary shows evidence of a deliberate act and no pre-incident recovery point has been chosen with security.

Holding is stable: primary fenced, recovery site up but not serving, no writes taken. It can be held for hours while somebody senior decides.

Business validation from a client position

Read-only / Safeexercised from outside, as a customer reaches it— Documented procedure; command form only. No transcript in this course covers this estate's application, so no output is shown.

$ curl -sS -o /dev/null -w '%{http_code} %{time_total} %{remote_ip}\n' https://app.example.net/rbdr-order-check

Run it from a host outside the estate, using an external resolver and a credential belonging to a real business role. A health endpoint answering from inside proves the process started. The recovery is finished when the transaction owner confirms, in business vocabulary, that a real transaction completed and the result is right.

What to record

The declaration: name, UTC time, criterion, and the elapsed time from onset — especially when that number is embarrassing, because it is the one that improves next time. Every recovery point by identifier and timestamp, with the accepted gap and who accepted it. Both decision points and the condition that settled each. The fencing actions and the proof they held. The point of no return, as a timestamp and as the write that crossed it. And the first backup taken at the recovery site, because it is now production.

Cross-course references

Linux for Production Sysadmins, Part LIV (Fencing and STONITH) — the fencing mechanics this runbook depends on, and why a node that is merely unreachable is not a node that has been fenced.
Linux for Production Sysadmins, Part LXIII (Cluster Time, DNS and Identity Dependencies) — the tier that has to start first and the failures it produces three tiers later when it does not.
PostgreSQL for Production Sysadmins, Part XV (High Availability, Failover and Disaster Recovery) — promotion at the database tier, and why the recovery target a standby actually reached is the number to check.
Secrets, PKI & Certificate Management, Part XV (KMS, HSM and Key Protection) — the custodianship that makes key material available at the recovery site when the primary is not.