RTO: everything between the incident and a working service

An RPO is a promise about the past — how much recent work an architecture is prepared to lose — derived from things the team controls directly: a schedule, a replication path, an archive interval. RTO is a promise about the future, and it is the one that gets broken, because it is derived from everything: monitoring, rosters, approval chains, spare capacity, a vendor queue, a DNS record and a person who must decide that a restore is warranted at all. Both are written in the same policy on the same afternoon, and usually only one was engineered.

The number in the policy is the runtime of one command

Ask a team where their four-hour RTO came from and the answer is usually the memory of a test: somebody restored a database onto a spare host, watched it finish comfortably inside an hour, applied a margin for whatever would go wrong on the night and wrote down four hours. The margin is not the problem. The thing being timed was the restore, and the restore is one term in a sum with a dozen others.

RTO is defined against the service, not the tool. The clock starts when the service stopped doing what the business needs — not when somebody noticed, not when a ticket was opened, not when the on-call engineer finally had a terminal. It stops when the service is doing that thing again and somebody has confirmed it. NIST SP 800-34 Rev. 1 derives recovery priorities from a business impact analysis and requires the recovery time objective to be set so that the maximum tolerable downtime is not exceeded: the number is bounded from above by what the organisation can absorb, and not from below by what a tool happens to do.

Those are two independent quantities, and a policy figure descended from neither is a coincidence rather than an objective. The upper bound is a business statement — past this point the event stops being an outage and becomes an existential problem — which nobody in the infrastructure team can derive. The lower bound is whatever the recovery path physically takes, which nobody in the business can derive. Engineering an RTO means measuring the lower bound honestly, holding it against the upper bound somebody else owns, and treating any gap as a funded decision rather than a number to be negotiated downwards in a meeting.

The stage that gets measured is therefore the only one behaving like a machine. The others are paced by people or by systems that queue, and neither gets faster because the target host has better disks.

Thirteen stages, and only one of them is the restore

Write out the recovery of one production service and count boundaries, not commands.

1. Detection. The gap between the service failing and anything establishing that it has. A failure that surfaces when a customer telephones has already spent it.
2. Escalation and decision. Somebody has to conclude that this is a recovery rather than a repair. An hour spent reviving a corrupted instance is an hour added to the budget.
3. Obtaining authorisation. Restores are destructive at the target and often need a named approver. If that person is not on the rota, the stage lasts as long as finding them takes.
4. Provisioning the target infrastructure. The machine, volume, cluster or namespace may not exist yet, and if it is built from code this stage includes finding whichever input the incident destroyed.
5. Retrieving the backup. Locating the right recovery point and, where the copy sits in an archive storage class, requesting a restore and waiting for the object to become readable at all. The Amazon S3 documentation is unambiguous that an archived object must be restored before it can be read.
6. Transferring it, over whatever link exists between the copy and the host that will restore it.
7. Decrypting and decompressing. Sometimes fused into the transfer as a stream; sometimes a separate pass needing scratch space that may not exist yet.
8. Restoring. The stage that was measured.
9. Replaying logs, to bring the restored copy from the state it was captured in to the state you actually want.
10. Starting dependencies in order. A restored database nothing can reach is not a service, and the wrong sequence usually means going round again.
11. Validating the data. Row counts, checksums, the most recent record — where a restore that “succeeded” is found to hold the wrong recovery point.
12. Validating the business transaction. Somebody performs the thing the service exists to do and confirms it worked end to end.
13. Cutting over DNS and routing. Balancers, service records and caches must point at the recovered system, bounded by TTLs and by clients that ignore them.

Four of these — 2, 3, 12 and part of 11 — are paced by people: their duration is set by who is awake and who is permitted to act. Most of the rest are strictly serial: you cannot decrypt bytes you have not transferred, nor replay a log onto a base image that has not finished restoring. A recovery is a chain, as long as the sum of its links.

The list is worth arguing about before it is worth timing, because each boundary has to be an event somebody can observe and stamp rather than an impression. Detection ends when an alert fires or a human says the words out loud, not when the graph first bent; authorisation ends when the approval is recorded, not when the approver was first paged. Boundaries defined loosely produce a budget whose rows cannot afterwards be reconciled against any log. The second discipline is completeness: a stage missing from the list is not worth zero minutes, it is merely unmeasured, and unmeasured stages have a habit of being the ones that consume the night.

Writing the budget down turns an opinion into arithmetic

A budget is a table with one row per stage, an estimate, and a note naming what paces it. Do it on paper before doing it with a stopwatch: the shape of the answer is usually visible immediately.

These estimates belong to one stated architecture and nothing else: a single 900 GB PostgreSQL primary with no standby; a nightly base backup and continuous write-ahead log archiving written to an object store; the most recent base backup living in an archive storage class; a recovery target built from Terraform into the same region; an effective 800 Mbit/s between store and target. The retrieval figure is an assumption of this example, not a property of any product.

The critical path sums to 2,242 minutes — 37 hours and 22 minutes — against a documented objective of four hours. The retrieval row alone is 1,800 of those minutes: 80 % of the budget, with every other stage combined making up the rest.

That ratio is why the table is worth drawing. Tuning the restore — the stage a vendor will happily sell an improvement to — cannot move the number: cut the restore, replay, decompression and transfer all to zero and the budget still reads 32 hours. The dominant row is not a performance problem but a placement decision taken months earlier by somebody optimising storage cost.

The rows also age at different rates, and that decides how often the table has to be redrawn. Transfer, decryption and restore scale with data volume, so they move whenever the dataset grows — the rows a capacity forecast already watches. Replay scales with the change accumulated since the base backup was taken, so it moves when write volume or base-backup frequency changes rather than when the database gets larger. Detection, decision, authorisation and validation scale with neither: they move when the roster, the approval policy or the validation script changes, and no capacity review looks at any of them. The pacing factor on each row is what names which change invalidates which estimate.

Four hours against thirty is not a tight RTO, it is a mismatch

A category error hides in the phrase “our RTO is aggressive”. An aggressive objective takes care, discipline and rehearsal to meet. A four-hour objective on top of a thirty-hour retrieval takes none of those, because no amount of care produces the number. It is arithmetically unreachable for the architecture that exists, and calling it aggressive keeps the fiction alive by making it sound like a matter of effort.

There are four honest responses. Change the architecture, so the newest recovery point lives where retrieval latency fits the promise while older points stay in the cheap tier. Change the promise, and tell the business four hours was never available at this cost. Change the scope by tiering: the ledger may need four hours while its seven-year archive needs three days. Or pre-stage — a warm target, escrowed keys, a standing approval — removing stages from the path before the incident.

None of the four is free, and what each costs is more informative than its wording. The first spends money every month on storage tiering to buy back hours on a day that may never arrive. The second spends credibility once, and it alone makes the published number true immediately. The third spends analysis, because somebody must decide which datasets sit in the fast tier and then defend the ones that do not. The fourth spends money and carries standing risk: a warm target is a second system to patch and monitor, and a standing approval is an approval nobody reviews. What pre-staging does not do is remove work: it moves work from inside the incident to before it, and it pays only if the pre-staged thing is verified on a schedule rather than assumed to still be there.

The only way to know the number is to measure it once

Every estimate above is a guess until a real recovery produces the row. The measurement is neither difficult nor expensive; it is simply never scheduled, because it produces a number somebody would have to act on.

Run the recovery onto infrastructure that does not already exist, so the provisioning row is real rather than assumed, and record a timestamp at each boundary as it happens — the human rows are the ones nobody remembers accurately afterwards.

DRILL_LOG="/var/log/dr-drill/$(date -u +%Y%m%dT%H%M%SZ).tsv"
mkdir -p "$(dirname "$DRILL_LOG")"

mark() {
  printf '%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" >> "$DRILL_LOG"
}

mark incident-declared
mark decision-made
mark authorisation-granted
mark target-provisioned
mark retrieval-requested
mark retrieval-complete
mark transfer-complete
mark restore-complete
mark replay-complete
mark dependencies-up
mark data-validated
mark transaction-validated
mark cutover-complete

Turning that log into a budget is a subtraction between consecutive lines, and the resulting table belongs in front of whoever funds the fix. Record the dataset size and the date beside it, because the number decays: the transfer and restore rows scale with volume, so a figure measured against 300 GB says nothing useful about 900 GB.

Read the human rows of a drill sceptically, though, because the conditions flatter them. A scheduled exercise is staffed by the person who knows the system best, in daylight, with the approver already in the room and nobody arguing about whether this really counts as a disaster. The stages a drill measures most generously are exactly the stages that dominate a genuine recovery at three in the morning. Two cheap corrections restore some honesty: record for each stage who performed it and whether they were the primary expert, and run at least one exercise without telling the on-call rota in advance. A budget that keeps the machine rows separate from the human rows is worth more than a single total that conceals both.

Production discipline

1. Start the clock at service loss, not at the terminal. An RTO beginning when the engineer starts typing has already excluded detection, escalation and authorisation — frequently the cheapest stages to improve.
2. Publish the budget as a table with the pacing factor on every row. A single number invites argument; thirteen rows move the conversation to the row that matters and make the dominant stage impossible to hide.
3. Identify the dominant row before optimising anything. Above, retrieval is 1,800 of 2,242 minutes, so reducing every other stage to zero still leaves 30 hours.
4. Treat an unreachable objective as an architecture decision, not an effort problem. Move the newest recovery point to a tier whose retrieval fits the promise, tier the promise, or change the number in the policy.
5. Measure the whole path once, on infrastructure provisioned during the drill. Re-measure when the dataset grows or the storage tier changes, and treat any stage you have never timed as unbounded rather than small.

Cross-course references

• Observability for Production Sysadmins — Part XIX (Alertmanager) covers the routing, grouping and notification that set the floor under the detection and escalation rows of this budget; both are decided in the observability stack long before any backup tool is invoked.
• Kubernetes for Production Sysadmins — Part XCIX (Complete Cluster Loss) works through a recovery in which the provisioning, dependency-ordering and cutover rows dominate rather than the restore — this same budget shape applied to a control plane.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XCVII (CI/CD Disaster Recovery) matters here because the provisioning row assumes a pipeline and artifact store that still work; when recovery runs through the delivery system, that system’s own recovery time is a term inside this table.

Quiz