RPO and RTO — defining recovery objectives · Docker & Containers

Two definitions, and they are not symmetric:

RPO — Recovery Point Objective. The maximum acceptable amount of data loss, expressed in time. RPO 1 hour means that after a disaster you may be missing up to an hour of work, and that is acceptable.
RTO — Recovery Time Objective. The maximum acceptable outage, expressed in time. RTO 4 hours means the service is expected back within four hours.

The asymmetry that matters: RPO is bought with backup frequency, RTO is bought with infrastructure, and the two are unrelated purchases. Backing up every five minutes improves RPO to five minutes and improves RTO by exactly nothing — you still have to provision a host, pull images and replay a dump. Teams who believe more frequent backups make recovery faster are budgeting for the wrong thing, and they find out during a drill or an outage.

Deriving the numbers instead of picking them

“RPO 1 hour, RTO 4 hours” appears in a great many documents and is usually a number that sounded responsible. The derivation is not difficult and it changes the answer.

The arithmetic, worked for a real shop

Take an e-commerce stack on one Docker host. Revenue is 8,000 currency units per hour during the working day.

Cost of downtime. Direct lost revenue is 8,000/hour. Add support cost, add the fraction of customers who go elsewhere permanently — a conventional multiple is 1.5x for a consumer service. Call it 12,000 per hour of outage.

Cost of data loss. This is the one people get wrong, because it is not “revenue during the lost window”. A lost hour of orders means orders that were paid for and are gone: the customer has been charged and has no order. The cost is the refund, plus the support contact, plus reputational damage — routinely 3 to 10 times the revenue in that window. Call it 40,000 per lost hour.

Now the engineering side, per year:

Change	Buys	Annual cost
Nightly dump to S3	RPO 24 h, RTO ~6 h	200
Hourly dump	RPO 1 h	+400
WAL archiving to S3	RPO ~5 min	+1,200
Warm standby host, images pre-pulled, volumes synced	RTO ~30 min	+9,000
Streaming replica with manual promotion	RPO seconds, RTO ~10 min	+14,000

Against 40,000 per lost hour, WAL archiving at 1,200/year is not a close call — it pays for itself if it prevents twenty minutes of data loss once in a decade. Against 12,000 per outage hour, a warm standby at 9,000/year that converts a six-hour recovery into thirty minutes saves 66,000 per incident and pays back on its first use.

The general shape: RPO is usually cheap to improve and expensive to get wrong. RTO is the opposite — improving it means paying for idle capacity all year. So the common correct answer for a small deployment is an aggressive RPO with a modest RTO, which is precisely the combination people rarely write down because “RPO 5 minutes, RTO 6 hours” looks inconsistent. It is not inconsistent; it is what the arithmetic says.

Decomposing RTO for a single Docker host

RTO is not one number, it is a sum, and the sum is dominated by one or two terms that are rarely the ones people optimise.

Phase	Typical, unprepared	Typical, prepared	What changes it
Detect	5–30 min	1–2 min	Alerting on the service, not on the host
Decide to fail over	10–60 min	2 min	A written trigger condition and a named decider
Provision a host	30–120 min	0 (already running)	Warm standby, or an image and IaC
Install and configure the engine	15–40 min	0	Pre-built host image with `daemon.json`
Pull images	5–30 min	0	Pre-pulled, or a local registry mirror
Restore volume data	20 min – 6 h	5–20 min	Volume size, transfer rate, restore parallelism
Start the stack and reach healthy	2–15 min	2–15 min	Health check start periods, migrations
Repoint DNS or the load balancer	1 min – 1 h	1–5 min	TTL, and whether the change is automated
Verify	10–30 min	5 min	A written content assertion, not a look around

Two lines deserve attention because they are consistently underestimated.

Decide to fail over is often the largest term in a real incident and appears in no plan. The host stopped responding at 02:14; at 02:47 the on-call engineer is still trying to determine whether it will come back, because failing over is irreversible-ish and nobody wants to be the person who did it unnecessarily. Thirty-three minutes of RTO consumed by an absent decision rule. The fix is a written trigger: if the host has not responded to SSH for 10 minutes and the out-of-band console shows no progress, we fail over. Put a name against it.

Repoint DNS is bounded below by your record’s TTL, and resolvers routinely ignore short TTLs. A 3,600-second TTL is an hour of RTO you cannot recover by working faster. Lower it to 60 seconds now, permanently, as a standing decision; the extra query volume is negligible and it converts an hour into a minute.

Read-only / Safemeasure your own terms— Real numbers for the phases you can measure without an outage.

STACK=/srv/app

# How long does the stack take from cold to all-healthy?
docker compose -f "$STACK/compose.yaml" down
time docker compose -f "$STACK/compose.yaml" up -d --wait

# How much data has to move?
docker system df -v | sed -n '/Local Volumes/,$p'

# What is the DNS TTL you are actually bound by?
dig +noall +answer app.example.com | awk '{print $2, $1}'

docker compose up -d --wait blocks until every service with a health check reports healthy, and returns non-zero if any does not. That makes it a measurement rather than a guess: the number it prints is the “start the stack and reach healthy” row of your table, on this host, today.

Read-only / Safemeasured, not estimated— One command turns a table row from an assumption into a number.

$ time docker compose -f /srv/app/compose.yaml up -d --wait

[+] Running 5/5
✔ Network app_default      Created
✔ Container app-db         Healthy
✔ Container app-redis      Healthy
✔ Container app-web        Healthy
✔ Container app-caddy      Healthy

real    3m48.114s
user    0m0.412s
sys     0m0.180s

Illustrative output

Three minutes forty-eight, with images already present and volumes already populated. Any RTO commitment shorter than that is arithmetically impossible before a single byte has been restored, and this is a four-minute experiment.

What a single Docker host can actually promise

Architecture	Achievable RPO	Achievable RTO	Honest description
One host, nightly dump offsite	24 h	4–8 h	Adequate for internal tools
One host, hourly dump offsite	1 h	4–8 h	RPO improved, RTO untouched
One host, WAL archiving offsite	5 min	4–8 h	Cheapest large RPO improvement available
Two hosts, cold standby provisioned	5 min – 1 h	30–60 min	Provisioning and engine install removed
Two hosts, warm standby, volumes synced hourly	5 min	15–30 min	Data restore mostly removed
Two hosts, streaming DB replica, manual promotion	Seconds	5–15 min	The realistic floor for standalone Docker
Anything claiming automatic failover	—	—	Not standalone Docker. See the HA part.

The last row is the boundary. Standalone Docker has no mechanism that notices a host has died and starts the workload elsewhere. restart: always restarts a container on a host that is still running its daemon; it does nothing at all when the host is gone. An RTO in single-digit minutes therefore requires a human who is awake, or an orchestrator. Committing to one without either is committing to a coin flip on whether the outage starts during working hours.

The four ways a stated RPO is quietly wrong

1. The backup runs at 02:00 and takes until 03:30. The RPO is measured from the end of a consistent capture, not from the schedule. A 90-minute dump means your worst case is 25.5 hours, not 24.

2. The backup has never been restored. An untested backup has no RPO, because RPO presumes recoverability. Until a restore has succeeded, the honest number is “unknown”.

3. Only the database is in scope. The database has an RPO of one hour and the uploads volume is copied weekly. Your RPO is one week — the system’s RPO is the worst of its components, not the best.

4. The offsite copy lags the local one. The dump completes locally at 03:30 and the sync to S3 runs at 06:00. If the host is lost at 04:00 you have last night’s offsite copy, not this morning’s local one. RPO is measured against the copy that survives the disaster you are planning for.

Every one of these makes the true RPO worse than the stated one, and none produces an error message. Compute the number from timestamps on the surviving offsite copy, not from the crontab.

Read-only / Safethe RPO you actually have— Age of the newest verified copy that would survive host loss.

export RESTIC_REPOSITORY='rest:http://backup-01.example.com:8000/app-01'
export RESTIC_PASSWORD_FILE=/etc/restic/password

# Newest snapshot in the OFFSITE repository, per backup path
restic snapshots --host app-01 --latest 1

# Same thing as an age in hours, for a monitoring check that can fail
newest=$(restic snapshots --host app-01 --latest 1 --json | jq -r '.[0].time')
age_h=$(( ( $(date +%s) - $(date -d "$newest" +%s) ) / 3600 ))
echo "newest offsite snapshot is ${age_h}h old"
[ "$age_h" -le 24 ] || { echo 'FAIL: offsite copy exceeds stated RPO' >&2; exit 1; }

Run that and compare the output to the RPO in your documentation. The gap between them is your actual exposure, and it is usually a surprise.

What if the business asks for RTO 15 minutes on a single-host stack?

The wrong answer is to agree and hope. The right answer is to show the decomposition and let them choose which term to buy down.

Take the table apart in front of them:

Detect: 2 min, achievable with alerting you already have.
Decide: 2 min, achievable with a written trigger and a named decider. Free.
Provision + engine install: 45 min unprepared, 0 with a standby host that is already running. This is the item that costs money.
Pull images: 10 min unprepared, 0 with images pre-pulled on the standby. Free, if the standby exists.
Restore data: 40 min for a 30 GB restore from S3, 5 min from an hourly volume sync to the standby. Cheap, if the standby exists.
Start and verify: 6 min. Irreducible.

Sum: 15 minutes is reachable — but only with a second host that is already running, already has the images, and already has recent data. Everything else on the list is free.

So the conversation is not “can we do 15 minutes”, it is “15 minutes costs one additional host, permanently, plus the discipline to keep it warm; 4 hours costs nothing extra”. That is a decision the business can actually make, and it is a far better conversation than the one where you agreed to 15 minutes and deliver 6 hours during the incident.

State the residual risk too: even with the standby, failover is manual. If the host dies at 03:00 the clock starts when a human wakes up. Fifteen minutes is fifteen minutes from the decision, not from the failure — and if that distinction is unacceptable, the requirement is an orchestrator, not a better runbook.

Knowledge check

Why this matters

RPO and RTO get treated as documentation, which is why they are wrong so often: a number in a document costs nothing to write and is never contradicted until the day it matters.

Two habits make them real, and neither is expensive. Derive them — from someone outside engineering, in currency per hour, written down with the assumption behind it. Measure them — time docker compose up -d --wait for the startup term, the age of the newest offsite snapshot for the RPO term, and a timed drill for the whole thing.

The single most valuable output of doing this is usually not a better number. It is discovering that the largest term in your RTO is the thirty minutes somebody spends deciding whether to declare a disaster, and that removing it costs one sentence in a runbook and one name.