RPO, RTO, and dependency modelling · Proxmox VE

Why this matters in production

DR plans fail because they assume dependencies that don’t exist, ignore the people who must execute them, or optimise the wrong axis. This lesson teaches the discipline.

Definitions

RPO (Recovery Point Objective): the maximum acceptable data loss in time.
RTO (Recovery Time Objective): the maximum acceptable downtime.
MTTR (Mean Time To Recover): how long recovery actually takes in practice.
MTBF (Mean Time Between Failures): how often the system fails.

The first two are targets. The last two are measurements. Plan with the targets; verify with the measurements.

Per-system targets

System	RPO	RTO
Customer-facing primary database	5 min	15 min
Customer-facing API	15 min	30 min
Internal tools	4 h	8 h
Dev environment	24 h	48 h

The RPO improvement that improved nothing

The situation. A team takes nightly PBS backups and syncs them to the DR site hourly. Asked to improve the RPO, they raise the sync frequency to every fifteen minutes. Bandwidth use goes up. The RPO does not move.

Why. PBS sync moves chunks, and chunks only exist once a backup has created them. The RPO is set by the backup schedule, not by the sync schedule. A nightly backup synced every fifteen minutes still leaves up to 24 hours of writes unprotected, and 95 of every 96 sync runs transfer nothing.

Why the first diagnosis is wrong. Sync frequency is the visible knob - it is on the job configuration, it has a schedule field, and it plainly runs more often afterwards. The constraint is one layer upstream and has no field on that page.

The general rule, worth applying to any replication mechanism: an RPO is set by the cadence at which the mechanism learns what changed. For backup-and- sync that is the backup. For ZFS pvesr it is the replication schedule, because that is when the snapshot is taken. For journal-based mirroring there is no schedule at all, because every write is the event.

The check:

# When did each guest actually last get a NEW backup - not a sync?
pvesh get /cluster/backup --output-format yaml | grep -E 'schedule|vmid|enabled'
pvesm list pbs-main --content backup | tail -20

If the newest snapshot for a guest is older than its stated RPO, the RPO is fiction regardless of how often anything is synced.

Writing an RTO that has arithmetic behind it

An RTO with no computation behind it is a wish, and it is usually optimistic by a factor of two. The computation is not difficult; it is just rarely done.

Read-only / Safethe recovery-time model - fill in your own measured numbers— Read-only arithmetic. The only input you must not guess is THROUGHPUT_MIBPS, which comes from timing one real restore.

set -euo pipefail

GUEST_GIB=512          # data to be written back
THROUGHPUT_MIBPS=110   # MEASURED restore throughput, not link speed
DETECT_MIN=10          # monitoring notices, someone is paged
DECIDE_MIN=15          # confirm, escalate, authorise
BOOT_MIN=4             # guest boots and services start
VERIFY_MIN=10          # someone confirms the service actually works

TRANSFER_MIN=$(( GUEST_GIB * 1024 / THROUGHPUT_MIBPS / 60 ))
TOTAL_MIN=$(( DETECT_MIN + DECIDE_MIN + TRANSFER_MIN + BOOT_MIN + VERIFY_MIN ))

printf 'transfer  %4d min\n' "$TRANSFER_MIN"
printf 'non-transfer %d min\n' "$(( TOTAL_MIN - TRANSFER_MIN ))"
printf 'total RTO %4d min  (%d h %02d m)\n' \
"$TOTAL_MIN" "$((TOTAL_MIN/60))" "$((TOTAL_MIN%60))"

Three observations that change how people write RTOs, and all three come out of that script rather than out of an opinion:

Throughput is not link speed. A restore reads chunks from the datastore, decompresses and decrypts them, and writes them to the destination storage. Any of those can be the constraint. Measure it once and record it; the detail is in live restore and the RTO arithmetic.

The non-transfer terms are large and nobody budgets them. Detection, decision and verification here total 35 minutes - and on a small guest they dominate completely. An organisation whose bottleneck is finding someone authorised to say yes does not get faster by buying more bandwidth.

Parallel restores do not scale. Twenty guests restoring at once share one datastore read path and one link. A DR plan that assumes twenty simultaneous full-speed restores assumes twenty times the bandwidth you have, which is why recovery order is a real design artefact and not a formality.

Dependency mapping

Every system depends on others:

flowchart TB
  DNS[DNS] --> APP[Application]
  IDM[Identity] --> APP
  DB[(Database)] --> APP
  APP --> LB[Load balancer]
  LB --> USER[End users]
  APP --> LOG[Log aggregator]
  LOG --> MON[Monitoring]

A complete dependency map identifies:

Hard dependencies: the system cannot function without them.
Soft dependencies: the system degrades but works.
Recovery order: what must come up first, second, etc.

The dependency map is the DR plan’s foundation. Without it, you cannot sequence recovery.

Recovery order

A typical recovery order:

Network and DNS (external resolution).
Identity (LDAP / Active Directory).
Database servers.
Application servers (in dependency order).
Load balancers / API gateways.
Monitoring (so you can see what works).
Internal tools.
End-user-facing systems.

Time budgets

For each phase of recovery, set time budgets:

Phase	Target
Disaster declaration → restore plan	< 30 min
Network + DNS restored	< 1 h
Identity restored	< 2 h
Database restored	< 4 h
Application restored	< 8 h
All services restored	< 24 h

If a phase exceeds budget, escalate.

What happens if the dependency map is wrong in one place?

The recovery stalls at that point and everything downstream of it waits, and the reason is rarely obvious because the missing dependency is usually something nobody thinks of as a system.

The dependencies that get omitted, consistently:

DNS resolution for the infrastructure itself. The PBS server is reached by name. The name is served by a resolver at the primary site. The primary site is the disaster. Now the DR cluster cannot find its own backups, and everyone is looking at storage.
Certificate authorities and time. Services validate certificates; TLS validation fails on a machine whose clock is wrong; NTP was one of the systems you have not restored yet. This produces authentication errors that look like credential problems for a long time before anyone checks the clock.
Licence servers. Software that phones home to a licence server which was in the destroyed site starts in evaluation mode, or refuses.
The identity system, in both directions. Restoring identity early is standard - but if the restore tooling itself authenticates against that identity system, there is a loop, and the credential that breaks it must be a local one that exists before identity does.
Monitoring. Not because anything depends on it, but because without it you cannot tell whether recovery is succeeding. Bring it up as soon as there is a network.

The test that finds these is not a review of the diagram. It is a tabletop where somebody asks “and how do you reach that?” at every step, and the answer has to be a mechanism rather than a name.

The written product is a recovery order with reasons, not just an order:

#	System	Depends on	Breaks if skipped
1	Network, routing	Nothing	Everything
2	DNS resolver at DR	1	PBS is unreachable by name
3	NTP	1, 2	TLS validation fails estate-wide
4	PBS storage attach	1, 2	No restores at all
5	Identity	4	Most applications refuse logins
6	Monitoring	1, 2	You are recovering blind
7	Databases	4, 5	Applications start and fail
8	Applications	7	—

The third column is what makes it useful under pressure. An order with no reasons gets reordered by whoever is in the room.

Production considerations

Common mistakes

Single RPO/RTO for the entire cluster.
Ignoring soft dependencies.
Not budgeting time per phase.

Key takeaways

Per-system RPO/RTO, not cluster-wide.
Dependency maps are the foundation of DR.
Rehearse quarterly.

Knowledge check

Why this matters

RPO and RTO are the only two numbers in disaster recovery that the business side of the organisation will actually read, which makes them the place where engineering honesty either happens or does not.

The failure is not usually a bad architecture. It is a number quoted without arithmetic - an RTO of thirty minutes that has never been measured, an RPO of five minutes that describes a sync schedule rather than a backup schedule. Those numbers then propagate. They go into an SLA, a customer contract, a board paper, and a staffing plan. By the time the real number is discovered, several commitments have been made on the wrong one.

Both are cheap to establish properly. One timed restore per quarter gives you a throughput figure. One honest look at when each guest last received a new backup gives you the real RPO. Twenty minutes of arithmetic converts two comfortable fictions into two numbers you can defend - and handing the true number to whoever wrote the SLA is usually worth more than any tuning you could do to improve it.