Measuring the restore time you actually have

Intermediate⏱ ~17 minrbdceph

What you'll learn

Instrument a restore drill by phase
Identify which phase actually dominates
Measure under contention rather than in isolation
Feed the measurement back into the architecture

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An RTO nobody has measured is an estimate of the transfer time, and the transfer is rarely the largest term. Drills exist to replace the estimate with a number.

Instrumenting by phase

Phase	Typically assumed	Frequently measured
decide to restore	0	45 min
locate and validate the backup	5 min	40 min
provision the target	10 min	90 min
transfer the data	60 min	75 min
application-level recovery	15 min	110 min
validate and cut over	10 min	60 min
total	1 h 40	7 h

ts() { date -Is; }
ts; rbd import /backup/vm-104.img rbd/vm-104-drill; ts
rbd -p rbd du vm-104-drill

/usr/bin/time -v rbd import /backup/vm-104.img rbd/vm-104-drill \
  2> /secure/drills/2026-Q3-import.log

Measuring under contention

ceph -s | grep -A2 'io:'
ceph osd pool stats rbd

A restore into an idle lab measures the tool. A restore into a cluster carrying production load measures the design — the same import competes for the same OSDs the surviving workloads are using, and recovery traffic may be running alongside it.

Condition during the drill	Effect on the measured transfer
idle cluster	optimistic; the number you will not get
production load	realistic for a partial failure
production load plus backfill	realistic for the event you are rehearsing
restore target on the same OSDs as the source	the two compete directly

# what headroom exists before the drill starts
ceph osd pool stats
ceph -s | grep -E 'recovery|client'

Removing the crutches

Perform the drill without the primary site's DNS, its credential store,
its orchestration, and its runbook. Anything the drill borrows from the
running estate is a dependency it did not test.

# resolve by address, authenticate from the sealed offline copy
curl -sf https://203.0.113.40/healthz
rclone --config /secure/drill/rclone.conf copy offsite:acme-backup/vm-104.img /restore/

What the measurement changes

Finding	Architectural response
decision time dominates	pre-authorise restore for named scenarios
provisioning dominates	keep a pre-provisioned target
transfer dominates	add a local tier or more bandwidth
application recovery dominates	more frequent consistent points
validation dominates	write the validation script in advance

ceph df
rbd -p rbd rm vm-104-drill

Rotate the scope: each quarter, drill something that has not been drilled before, and prefer whatever nobody has ever restored.

Quiz

Knowledge check · 4 questions

Q1. Why record each phase of a restore drill separately?

To compare backup tools against one another, since a per-phase breakdown is the only way to see which product handles metadata, sparse regions and interrupted transfers better, and a total hides all three To satisfy the audit requirement for evidence that every documented stage was exercised, which a single end-to-end figure cannot demonstrate however favourable that figure happens to be Because the phases respond to different fixes — decision time needs pre-authorisation, provisioning needs a warm target, transfer needs bandwidth — and a single total says nothing about which to change Because the total is unreliable whenever the drill runs against a loaded cluster, so only the individual phases can be compared honestly between one quarter and the next
Q2. A restore drill that uses the production site's DNS, credential store and orchestration has not tested the DR design.

True False
Q3. Convert a claimed RTO into a measured one.

The DR document claims a two-hour RTO for a 4 TiB RBD image. No restore has been performed. The team proposes to validate it by timing an `rbd import` in the lab.
Q4. What should a restore drill deliberately refuse to use?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Timestamp every phase and publish them separately; a single total hides which of decision, provisioning, transfer, recovery and validation is the one to fix. Refuse the production estate’s DNS, credentials and orchestration during the drill — anything borrowed is a dependency that was never tested.

Cross-course references

Kubernetes: a restore rehearsal on a healthy control plane exercises none of the failure it rehearses
Linux: measure each stage; totals conceal the stage that is worth optimising