Estimating restore duration from evidence · Backup & DR

The previous lesson defined RTO as the interval from incident to working service and left one term inside that interval deliberately unmeasured: how long the restore itself takes. That term is the one everybody estimates and almost nobody measures, and it is where a confident number does the most damage. A plan built on four hours does not fail at hour four. It fails at hour five, with the copy a third of the way through and every step that was supposed to follow it still unstarted. What follows is not a duration. It is a method for producing one, and a discipline for writing it down in a form somebody else can check.

The floor is volume over throughput, and it is only a floor

Every restore has an arithmetic minimum. The bytes have to be read from wherever they are kept and written to wherever the service will run, and no amount of engineering makes that free. Dividing the volume by an effective throughput gives that minimum. It is worth computing before anything else, partly because it is the only part of the estimate that is not a judgement call, and partly because it frequently settles the question on its own: if the floor already exceeds the RTO, no amount of tuning downstream will rescue the plan and the architecture has to change instead.

Effective throughput is not the figure on a device datasheet and not the figure in a vendor’s marketing page. It is what you measured, end to end, along the path the restore will actually take — repository backend, network, restoring host, target volume — with the slowest link in that path setting the value. A number you did not measure yourself is a placeholder, and a placeholder should look like one in the runbook until it is replaced.

# A floor, not an estimate: bytes to move divided by a throughput figure you
# measured yourself along the path the restore will take.
VOLUME_GIB=800
THROUGHPUT_MIB_S=180
floor_seconds=$(( VOLUME_GIB * 1024 / THROUGHPUT_MIB_S ))
printf 'floor %s s (%s min) for %s GiB at %s MiB/s\n' \
  "$floor_seconds" "$(( floor_seconds / 60 ))" "$VOLUME_GIB" "$THROUGHPUT_MIB_S"

The result of that arithmetic must be labelled a floor everywhere it appears. Writing restore: 1.3 h into a recovery plan, when 1.3 hours is what it would cost if moving bytes were the only work involved, quietly converts a lower bound into a commitment. Everything in the next section is the difference between those two things, and in most estimates the difference is larger than the floor.

Seven things that sit between the floor and the wall clock

Decryption and decompression are CPU work. In an encrypted repository every block that comes off the backend has to be decrypted and its authentication tag verified before it can be written to the target, and in a compressed repository it has to be inflated as well. That work is bounded by the restoring host’s cores and by how much of the tool’s read path is actually parallel. A restore can sit at a fraction of the disk’s capability with the CPU pinned, which is visible during a rehearsal and invisible in the plan.

Small files and large files are different workloads. Streaming one 500 GiB disk image is the case closest to the floor, because it is one long sequential write. Two hundred million files averaging 4 KiB is a different problem entirely: each one needs a directory entry, an inode, ownership, permissions, timestamps and possibly extended attributes. The unit of that workload is files per second, not mebibytes per second, and the two do not convert into each other. Estimating a small-file tree from a bulk throughput figure is the single most common way a restore estimate ends up wrong by an order of magnitude.

Metadata operations extend beyond the files themselves. Directory trees have to be created before their contents, hardlinks reconstructed, ACLs and extended attributes reapplied, sparse regions handled, and the tool’s own index consulted for every object it places. None of this moves user data and all of it consumes wall clock.

Concurrency is bounded on both ends. Every restore tool has some ceiling on how many objects it fetches and writes at once, backends impose their own request limits, and a single stream will not fill a wide network path. What those ceilings actually are is a property of the tool version and the backend in front of you, and none of them appear in the floor calculation. Worse, parallelism that helps at the repository can starve the target: many concurrent writers to one spinning volume produce a seek pattern slower than a single sequential writer would have been, so tuning that halves the fetch time can lengthen the restore overall. The only way to find the setting that helps is to vary it during a rehearsal and time the result.

Some data has to be rehydrated before it can be read. Objects held in an archive storage class must be restored to an accessible state before they can be downloaded, and that step is a queue rather than a transfer — its duration is not a function of your bandwidth at all. A recovery plan that reads from an archive tier has to carry that latency as an explicit, separate line item ahead of the transfer, because no throughput improvement shortens it.

The target has a write path of its own. The restore is only as fast as the volume receiving it. A freshly provisioned volume on shared infrastructure, a network filesystem, a RAID level that amplifies writes, or a target that is simultaneously serving other work will each cap the restore below whatever the source could deliver. Measuring the source and assuming the target keeps up is the second most common estimation error.

Validation is part of the duration. The restore is not finished when the tool exits; it is finished when somebody has established that what landed is correct. Checksum comparison, database consistency checks and an application smoke test all take time, and they are not optional — the previous part of this course showed a restore that exited non-zero and still left a corrupt file on disk with a plausible name and a plausible size. If verification is what turns a copy into a recovery, then verification belongs inside the number, and the rehearsal that produced the number has to have included it.

Measured: 1.370 KiB written, 60.000 MiB to read back

The claim that backup duration predicts restore duration is testable on a repository small enough to reason about completely. A restic repository was created and two backups were taken from a 60 MiB source tree of three files.

Service impact possibleinitialise, then the first backup — everything is new— EXECUTION VALIDATED on restic 0.19.1. Both commands are shown because the first output line comes from the init, not the backup; the source tree was three files totalling 60 MiB.

$ restic init && restic backup /work/prod --tag daily

repository initialised
no parent snapshot found, will read all files

Files:           3 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 60.005 MiB (60.008 MiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3fe43af4 saved

>>> exit code: 0

One file was then modified and the backup was repeated. This is the shape of every night after the first one, in every incremental backup system.

Service impact possiblethe second backup — one file changed out of three— EXECUTION VALIDATED on restic 0.19.1, run against the same repository immediately after the capture above.

$ restic backup /work/prod --tag daily

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved

>>> exit code: 0

Read those two numbers next to each other. The second run added 2.062 KiB to the repository, 1.370 KiB of it stored, and it produced a snapshot that represents the state of a 60.000 MiB tree. Whatever that run cost in time, I/O and bandwidth, it is a measurement of a few kilobytes of delta. The restore of that snapshot is a different quantity entirely.

Destructivethe restore — the whole tree, not the delta— EXECUTION VALIDATED on restic 0.19.1, same capture as the two backups above. Snapshot 3fe43af4 restored into an empty scratch target; a restore overwrites whatever is at the target path.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00

>>> exit code: 0

The restore reports 7 files/dirs, 60.000 MiB. There is no incremental restore of a snapshot: the snapshot is a complete view of the tree, so reconstructing it means locating and writing every chunk it references, whether that chunk was stored last night or on the day the repository was initialised. Scale the ratio up and the asymmetry becomes the whole problem. A nightly backup that moves a 40 GiB delta of a 30 TiB estate measures 40 GiB. The restore measures 30 TiB, and nothing in the backup’s timing tells you what the second one costs.

Why a deduplicating repository turns one file into many partial reads

A deduplicating repository does not store files. It splits every file into content-defined chunks, hashes each chunk, and writes the chunk only if that hash is not already present. Chunks are then grouped into pack files, and a pack is written out once enough of them have accumulated — which means the chunks inside it are whichever ones happened to be new at that moment, from whatever files were being read at the time.

The measured repository above holds a 60.000 MiB source tree in seven packs; restic check --read-data reports its progress across them as 7 / 7 packs, and the largest of those packs was 17374653 bytes. Packs, not files, are the unit the repository actually manages.

Now invert that arrangement, which is what a restore does. To rebuild one file, the tool reads the ordered list of chunk hashes from the tree object, resolves each hash to the pack that holds it, and fetches it. Nothing guarantees that those chunks are adjacent, that they live in the same pack, or that they were written by the same run. A file whose contents drifted over months has its chunks distributed across the packs of every run that saw a change, alongside chunks belonging to entirely unrelated files.

On local disk that costs seek time and read amplification — pulling a whole pack for the few kilobytes of it you need. On object storage it costs one request per pack, each carrying its own round trip, and the restore becomes latency-bound rather than bandwidth-bound. That is why adding bandwidth to a slow object-storage restore frequently changes nothing at all.

The mechanism that produces a good deduplication ratio is precisely the mechanism that scatters the reads: the more sharing there is between files and between snapshots, the fewer bytes are stored and the less locality any single file retains. A repository reporting excellent space savings is making a statement about its write path, and no reassuring statement whatsoever about its read path.

Backup duration and restore duration measure different things

Once the mechanism is clear, the reasons the two figures diverge stop being a list of surprises and become predictable.

Backup after the first run is incremental and restore is not. The backup skips unchanged files after a cheap comparison; the restore writes every byte. Backup reads from production storage that was sized for production load, while restore writes to replacement infrastructure that may have been provisioned minutes earlier. Backup runs across a fleet in parallel, so its wall-clock figure hides per-host cost inside overlapping jobs; a restore is usually one target, one stream of decisions, one bottleneck. Backup runs in a quiet window chosen for low contention; a restore runs during an incident, when the network is degraded, the platform team is doing five other things, and possibly several restores are competing for the same repository.

Add the scatter described above and the two numbers routinely differ by an order of magnitude, in the direction nobody wants. This is why “our backups finish in forty minutes” is not evidence about recovery. It is evidence about backups. The only evidence about restore duration is a restore that happened, timed with a clock.

Three ways a restore estimate becomes fiction

“The backup window is twenty minutes, so a restore is roughly twenty minutes.” The measured capture above added 1.370 KiB of stored data on its second run while producing a snapshot of 60.000 MiB. The backup measured a delta; the restore has to reassemble the whole tree. The two are not the same quantity and cannot be substituted for one another.

“The replica is our backup, so we just fail over — the restore time is zero.” A replica gives you a fast path for one failure mode and no path at all for the ones that replicate: a deletion, a bad migration, an encrypted filesystem. The restore estimate you skipped is exactly the one you will need on the day the replica faithfully reproduces the damage.

“We restored 10 GiB in six minutes last year, so 2 TB is about twenty hours.” Linear extrapolation across two orders of magnitude assumes the bottleneck stays where it was, and it rarely does: at 10 GiB much of the working set can sit in page cache and the target is never pushed for long, and at 2 TB neither of those is likely to hold. The small measurement was real. It was a measurement of a different bottleneck.

An estimate that names its volume, path, hardware and date

A restore duration is a measurement, and a measurement whose conditions are not stated cannot be reproduced, checked or invalidated. Four elements are the minimum for a figure to be worth writing down.

The volume it applied to, in bytes and in file count, because those two numbers select which of the multipliers above dominate. The path it travelled: which repository, which storage tier, across which network, onto which target — a figure measured from a local cache is not a figure for the off-site copy. The hardware and backend it ran on, because throughput is a property of that specific combination and of nothing else. And the date, because every one of those inputs changes underneath the estimate.

Stated fully, the shape is: this volume, of this file count, over this path, on this hardware, on this date, took this long, and here is the transcript. Anything shorter is a rumour with a unit attached.

The date matters more than it looks. Estimates decay silently as data grows, as a backend is migrated to a different tier, and as the restoring host is replaced with a different instance type. A restore time measured against 400 GiB is not a restore time for the 900 GiB that volume now holds, and nothing in the runbook will announce that. Re-measure on a fixed cadence regardless of whether anything looks like it changed, and re-measure immediately whenever volume, path, hardware or backend changes materially.

Production discipline

Compute the floor before anything else, and label it a floor. Volume divided by measured effective throughput is the only part of the estimate that is not a judgement, and if it already exceeds the RTO, the architecture is the thing that has to change.
Measure throughput along the restore path, not the backup path. They use different directions, different tiers and different targets; the second backup in the capture above stored 1.370 KiB while the snapshot it created still requires 60.000 MiB to be read back.
Rehearse within one order of magnitude of the real volume. Small rehearsals measure cache behaviour rather than recovery, and every multiplier that matters — metadata rate, target write ceiling, concurrency limits — only appears at scale.
Put validation inside the stated number. A restore that has not been verified is not a recovery; the checksum comparison, the consistency check and the smoke test are part of the duration the business is being promised.
Stamp every restore time with volume, file count, path, hardware and date, and give it an expiry. An unstamped figure cannot be checked by anyone else and cannot be noticed going stale by anyone at all.

Cross-course references

Linux for Production Sysadmins — Part XLI (Storage Performance) supplies the method for producing the throughput figure this lesson’s floor calculation depends on, measured on your own devices along the path the restore will use rather than taken from a datasheet.
Ceph & Distributed Storage for Production Sysadmins — Part LXXII (Benchmarking) and its treatment of how a benchmark’s conditions determine its result apply directly here, because a restore rehearsal is a benchmark whose volume, path and hardware must be stated for the number to mean anything.
Observability for Production Sysadmins — Part LXXIV (Capacity Planning) covers the growth-tracking that tells you when a restore estimate has expired, which is the failure mode this lesson’s stamping discipline exists to catch.