Backup network design and restore throughput · Backup & DR

Rebuilding a catalogue, where the previous lesson ended, answers which recovery points exist and where their bytes are kept — and says nothing at all about how long those bytes take to arrive. Part II of this course reduced that second question to a floor, volume divided by a throughput figure you measured yourself, and then listed the multipliers sitting above it. One term inside that floor deserves a lesson of its own, because it is the one most often supplied by a purchase order instead of a measurement.

The network between a repository and a recovering host is a stage of the RTO exactly like decryption or metadata work, with a duration that can be computed and a contention problem that can be predicted. It is also, in most estates, the only stage nobody has ever timed, because the number that stands in for it — the link speed on the invoice — arrives already looking like an answer.

A link speed is a ceiling, a sustained rate is a measurement

A circuit described as 10 Gb/s advertises a signalling rate. Convert it and you get roughly 1192 MiB/s, and that figure is a ceiling in the same sense that a speed limit is a ceiling: it tells you what the road forbids, not what the traffic does. Between that ceiling and the rate a restore observes sit protocol framing, TCP behaviour on the actual path, TLS on the wire, the repository’s own read path, the receiving host’s CPU, and the target volume’s write path. The rate a restore observes is the smallest of every stage along that path, and which stage that is cannot be deduced from a diagram.

So the number a recovery plan needs is a sustained rate along the restore path, measured in the direction the restore travels, from the storage the restore will read to the target it will write. Sustained matters because bursts lie: a transfer that opens at several hundred mebibytes per second and settles at eighty has a burst figure and a sustained figure, and only the second one survives contact with a dataset large enough to matter. Direction matters because paths are frequently asymmetric — an upload-shaped link, an object store whose read path is not the mirror of its write path, a firewall that inspects one direction more aggressively than the other.

# A sustained rate is measured along the path the restore will take, not read
# off a NIC. Pull a known volume out of the repository into the restore host
# and time it end to end, with nothing cached.
REPO_MOUNT=/mnt/backup-repo
PROBE="$REPO_MOUNT/restore-probe.bin"
PROBE_MIB=8192
start=$(date +%s)
dd if="$PROBE" of=/dev/null bs=1M status=none
end=$(date +%s)
elapsed=$(( end - start ))
if [ "$elapsed" -gt 0 ]; then
  printf 'single stream: %s MiB in %s s = %s MiB/s sustained\n' \
    "$PROBE_MIB" "$elapsed" "$(( PROBE_MIB / elapsed ))"
fi

That measurement produces one of the two numbers a plan needs. The other is the aggregate: what the whole path delivers when several transfers run at once. These are different quantities and they diverge dramatically on long paths. A single connection carries only as much data as it can hold unacknowledged, divided by the time an acknowledgement takes to come back, and on a high-latency circuit that product is often a small fraction of the link. Eight concurrent transfers each get their own share of that behaviour, so the aggregate can be many times the single-stream figure while the link itself has not changed at all.

# The same path measured twice: one stream, then several. The pair is the
# result; either number alone is misleading.
REPO_HOST=repo01.example.net
iperf3 -c "$REPO_HOST" -t 60
iperf3 -c "$REPO_HOST" -t 60 -P 8

Record both, with the date and the endpoints, and label which one applies to which case. A single large image restored by one process is governed by the single-stream figure. A platform restoring forty hosts concurrently is governed by the aggregate — until something shared stops it, which is the subject of the last section of this lesson.

Measured: 1.49s to write it, 1.07s to read it back, .73s the next night

The asymmetry that makes network sizing hard is not primarily about the network at all; it is about what a backup moves compared with what a restore moves. The course captured it deliberately on storage that removes the disk as a variable, so that the shape is visible without a device’s characteristics obscuring it. Two datasets of the same size and very different shapes were used.

  dataset 'big'  : 1 file of 400 MiB
  dataset 'many' : 8000 files of 50 KiB
  big  on disk: 400M
  many on disk: 407M

Destructive400 MiB in one file and in 8000 files — each backed up, then restored— EXECUTION VALIDATED on restic 0.18.0 compiled with go1.24.4 on linux/amd64, kernel 7.0.0-29-generic, 12 cores, tmpfs. Part A of the restore-throughput and incremental-chain capture. That capture recorded a timing report rather than the command lines that produced it; big and many are its own names for the two datasets, and the command shown is the pair of operations it timed for the first of them. A restore overwrites whatever is at the target path.

$ restic backup big && restic restore latest --target restored-big

--- backing up 'big' ---
backup  : 1.49s   (285.7 MiB/s)
--- backing up 'many' ---
backup  : 1.29s   (333.3 MiB/s)

--- restoring 'big' ---
restore : 1.07s   (400.0 MiB/s)
--- restoring 'many' ---
restore : 1.36s   (307.6 MiB/s)

Two things have to be said about those figures before anything is built on them, and the capture says both itself. The first is about the file-count effect, and it is a caution against over-reading: “The file-count effect here is modest: restoring 8000 small files took about a fifth longer than restoring the same bytes as one file.” The capture then explains why the effect is small here — the test “deliberately ran on tmpfs, which removes seek time, queue depth and network entirely, so it isolates the per-file overhead inside the tool and nothing else.” On real storage across a real path, seek time and per-object round trips are exactly what the small-file case adds, and they were removed on purpose. The second caution is blunter: the numbers “were measured inside a container on a tmpfs, on one machine, on one date”, and “no number here should be carried to other hardware.”

What survives transplanting is the asymmetry, which the capture isolates next.

Service impact possiblethe same 400 MiB backed up twice and restored once— EXECUTION VALIDATED on restic 0.18.0, same capture as above, continuing Part A. The second backup was taken with nothing changed in the source dataset; the last three lines are the capture's own recap of the two figures above it against the restore already reported.

$ restic backup big   # second run, source unchanged

--- the asymmetry that IS large, and is the point of the exercise ---
a second backup of 'big' with nothing changed:
backup  : .73s

first backup of 400 MiB : 1.49s
second backup, unchanged: .73s
restore of 400 MiB      : 1.07s

The capture draws the conclusion in two sentences worth memorising: “The nightly job reports the middle number. The incident needs the last one. Backup duration is not a predictor of restore duration, because backup is incremental and restore never is - and the gap grows with every day the dataset changes slowly.”

That last clause is the sizing argument. A dataset that changes slowly produces a shrinking nightly transfer and a growing full-restore transfer, so the two numbers move apart over time on their own, without any decision being taken and without any alert firing.

The path sized for the delta and the restore that ignores it

Put the arithmetic on paper with stated assumptions, because the ratio is usually larger than people expect. Take a 42 TiB estate that changes 3% per day, a backup window of eight hours, and a recovery time objective of six hours for a full restore. Nothing here is a property of a tool; every input is an architectural choice or a business constraint.

# The path that carries the nightly change is not the path that carries a
# full restore. Size both, from the same stated assumptions.
FULL_TIB=42
DAILY_CHANGE_PCT=3
WINDOW_H=8
RTO_H=6
daily_gib=$(( FULL_TIB * 1024 * DAILY_CHANGE_PCT / 100 ))
backup_mib_s=$(( daily_gib * 1024 / (WINDOW_H * 3600) ))
restore_mib_s=$(( FULL_TIB * 1024 * 1024 / (RTO_H * 3600) ))
printf 'nightly %s GiB in %s h needs %s MiB/s sustained\n' \
  "$daily_gib" "$WINDOW_H" "$backup_mib_s"
printf 'full restore %s TiB in %s h needs %s MiB/s sustained\n' \
  "$FULL_TIB" "$RTO_H" "$restore_mib_s"

The nightly delta is 1290 GiB and needs about 45 MiB/s sustained across eight hours. The full restore needs about 2038 MiB/s sustained across six hours. That is a factor of roughly forty-five between the two requirements, from a change rate that most estates would consider high. A path provisioned so that the nightly job finishes comfortably is, on these assumptions, short of the recovery requirement by more than an order of magnitude — and the nightly job will keep reporting success every single night while that remains true.

This is why “the backups run fine” is not evidence about the recovery path, in exactly the way Part II of this course argued that backup duration is not evidence about restore duration. It is the same asymmetry, one layer down.

Three network claims that survive until the first real restore

“The circuit is 10 Gb/s, so we can move 4 TiB in an hour.” The signalling rate is a ceiling, and a single restore stream on a long path routinely delivers a small fraction of it. Until somebody has measured the sustained rate along the restore path, in the restore direction, the plan contains a number from a contract rather than a number from a wire.

“Just rsync it over the WAN when we need it.” This treats the recovery transfer as something to be arranged on the day. A transfer nobody has timed has no place in an RTO, and the day it is first attempted is the day the estate is down, the link is shared with whatever production still works, and the person running it is guessing at concurrency settings under pressure.

“The replica is our backup, so recovery is a failover and the network is not in the path.” A replica removes the transfer for exactly the failure modes it also replicates. On the day the damage is logical rather than physical, the transfer you never sized is the only path back, and it is now being attempted from the copy furthest away.

Why one stream leaves most of a wide path idle, and what parallelism actually buys

A single TCP connection can only have a bounded quantity of data in flight — sent but not yet acknowledged. Its throughput is therefore approximately that in-flight window divided by the round-trip time of the path. The consequence is uncomfortable: on a path with a long round trip, a connection needs a correspondingly large window merely to keep the link busy, and if either endpoint’s buffering, the application’s own read pattern, or the sender’s congestion state holds the window below that value, the link sits idle for most of every round trip while the sender waits.

Loss makes this worse in a way that is specific to long paths. Congestion control reacts to a loss event by reducing the window, and rebuilding it takes a number of round trips proportional to how large the window needs to be. On a short path those round trips are cheap and the window recovers almost immediately. On a long path the same loss rate holds the average window — and therefore the average rate — far below what the circuit could carry. This is why a path can pass every reachability check, show no errors on any interface, and still deliver a fraction of its rate to one transfer.

Parallelism is the standard answer, and it is worth being precise about what it does. Several concurrent connections each maintain their own window and each recover from their own loss events independently, so together they keep more data in flight and absorb loss without a single collapse. What parallelism does not do is create capacity. It converts an application-level limitation into use of capacity that already existed — and only up to the point where some shared resource becomes the binding constraint. That resource is rarely the circuit. It is the repository’s disks, the request rate the object store will sustain, the encryption throughput of one restoring host, or the write path of the target volume.

The practical consequence for a recovery plan is that two measurements are required and neither substitutes for the other: a single-stream sustained rate, which governs one restore, and an aggregate sustained rate, which governs the estate. Beyond the point where the aggregate binds, adding streams stops adding throughput and starts redistributing it — which is the mechanism behind the last section of this lesson.

Contention: the recovery runs while production is still running

A backup window is chosen for quiet. A recovery happens when it happens. That one sentence invalidates most of the throughput evidence an estate collects, because the evidence was gathered at 02:00 on an idle network and will be spent at 14:00 on a network still carrying whatever production survived the incident.

The contention is bidirectional and both directions hurt. A restore that takes the capacity it needs degrades the services that are still working, turning a partial outage into a broader one; a restore that is throttled to protect those services takes longer, and the difference lands directly in the RTO. There is no configuration that avoids the trade. There is only the choice of making it deliberately, in advance, with the arithmetic written down — or discovering it during an incident, where it will be made by whoever is loudest.

A separate recovery path is a design decision with a number attached

The case for a dedicated path between the repository and the recovery targets is not that it is faster in the abstract. It is that it removes the trade above: a restore on a separate path cannot degrade production, and production cannot slow the restore, so the measured sustained rate remains valid during the incident instead of becoming an optimistic figure from a quiet night.

Whether that separation is worth its cost is decided by the same arithmetic as everything else in this part. Compute the restore window on the shared path with a realistic contention assumption, compute it again on a dedicated path, and compare the difference against the RTO. If the shared path meets the objective even when contended, the separation buys reassurance rather than capability. If it does not, the separation is not a luxury — it is the only remaining way to meet a commitment that has already been made to the business.

Where a separate path is not affordable, the fallback is explicit prioritisation rather than hope: a stated share of the link reserved for recovery traffic, a documented ceiling on what restores may consume, and a named person authorised to lift it. All three are cheap to write and impossible to invent at 03:00. The essential property is that the number used in the RTO calculation and the number enforced on the network are the same number.

Many restores at once find the shared bottleneck

The failure mode that ends a large recovery is rarely a slow link. It is the whole estate being restored simultaneously into a shared resource that was sized for one restore at a time.

The pattern is consistent. Forty hosts begin restoring together, each measured individually at a comfortable rate. The repository’s storage — or the object store’s request budget, or the hypervisor’s datastore, or the single 10 Gb/s uplink out of one rack — saturates. Every restore slows to a fraction of its rehearsed rate. Nothing fails, nothing errors, nothing alerts; every job continues making progress and every one of them will complete eventually. But because they are all sharing the aggregate, they all finish at roughly the same late moment, which means that for most of the recovery window nothing is finished. An estate at 60% of every restore is an estate with zero services running.

Serialising is counter-intuitive and usually correct. Restoring the top-tier services first, at full available aggregate, brings them back at the rehearsed single-restore rate and produces working services early. The lower tiers then follow. The total elapsed time is similar; the distribution of value across that time is completely different, and the distribution is what the business experiences. This is the practical application of the recovery dependency graph and the service tiers from earlier in the course: the ordering they define is what turns a shared bottleneck from a disaster into a queue.

Three properties make that queue real rather than aspirational. First, admission control — a limit on how many restores may run at once, enforced by the platform rather than by everyone agreeing to wait. Second, a pre-agreed order, derived from tiers and dependencies before the incident, so the queue is not negotiated live. Third, a measured aggregate, so the concurrency limit is set from evidence about where the shared resource binds rather than from a guess. Without the third, the first two are a policy with an arbitrary number in it.

What to take from this

Measured on restic 0.18.0 on tmpfs, 400 MiB took 1.49s to back up first, .73s to back up again unchanged, and 1.07s to restore. The nightly job reports the middle number and the incident needs the last one.
The same capture recorded 285.7 MiB/s and 333.3 MiB/s for backups and 400.0 MiB/s and 307.6 MiB/s for restores, and states explicitly that “no number here should be carried to other hardware” — they were taken “inside a container on a tmpfs, on one machine, on one date”.
Restoring 8000 files of 50 KiB took 1.36s against 1.07s for the same bytes as one file: the capture calls the file-count effect “modest” here and explains that tmpfs “removes seek time, queue depth and network entirely”, which is precisely where the small-file penalty normally lives.
On stated assumptions — 42 TiB, 3% daily change, an 8-hour window and a 6-hour RTO — the nightly delta of 1290 GiB needs about 45 MiB/s sustained while the full restore needs about 2038 MiB/s, a factor of roughly forty-five between a path that is adequate and a path that is not.
A single-stream rate and an aggregate rate are two different measurements of one path, and each governs a different case: one restore against an estate restoring at once. A plan carrying only one of them is answering only one of the two questions.
Concurrent restores that share a saturated resource all progress and none complete; an estate at 60% of every restore has zero services running, which is why a measured aggregate, an admission limit and a pre-agreed order are what convert a bottleneck into a queue.

Cross-course references

Linux for Production Sysadmins — Part XLII (Network Performance) supplies the measurement technique this lesson depends on: how to obtain a sustained single-stream and aggregate rate on a real path, which is the only legitimate source for the throughput term in a restore window calculation.
Ceph & Distributed Storage for Production Sysadmins — Part XXIX (Network Design) works through the separation of client traffic from replication and recovery traffic onto distinct paths, which is the same architectural decision this lesson costs out when a restore must run without degrading production.
Ceph & Distributed Storage for Production Sysadmins — Part LX (Recovery Tuning) covers throttling recovery so it does not starve client I/O, and the trade it describes is identical to the one here: every unit of bandwidth reclaimed for production is a unit added to the recovery time.