The restore is running correctly and will finish long after the deadline

Reported symptoms

A storage controller fails at 02:04 on 2026-08-27 and takes the array under the rbdr-ledger file service with it. The documented recovery is a full restore of /srv/rbdr-ledger. It starts at 02:14.

At 08:19 it is still running, having written 1.01 TiB of a 4.10 TiB recovery point. The DR plan publishes 90 minutes for this service, and that is the number the incident commander has repeated to the business every hour since 02:30.

Nothing in the restore looks wrong. No errors, no retries, no warnings, and a transfer rate that has not left the band 47 to 49 MiB/s since the first minute. Every escalation asks what has broken; every check answers that nothing has. Projected over the bytes that remain, the restore finishes at about 03:07 the following morning.

Evidence provided

The stage timings say where the hours are going.

$ rbdr-restore-run --status

stage                     elapsed     state
repository open + index   00:00:41    complete
snapshot selection        00:00:03    complete
data                      06:05:12    running

bytes written           1.01 TiB of 4.10 TiB
files written           2,761,004 of 11,204,918
rate (5 min mean)       48.1 MiB/s
errors / retries        0 / 0

Size against rate is the whole projection, and it needs no tooling:

RBDR_SIZE_MIB=4299161
RBDR_RATE_MIBS=48
secs=$(( RBDR_SIZE_MIB / RBDR_RATE_MIBS ))
printf 'full restore at %d MiB/s: %dh %02dm\n' \
  "$RBDR_RATE_MIBS" $(( secs / 3600 )) $(( secs % 3600 / 60 ))

Then the origin of the published figure, which is recorded in the repository that holds the plan.

$ git log -1 --format=%B -- dr-plan/rbdr-ledger.yml

dr-plan: add recovery objectives for rbdr-ledger

RPO 24h, from the nightly schedule.
RTO 90m, rounded up from the nightly job mean of 84m.
No restore timing available; revisit after the next DR exercise.

The relationship that commit assumed had already been measured, on a dataset small enough to hold the whole comparison in one capture.

$ the harness summarising backup and restore seconds for two datasets of the same size

==================================================================
SUMMARY OF PART A
==================================================================
                                 backup    restore
400 MiB in 1 file                 1.49s      1.07s
400 MiB in 8000 files             1.29s      1.36s

READ THESE NUMBERS CAREFULLY, INCLUDING WHAT THEY DO NOT SHOW.

The file-count effect here is modest: restoring 8000 small files took
about a fifth longer than restoring the same bytes as one file. This
test deliberately ran on tmpfs, which removes seek time, queue depth and
network entirely, so it isolates the per-file overhead inside the tool
and nothing else. It does NOT measure what the same shapes would cost on
real storage, and no number here should be carried to other hardware.

--- the asymmetry that IS large, and is the point of the exercise ---
a second backup of 'big' with nothing changed:
backup  : .73s

first backup of 400 MiB : 1.49s
second backup, unchanged: .73s
restore of 400 MiB      : 1.07s

The nightly job reports the middle number. The incident needs the last
one. Backup duration is not a predictor of restore duration, because
backup is incremental and restore never is - and the gap grows with
every day the dataset changes slowly.

Take that capture at exactly its own valuation. It establishes a direction, not a multiplier: on 400 MiB the unchanged backup and the restore differ by less than a factor of two, and it calls its own file-count effect modest. Its text forbids carrying the seconds anywhere else. The sixteen in this incident comes from the site’s own dataset size and its own measured rate.

The assurance record completes the picture: fourteen consecutive monthly restore tests for rbdr-ledger, all PASS, each restoring a sampled subset of files and checking their checksums, none recording an elapsed time.

Work the evidence before reading on

1. Two stages completed in forty-four seconds and one has run for six hours. Which quantity does each kind of stage scale with?
2. The nightly job moves 96 GiB and the restore moves 4.10 TiB. Before reading further, work out the ratio and compare it with the ratio of the two runtimes.
3. The capture shows a .73s backup against a 1.07s restore. What does that entitle you to claim about this incident, and what does it not?
4. Fourteen monthly tests reported PASS. Write down the question each one answered, then write down the question the plan needed answered.
5. The obvious alternative is that the restore is degraded — a throttled link, a retry storm, a failing target disk. Which piece of evidence eliminates it?

Root cause

The alternative had to be eliminated first

If the restore were degraded, this would be an incident with a fix in it, and the right move would be to find the throttle. Three observations close that door. The rate has stayed inside 47 to 49 MiB/s since the first minute, which is not what contention or retries look like. The driver reports zero errors and zero retries. And 48 MiB/s is the write ceiling measured on this target before the restore started, so it is already consuming everything the target can accept.

The restore is not slow. It is running at full speed and the deadline was wrong. That is the distinction between an incident and a design defect, and it changes who has to act.

The published objective was derived from the backup window

The plan says 90 minutes because the nightly job averages 84. That job is incremental: it moves 96 GiB of changed data and spends most of its runtime walking metadata to decide what changed. The restore has no scan phase and no increment. It writes 4.10 TiB — forty-three times the bytes — through the target’s write path rather than the source’s read path. Two independent errors sit in one number: the byte counts differ by a factor of forty-three, and the runtimes are dominated by different work.

No restore had ever been timed

The assurance programme is why the wrong number survived fourteen months of review. Its monthly test restores a sample and verifies checksums, which answers whether the recovery point is readable. It records PASS, and PASS reads as coverage.

It never recorded a duration and never restored the whole dataset. So when the plan template demanded a figure for the recovery time cell, no measurement existed to put there. The commit message says as much and promises to revisit. The only number in reach was the backup runtime, and a filled cell stops attracting attention.

Resolution

Do not touch the running restore. It is moving at the rate the hardware allows, no faster path exists for these bytes tonight, and a restart discards six hours.

Correct the number the business is deciding against, and hand over the arithmetic rather than the conclusion so the commander can see what would have to change for the answer to change. Confirm the rate independently rather than trusting the tool that reports it, by measuring what actually lands on the target over five minutes.

Then change what the restore is asked to deliver first. Ninety minutes was written for a service, not for 4.10 TiB, and the service does not need all of it to open. Run the remaining work as an ordered set of paths — the current financial period ahead of the archive — and bring rbdr-ledger up read-only on that subset while the remainder streams in behind it. That turns one 25-hour outage into a short degraded window and a long backfill, without making a single byte move faster.

Verification

Run one full timed restore of rbdr-ledger to isolated infrastructure, recording each stage separately. The figure that goes into the plan is the sum of those stages from that run, with the date, the dataset size and the target hardware beside it.

Then check that the published figure and the measurement agree by arithmetic rather than by assertion: dataset size divided by measured sustained rate, plus the fixed stages, must reproduce the published number. Re-run that division when the dataset crosses a growth threshold, because an objective measured at 4.10 TiB is a statement about 4.10 TiB.

Finally, confirm the assurance record now carries an elapsed time on every entry, and that a test with no recorded duration reports as incomplete rather than PASS.

Prevention

Time the restore, because that is the number the incident needs. Backup and restore move different amounts of data through different bottlenecks. An objective never produced by a stopwatch is a target, and the plan should say which of the two it is holding.

Make the objective a function, not a constant. Expressed as dataset size over measured sustained rate plus the fixed stages, it recomputes as it ages, and the growth that invalidates it becomes visible before the outage does. Require a pointer to the timed restore it cites; an empty pointer is a finding, and it is the state that let a backup runtime be borrowed.

Design recovery order, not just recovery. When the full dataset cannot meet the objective, decide in advance which subset must, and record that ordering in the plan so it is not invented at 08:19 during the outage.