Capacity planning for a recovery estate

Deduplication and compression, measured on real data in the previous lesson, are two terms in a capacity model and neither of them is the model. A repository that stores half the bytes of the tree it protects is still growing, still holding copies it may not delete, and still has nowhere to put a restore. Capacity here is not a number to look up; it is a build-up you can show another engineer term by term, each term carrying a value and a date.

The build-up starts with what the job actually selects

Protected data is the first term and the one most often taken from the wrong place. It is not the provisioned capacity of the volumes, not the total df reports, and not the size of the array. It is the logical bytes the include and exclude rules actually select on the night the job runs. Thin-provisioned volumes report a size nobody is storing, sparse files a length nobody is reading, and an excluded build cache never enters the repository at all. Take this number from what the tool processed.

Measuring it is a small discipline of its own. A database contributes the size of the artefact the backup actually takes, which is not the size of its data directory: a logical dump and a physical copy of the same instance differ by whatever indexes, free space and dead rows the files are carrying. A virtual machine contributes its allocated disk if the image is copied whole, and only its used blocks if the tool reads the guest filesystem instead. Record which convention each number follows, because an estate that mixes the two cannot be added up.

Copy count multiplies everything after it, and it is three estimates rather than one estimate tripled: each copy can hold a different retention depth on a different medium with a different reduction ratio. Retention depth is the third term, in restore points rather than days, because a policy holding thirty dailies, twelve monthlies and seven yearlies keeps forty-nine. Multiplying those three gives the full-copy model — correct for a tier that writes an independent image every time, wrong by an order of magnitude for anything incremental-forever.

# One incremental-forever copy, in GiB. The four inputs are placeholders:
# replace each with a figure measured on your own data, stamped and dated.
PROTECTED_GIB=2000          # bytes the include rules select
CHANGE_RATE_PCT=3           # measured new-chunk fraction per run
RESTORE_POINTS=90           # retention depth, this copy
REDUCTION_PCT=45            # measured dedup and compression

logical=$(( PROTECTED_GIB + PROTECTED_GIB * CHANGE_RATE_PCT * RESTORE_POINTS / 100 ))
stored=$(( logical * (100 - REDUCTION_PCT) / 100 ))
printf 'logical %s GiB -> stored %s GiB for one copy\n' "$logical" "$stored"

At those inputs the deltas contribute 5400 GiB against an initial 2000 GiB. Multiply by the copy count, then add the three terms the arithmetic omits: immutability window, restore workspace and growth.

Growth is the term that decides when the plan expires, and it has two independent parts — the protected volume grows as the business does, and the change rate can grow on its own as workloads shift. Both belong in the model as a rate against a horizon, and the horizon comes from procurement rather than from a calendar. If new capacity takes eight weeks to arrive and another two to enter service, the plan needs a threshold that fires ten weeks before the pool would otherwise fill, monitored like any other production signal. An estate sized for the day the spreadsheet was written is sized for a date that has already passed.

Change rate is the dominant term, and it can be measured tonight

The first run costs the protected volume once; every run afterwards costs only the chunks the repository has not already seen. Over ninety restore points the deltas total ninety times the change rate times that volume: 0.9 times it at one per cent, 4.5 times at five. Guessing the rate wrong by a factor of five moves the estimate by roughly a factor of three, which is why it deserves more measurement effort than the rest combined.

It is also the term people define incorrectly. Change rate is not the fraction of files modified, nor the fraction of bytes that differ from yesterday. It is the fraction of chunks the repository had never stored before. Content-defined chunking makes an append and an in-place overwrite cheap, because the surrounding boundaries do not move; it makes a re-encode or a full table rewrite expensive, because every boundary after the first change shifts and the file arrives as new chunks. Two workloads with identical byte-change percentages can cost the repository entirely different amounts.

The measurement is free: the tool reports it on every run.

$ restic backup /work/prod --tag daily

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved

>>> exit code: 0

Added to the repository: 2.062 KiB (1.370 KiB stored) is two measurements and the build-up needs both: the first is the new chunk volume, the numerator of the change rate; the second is what those chunks occupied after compression, which is what consumes capacity. Divide the first by the volume the run processed — here 60.000 MiB — and that is one night’s change rate.

One night is not a change rate. Collect the series for a full business cycle, because the shape matters more than the average: a month-end reconciliation or an upgrade that rewrites every row produces one night worth ten ordinary ones, and the repository has to hold it.

Where a tool does not report the delta directly, the series can be reconstructed from the repository’s own growth between runs, provided nothing was pruned in between: the difference in occupied bytes across two consecutive nights is the stored delta, which is what capacity paid, not the logical figure. Either way, keep the two figures apart. Logical added is the change rate; stored added is what that change cost; and the ratio between them is a reduction figure measured on your data, which is the only kind this lesson accepts.

# An example series: each run's added bytes over the protected volume.
PROTECTED_GIB=2000
ADDED_GIB='18 22 19 240 21 20 23 19 25 21 20 22 19 21'
printf '%s\n' "$ADDED_GIB" | tr ' ' '\n' \
  | awk -v p="$PROTECTED_GIB" '{ r = $1 * 100 / p; if (r > w) w = r } END { printf "worst night %.2f%% of %s GiB\n", w, p }'

Measured: the window in which capacity cannot be reclaimed

Every other resource shortage has the same escape hatch: remove something. Immutable storage removes it deliberately, and a model that assumes otherwise will be wrong at the worst moment.

The lag starts before object storage is involved. The captured restic run has forget reporting Would have removed the following snapshots with exit code 0, alongside the rule that governs the bytes: forget removes snapshots, space is reclaimed only by prune, and a chunk goes only when no remaining snapshot references it. Dropping a restore point from a policy is a metadata edit.

On versioned object storage, even the delete is not a delete.

$ mc rm prod/rbdr-immutable/backup-0900.tar

Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).

>>> exit code: 0

The command succeeded, the object left an ordinary listing, and consumption did not fall: the capture shows both a 0B delete marker and the original 38B version still present afterwards. A cleanup that measures success by whether the object still lists is measuring the wrong thing. Removing the version is what returns the bytes, and that is what the lock refuses.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten

>>> exit code: 1

The capture repeated that attempt with --bypass as the production identity, then as the full administrator; both were refused with exit code 1. The object carried X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z under X-Amz-Object-Lock-Mode: COMPLIANCE, and until that timestamp passes no identity can free those bytes. In the same capture’s GOVERNANCE bucket, mc rm --bypass removed the protected version with exit code 0: COMPLIANCE buys resilience against a compromised administrator and gives up the emergency capacity lever, which is a capacity decision as much as a security one.

Sizing the term is arithmetic once the mechanism is clear. Multiply the worst night’s stored ingest by the lock period and treat the product as capacity already committed, whatever the retention policy says: a tier locking for thirty days, with a worst night storing 400 GiB, has twelve thousand GiB spoken for before a single restore point is counted.

The restore workspace nobody put in the model

Recovering a 2 TB database needs somewhere to put 2 TB before it is a database again. That space is not in the repository estimate, it is frequently not in any estimate, and it gets discovered during the incident that needed it.

The workspace is sized at the logical volume, not the stored volume: the reduction ratios describe what the repository holds, while the target receives the data expanded, byte for byte. It is also not in place. Overwriting the damaged original before the restored copy is verified destroys the one artefact that might still be partly readable, so the workspace sits alongside it. Staged paths add more — download then extract is two copies, restore then replay logs needs an archive area, load then rebuild indexes needs temporary space that can rival the table.

Then multiply by concurrency, because a real disaster does not restore one system. Sizing the workspace for a single recovery is sizing for the least likely scenario in the failure model, and two simultaneous restores that each need their own staging area will discover that at the worst time.

Where the space lives matters as much as how much of it there is. Carving the workspace from the pool that holds the repository means a large restore can drive that pool to full, stopping backups during the incident — and behind an object lock there is no deleting your way back out. “We will spin up a volume” is a plan only if somebody has confirmed that the quota permits it, that the hypervisor has the free space, or that the array has a populated shelf; capacity sitting on a purchase order is not capacity. Treat the workspace as a named line in the recovery plan: a path, a size, an owner, and a check that it can be provisioned in the time the plan assumes.

A ratio without a dataset and a date is not a number

Every ratio in the build-up is a property of a dataset at a point in time, not of the tool, and the captured repository demonstrates it inside a single 60 MiB tree. Its first run reported Added to the repository: 60.005 MiB (60.008 MiB stored) — stored is larger than added, so compression returned nothing on that content. Its second run, same repository and same binary, reported 2.062 KiB (1.370 KiB stored), a saving of about a third.

That is the argument against an estate-wide ratio. Incompressible content — media, compressed archives, encrypted blobs — returns nothing; text, logs and near-duplicate machine images return a great deal, and the mixture shifts the moment one team enables application-level encryption. A vendor ratio describes somebody else’s dataset, chosen because it demonstrated well, so treat it as marketing until it has been reproduced locally.

The stamp that makes a ratio usable names five things: the value, the dataset, the tool and version, the window, and the date. 2026-08-28, restic 0.19.1, /work/prod, 3 files, 60.000 MiB: first run stored 60.008 MiB for 60.005 MiB added can be checked and can be noticed going stale. “About 2:1” cannot.

Give every ratio an expiry alongside its date. Re-measure on a fixed cadence, and re-measure immediately when a dataset changes character: an application that starts writing compressed objects, a migration from a text export to a binary one, or encryption enabled at the application layer will each move a reduction figure without moving anything a dashboard is watching. A ratio eighteen months old is describing an estate that no longer exists, and it will not announce that itself.

Production discipline

1. Show the build-up, not the total. Protected data, copies, retention points, change rate, measured reduction, immutability window, restore workspace and growth are eight lines, each with a value and a date; one headline number hides which term is the guess.
2. Measure change rate over a full business cycle and plan on the worst night. The captured run reported 2.062 KiB (1.370 KiB stored) against a 60.000 MiB tree — one night’s reading, and the month-end rewrite belongs in the same series.
3. Count the immutability window as capacity already spent. A COMPLIANCE-locked version carrying X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z was refused with exit code 1 to the production identity and to the full administrator.
4. Give the restore workspace a path, a size and an owner. Size it at the full logical volume of the largest protected system, multiply for staged paths and concurrent recoveries, and keep it off the pool the repository depends on.
5. Stamp every ratio with dataset, tool version and date, and give it an expiry. The captured repository stored 60.008 MiB for 60.005 MiB added on one run and 1.370 KiB for 2.062 KiB on the next.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part LXV (Why Full Clusters Are Dangerous) sets out what happens when a pool reaches capacity rather than merely approaching it, and each consequence applies here with the usual remedy removed, because a repository behind an object lock cannot be emptied.
• Observability for Production Sysadmins — Part LXXIV (Capacity Planning) supplies the trending method that turns this build-up into a forecast, which is what the change-rate term needs: a tracked series with a worst case, not the single reading one night produces.
• Kubernetes for Production Sysadmins — Part L (PersistentVolumes and Claims) explains the provisioning path a restored workload depends on, which is where the restore-workspace term becomes concrete: the claim must be satisfiable at full logical size, during the incident, before the workload starts.

Quiz