Cost engineering for a recovery estate

An archive tier is a control whose entire justification is money, and it is not the last one this course has introduced that will be argued in those terms. A second copy, a longer retention window, an immutable bucket, a quarterly restore exercise: each arrives at a budget review as a line item rather than as a failure model, and the person defending it is usually the only one in the room who knows which failure the line holds back. An engineer who can describe a control but cannot price it loses that argument to somebody who was never told what was being removed. So the arithmetic is the subject here, including the parts that weaken your own case.

Five places the money goes, and only one is the storage line

The first term is capacity multiplied by retention depth, and the operative word is multiplied. A 2 TiB dataset captured as a daily full and held for 30 days is 60 TiB stored before any reduction; held for 90 days it is 180 TiB. Incremental chains change the shape of that curve but not its direction, since the stored total is the first full plus every change interval inside the window. Then the copy count multiplies again — and the second and third copies are what a cost review proposes to remove, because they are the ones nobody reads.

The second term is the storage class and what it charges to give the data back: a price per stored byte, a minimum storage duration billed whether or not the object survives it, and a retrieval path with its own latency and charge. The previous lesson established all three, and the third is charged on the worst day of the year. Transitions and retrievals are billed per thousand objects too, which makes a repository of many small pack objects expensive to move and to read back regardless of how many bytes it holds.

The third term is egress. A second copy at a different provider pays egress on the changed data every night and on every byte a recovery reads — including every restore test, which pays exactly what an incident would, twelve times a year, on purpose.

The fourth term is the compute that verification and restore testing consume. Part I measured what the cheap integrity check does and does not prove; the cost consequence is that the check which reads the data does the same I/O a full restore does, every run. A restore test costs more, because it needs a target: storage for the restored data and an instance to run the recovered service, both for the duration of the test rather than of the command.

The fifth term is the engineering time an exercise consumes, and it is routinely the largest. Eight people for six hours is 48 engineer-hours before preparation and write-up, and none of it appears on an invoice. That invisibility is why it is proposed first for removal, and why removing it is the most expensive saving available: the exercise is the only thing producing evidence that the other four terms buy anything.

Put the terms in one expression, using the price your account actually pays rather than a list price:

RAW_GIB=4200
COPIES=3
PRICE_GIB_MONTH=0.021
awk -v raw="$RAW_GIB" -v n="$COPIES" -v p="$PRICE_GIB_MONTH" \
  'BEGIN { printf "%.2f per month for %d copies of %d GiB stored\n", raw * n * p, n, raw }'

A reduction ratio is a measurement of one dataset, never a plan

Deduplication and compression are the mechanisms that break the multiplication above, and they are where cost models acquire their most confident wrong numbers. Both are real. Neither has a value until it has been measured on the data it will be applied to.

This course’s own captures make the point. Measured on restic 0.19.1, the first backup of a 60.000 MiB tree reported Added to the repository: 60.005 MiB (60.008 MiB stored) — stored exceeds added, so the reduction on that content was not small but negative, because the data did not compress and the format carries overhead. The second backup of the same tree, one file changed, reported Added to the repository: 2.062 KiB (1.370 KiB stored). Same tool, same repository, same dataset; two reduction figures four orders of magnitude apart, decided entirely by how much changed between runs.

Borg separates the two mechanisms in its output. Measured on borg 1.4.0, the first archive reported 41.94 MB original, 41.94 MB compressed and 41.94 MB deduplicated; the second reported 41.94 MB original, 41.94 MB compressed and 613 B deduplicated. Compression achieved nothing on either run because the content was already incompressible; deduplication achieved almost everything on the second because the content was already stored. A single quoted “ratio” hides which of the two is doing the work.

Which one it is decides whether the ratio transfers. Between-run deduplication scales with how little changes, so it is large for machine images and system trees and small for a database whose pages churn. Compression and within-run deduplication scale with redundancy inside the data, so both collapse towards 1:1 on anything already compressed or encrypted. A vendor’s 20:1 is a true statement about somebody else’s data.

So measure, then state a floor. Separate what a restore would write from what the repository occupies:

export RESTIC_REPOSITORY="s3:https://s3.example.net/backups-prod"
restic stats --mode restore-size latest
restic stats --mode raw-data latest

That ratio holds only for the data that produced it, and only while that data keeps its shape: it expires the quarter the database is encrypted at rest or a workload with different entropy arrives. A plan that survives review therefore carries the measured ratio, the date of the measurement, and a 1:1 floor for the portion of the estate known to be already compressed or encrypted. A plan carrying one vendor ratio and no date has no failure mode — it cannot be wrong until the capacity runs out.

Capacity that the retention window will not give back

That last point was captured directly. A bucket was created with locking enabled and a default COMPLIANCE retention, the nightly object was written, and the metadata recorded on that version read X-Amz-Object-Lock-Mode : COMPLIANCE with X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z. A production identity holding full read, write and delete on the bucket — the credentials an attacker takes from a compromised production host — then deleted it.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

The delete exited 0 and the bucket now holds two objects where it held one. That is versioning, not the lock; the lock appears when somebody reaches for the bytes themselves.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Read that as a security result and it says the backup survives a compromised administrator. Read it as a cost result and it says what the security reading hides: inside the retention window, nobody can lower the bill. Not the operator who fat-fingered a 40 TiB upload, not the team that found the wrong prefix was being captured, not the account’s full administrator. The bytes are billed until the retain-until date passes, and the window cannot be shortened retroactively for versions that already carry a record.

The attacker case follows from the same mechanism and is the one no capacity model contains. The compromised identity could not delete, but nothing in the lock stopped it writing, and every object it writes inherits the default retention — billed for the full window and removable by nobody. So the sizing question for an immutable copy is not “how much do we back up” but “how much can be written here in one retention window by anything holding a write credential”. That has an engineering answer — a dedicated bucket, a write path no production host holds credentials for, quotas or budget alarms in front of it — and no retention answer at all.

The same capture prices the mode choice. Under GOVERNANCE an administrator holding the bypass right removed the protected version at exit code 0, where under COMPLIANCE the same privilege level was refused at exit code 1. Compliance mode buys a guarantee that holds against your own administrators and sells the ability to correct an expensive mistake; choose per bucket, and write down which you bought.

The seven-day retention that was approved as a cost saving

The second trap this course has measured is dangerous precisely because it arrives wearing a cost label.

A versioned backup bucket grows. A rule expiring current versions at thirty days is added, and consumption keeps rising, because on a versioned bucket that expiration writes a delete marker and leaves the bytes. Somebody correctly diagnoses the arithmetic and adds the clause that actually returns storage: NoncurrentVersionExpiration with NoncurrentDays set to seven. Consumption falls. The ticket closes as a capacity fix.

What that change did was set the recoverable depth of every object in scope to seven days. The bucket still reports versioning as enabled, the retention policy document still says four hundred days, and the two statements live in places nobody reads together. Worse, non-current age counts from the moment a version stopped being current rather than from when it was written, so versions superseded long ago were already past the threshold the day the rule was applied — the loss is immediate, not phased. The current-object inventory does not change, so every instrument built to count recovery points keeps reporting a full set.

The rule is short: a change that reduces storage consumption has changed retention until somebody proves otherwise. It gets the review a retention change gets, with a named owner, and the reviewer’s question is not whether the rule is well written but which recovery points it removes and which obligation that breaches. The backstop is the signal that does not depend on anybody reading the rule correctly — the age of the oldest recovery point per protected system, alerted against what that system’s plan claims it can reach.

Present an exposure, and let the business own the answer

Technically correct engineers lose these arguments because they arrive with a preference. “I would rather keep the copy under compliance-mode retention” is a preference, and a preference loses to a number every time. What works is to stop defending the control and start describing what the estate can and cannot survive at each price, then hand the choice over.

That takes three statements and no adjectives. Name the change and its saving in the units finance uses. Name what it does to a stated recovery capability, in the units the plan uses — the oldest recoverable point for this system becomes fourteen days rather than four hundred; the estate stops being able to recover from a compromised administrator, which is the failure the locked copy exists for. Then name the failure mode the removal exposes, and say honestly what you do and do not know about its likelihood, because an invented probability turns your one credible statement into a disputed one.

Let the decision be made by somebody who owns the outcome, and record it as a decision rather than as a silence. A recorded exposure carries a named owner, a date, the amount saved, the capability given up, and a review date — the last of which is what stops an exposure accepted for one quarter’s budget from becoming permanent because nobody reopened it.

The final step separates a governance ritual from an engineering control: instrument the accepted exposure. If the decision reduced the recovery window to fourteen days, the alert threshold on oldest recovery point age moves to fourteen days the same day, and the plan’s stated capability moves with it. Otherwise the estate keeps alerting against a capability it no longer has, and the accepted exposure quietly becomes an undetected one.

Production discipline

1. Model retention as a multiplier, never as a policy setting. Stored total is dataset multiplied by retention depth multiplied by copy count, so a proposal to extend a window from 30 to 90 days is a proposal to triple a capacity line before any reduction is applied.
2. Carry a measured reduction ratio with its date, and a 1:1 floor for data that will not reduce. Measured on restic 0.19.1, one backup stored 60.008 MiB for 60.005 MiB added while the next stored 1.370 KiB for 2.062 KiB added; measured on borg 1.4.0, a second 41.94 MB archive deduplicated to 613 B with compressed size unchanged. The tool did not decide those numbers, the data did.
3. Size an immutable bucket for what a write credential can put in it during one window. Measured on MinIO RELEASE.2025-09-07T16-13-09Z, a COMPLIANCE-retained version refused the production identity, that identity passing --bypass, and the full administrator, all at exit code 1 — so inside the window nobody can lower the bill, including you.
4. Treat any change that reduces storage consumption as a retention change. Adding NoncurrentVersionExpiration with NoncurrentDays: 7 returns bytes and sets recoverable depth to seven days, while the bucket still reports versioning enabled and the policy still claims its original window.
5. Convert every cost proposal into a named exposure with an owner, a review date and a moved alert threshold. A decision recorded only as a saving leaves the estate alerting against a recovery capability it no longer has, which is how an accepted exposure becomes an undetected one.

Cross-course references

• Terraform for Production Sysadmins — Part XXIII (Policy as Code) is where the rules this lesson argues for stop depending on a reviewer’s attention: a policy that rejects a lifecycle expiration on a bucket tagged as holding recovery points, or that requires object lock on it, catches the seven-day change described here at plan time rather than at the next incident.
• Observability for Production Sysadmins — Part LXXVI (Cost Management) applies the same decomposition to a telemetry estate, where retention depth and query access pattern set the bill exactly as they do here; read it for the method of pricing a retention choice against the access it will actually see, which is what turns the arithmetic in this lesson into a defensible number.
• Ceph & Distributed Storage for Production Sysadmins — Part LXVI (Capacity Forecasting) supplies the forecasting discipline this lesson depends on for its hardest input, namely the capacity an immutable copy will reach before its oldest versions age out, which cannot be derived from the current dataset size.

Quiz