A defensible reference architecture

Retention, legal hold and sovereignty were the last constraints the estate had to accept from outside itself, and with them every part of the machine has been examined separately. What follows is the assembly. Not a product list and not a diagram, but a set of properties a recovery estate either has or does not have, each one stated with the failure it removes and with the evidence that shows it is present rather than intended. This is the form the material takes in a review, where an assertion about the architecture is worth nothing and a measurement somebody took is worth everything.

flowchart LR
    Prod["Production data and control planes"] --> Local["Fast local recovery points"]
    Local --> Repo["Verified backup repository"]
    Repo --> Immutable["Immutable copy in a separate security domain"]
    Immutable --> Offline["Offline or long-horizon retention"]
    Keys["Independent key and credential escrow"] --> Repo
    Keys --> Target["Clean recovery target"]
    Control["Catalogue, policy and infrastructure-state escrow"] --> Target
    Repo --> Target
    Immutable --> Target
    Target --> Validate["Dependency, invariant and business-transaction validation"]
    Monitor["Independent recovery-point age and restore-test telemetry"] --> Validate
    Validate --> Evidence["Dated RPO, RTO and first-attempt restore evidence"]

Each edge is an operational dependency that needs its own evidence. Extra copy count does not replace the separate security domain, escrow does not replace a clean target, and a completed restore does not become recovery capability until independent validation records the result.

An inventory keyed on data, and on the failure it must survive

Everything downstream rests on the first artefact, and the first artefact is usually keyed wrongly. An inventory generated from the backup platform lists what the platform protects. The systems it does not protect are not rows with a problem in them — they are absent, and absence is the one defect no report renders. So the inventory has to be keyed on the data: every data set the business depends on, enumerated from a source that is not the backup tool. The hypervisor inventory, the cloud account, the asset register and the DNS zone are all adequate sources; the backup catalogue is not, because it is the thing being audited. A data set nothing protects then appears as a row with an empty cell, which a reviewer can see and argue about.

The second key is the failure model, and it is the one that turns an inventory into an architecture. A row reading “backed up nightly to the NAS” has answered a question nobody asked. The row has to say which failure each copy survives, because copies are not interchangeable. An array snapshot survives a deleted file and does not survive the array. A mirror survives the array and does not survive an operator deleting a table, because it copies the deletion faithfully and on schedule. A repository in a separate account survives the account, and survives a credential compromise only if the credential that reaches it is not the one that was compromised.

The failure this property removes is the silent gap: a system nobody protected, or a system protected only against the failure that did not happen. The evidence is the reconciliation itself — a dated comparison between the inventory and the independent source, showing how many rows the source had, how many the catalogue had, and what the difference was. A reconciliation that has never produced a difference has probably never run.

Four axes of independence, and the axis usually missing

Copy counting is where most estates stop, and three copies with one shared fate is one copy with extra invoices. Independence is not a single quality; it has four axes, and a copy is only independent on the axes somebody checked.

The storage system is the first: the media and the software that writes it. Two copies written by one product into one system share its bugs, its firmware and its silent-corruption behaviour. The failure domain is the second — the building, the power feed, the network. This is the axis every plan already has, and it is the one an audience assumes you mean when you say the copies are separate.

The security domain is the third, and it is the axis that is almost always missing. It asks a single question: which identities can reach this copy, and are they the same identities that can reach production? Two copies in two continents administered by one credential set are one copy with respect to a compromised administrator, which is the failure mode the last decade has been about. The fourth axis is time: a copy that holds an earlier state and is not continuously reconciled with production survives an action that was faithfully propagated everywhere else.

The failure this property removes is the single event that reaches every copy — one bug, one building, one credential, one deletion replicated at line rate. The evidence is different for each axis, and only the security-domain axis has evidence you can take on demand: hand the production credential the delete command and record what happened.

A copy that refuses the delete, and a writer that cannot ask

Two of the properties in this architecture are about deletion, and they are routinely confused because both are described with the word “protected”. They answer different threats and neither substitutes for the other.

The first is that at least one copy is under a retention that refuses deletion, with the mechanism and the window named. “Under object lock” is not a property; it is a category. What is defensible is a sentence of the form: this copy is held in COMPLIANCE mode with a three-day default retention, and the version written at 13:28:20 UTC carried X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z. Both halves matter, because the mode decides who the refusal applies to and the date decides when it stops.

The second is that the identity that writes backups cannot delete them. That one is a policy statement, and it is the weaker of the two — but it removes the overwhelmingly commonest incident, which is a script or an operator doing the wrong thing with the right credential.

Watch what the measured capture does to the intuition that these are the same control. A production identity was created holding the built-in readwrite policy — full read, write and delete on the bucket, exactly what an attacker harvests from a compromised production host — and pointed at the protected object.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- deleting the VERSION rather than placing a marker ---
$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Three commands, three different lessons. The first succeeded: a delete with no version identifier placed a marker and exited 0, so the object vanished from an ordinary listing while every byte stayed where it was. The second and third were refused with identical wording, and the third is the one to read twice — the alias that created the bucket, created the users and set the retention policy was told the same thing as the compromised production credential. There was no role above the refusal to escalate to.

Key material escrowed outside the estate, in custody that is not yours

Encryption converts a confidentiality problem into an availability problem, and the architecture has to say who holds the second half. The measured failure is unusually pure: with the passphrase file living inside the directory being protected, losing the site left an intact repository — 11 MiB, six files, every byte readable — and restic snapshots answered Fatal: wrong password or no key found with exit code 12. Nothing was corrupt. Nothing was missing. The data survived the disaster and the ability to read it did not.

The property that removes this is that key material is escrowed outside the estate, held by custodians who are not the production administrators. The second clause is the one that gets dropped. An escrow copy held by the same people, in the same directory service, reachable with the same credentials, is a second copy on the wrong side of every axis in the previous section.

The mechanism is cheap. A repository can carry more than one key, each derived from a different passphrase, each unlocking the same master key.

$ restic key add --new-password-file /work/recovery-pass

  saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90
>>> exit code: 0

$ restic key list
 ID        User  Host          Created
--------------------------------------------------
 66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Two key identifiers against one repository, the asterisk marking the one in use. Neither passphrase can derive the other. When the same disaster was repeated with escrow in place, the restore run under the recovery passphrase returned the file with md5 9eb4e2ad8e08e1dcaaf87ababab964b0, identical to the source recorded before the repository existed — recovered with a passphrase production never held.

The evidence is that restore, not the existence of the escrow. A sealed envelope proves custody; only a decryption proves recoverability. Where the escrow is offline, the same rule applies to the offline form: the Borg paper key export is a printable block of encrypted key material, useless on its own, which is precisely why the block and its passphrase go to different custodians and why somebody has to have typed one back in at least once.

Verification that reads, a restore that lands elsewhere, three ages

Three properties in this architecture are measurements rather than mechanisms, and they are the three a review should ask for first.

Verification reads the data on a stated rotation. The distinction is not academic: on restic 0.19.1, a repository with ten bytes overwritten in the middle of a 17,374,653-byte pack passed the ordinary check with no errors were found and exit code 0, then lost a file on the next restore. Only a check that reads and re-hashes the packs speaks about bytes at rest.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

The rotation matters as much as the command, because the interval between full reads is the width of the window in which corruption goes undetected. On a repository too large to read weekly, restic documents a subset form, and the honest statement becomes a cycle length rather than a status:

REPO=s3:https://obj.example.net/backups-prod
SUBSET=3
restic -r "$REPO" check --read-data-subset="$SUBSET/13"

Thirteen weekly slices is a quarterly cycle. Say so in the plan, because “the repository is verified” and “every pack was read within the last 90 days” are different sentences and only the second one has a number in it.

A restore is proven onto infrastructure that is not the original, with the elapsed time measured per stage. Restoring onto the source host proves the archive opens; it proves nothing about the package that was installed by hand in 2021, the mount that exists only there, or the certificate that never left it. Per stage, because a single end-to-end figure cannot be improved: provisioning, retrieval, transfer, application start and validation each have their own duration and their own owner, and the one that dominates is rarely the one the plan assumed.

Recovery capability is reported as three ages per system: the age of the newest recovery point, the age of the newest completed data verification, and the age of the newest proven restore. They decay independently and no one of them implies another — a proven restore reads only the blobs that snapshot needed, so it says nothing about the rest of the repository. Publishing all three side by side stops the freshest number standing in for the others, which is exactly what a job success percentage does.

The dependency graph, and the two rehearsals

The last two properties are about order and about practice.

The recovery dependency graph has no undocumented cycle. Every estate has cycles; the defect is the undocumented one. The credentials for the backup repository live in the secret manager, whose own recovery needs the repository. The restore orchestration authenticates against the identity provider, restored from a backup the orchestration performs. Name servers resolve the endpoint the restore reads from. None of these are avoidable in general, and all of them are survivable when written down, because a documented cycle has a break point — a sealed unseal key, a static host entry, a credential on paper — and an undocumented one is discovered by a team at hour three with nothing to try.

A rehearsed failover with a rehearsed failback. The failover half is the one that gets exercised, because it is the dramatic half. The failback is where estates lose data, and for a structural reason: during the outage the recovery site accepted writes, and returning to the primary means merging or discarding them. That decision is not a technical step, it needs a person with authority to make it, and the rehearsal is where you find out whether the plan names that person. A failover exercise that ends with the service running in the recovery site has tested half the property and reported the whole one.

The evidence for both is a dated exercise record naming what was executed, what was simulated, what broke, and what the elapsed time was per stage. An exercise that produced no surprises is worth reading twice, because it usually means the difficult stage was the one that got simulated.

Production discipline

1. Key the inventory on data and on failure, and reconcile it against a source that is not the backup catalogue. Publish the difference the reconciliation found, with its date; a comparison that has never produced a discrepancy has probably never run against anything.
2. State independence on four axes, and test the security-domain axis rather than asserting it. Storage system, failure domain, security domain, time. The test is a deletion attempt with a real credential and a recorded exit code: the measured refusal was exit 1 for the production identity and exit 1 again for the full administrator.
3. Never say “under retention” without naming the mode and the date. COMPLIANCE refused the bucket owner and the administrator; GOVERNANCE was removed by an administrator passing --bypass at exit 0; the captured window ran to 2026-08-31T13:28:20.402Z and the refusal ends there. And the property is bought at bucket creation — the retrofit attempt returned exit code 1.
4. Escrow key material to custodians outside the production administration boundary, and prove it by decrypting. An intact repository with no reachable passphrase returned Fatal: wrong password or no key found, exit 12; the escrowed passphrase returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0, identical to the source. Only the second of those is evidence.
5. Report three ages per system and defend each with the artefact it came from. Newest recovery point, newest completed data verification, newest proven restore — never a job success rate, because the repository that reported no errors were found at exit 0 lost a file on the very next restore. Every property in this lesson reduces to the same sentence, and it is the sentence this course exists to make true: recovery capability is something you measure, not something you have.

Cross-course references

• Observability for Production Sysadmins — Part CXIV (Final Production Reference Architecture) assembles the telemetry estate by the same method used here, stating properties and the evidence for each rather than naming products; read it alongside this lesson because the three ages defended above are metrics that must be collected and alerted on by a platform whose own recovery appears as a row in the inventory described in the first section.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) supplies the custody models this lesson’s escrow property depends on, including what it means for key material to be held where the production administrators cannot reach it; this lesson contributes the recovery-side requirement that the custody arrangement be proven by a decryption rather than by an inventory entry.
• Linux for Production Sysadmins — Part L (Disaster Recovery) covers the failover and failback mechanics at host and cluster level, which is the layer below the rehearsal property stated here; the exercise record this lesson asks for is what turns those mechanics into a measured elapsed time per stage instead of a documented procedure nobody has run.

Quiz