Capstone — design, build and defend a production recovery estate · Backup & DR

Every mechanism in this course has been examined on its own, and every one of them looked adequate in isolation. The capstone puts them in one estate and asks the only question that matters: when a specific thing goes wrong, does this collection of parts return a working service with correct data inside a time you committed to? You will find, as most engineers do the first time, that the parts are fine and the joins are not.

Nothing here needs a spare machine, a cloud account or a licence. Everything runs as containers and disposable volumes on one host, using the same fixtures the labs used, so that stage sixteen can destroy half of it without consequence. Budget five hours across two sittings. The build is roughly three of them; the twelve incidents are the rest, and they are where you find out what you actually built.

What you are building, and the diagram you owe

The estate is deliberately small and deliberately awkward. One application server with file data, one PostgreSQL database, one Kubernetes namespace with a persistent volume, one repository, one object store, one second site, and one operator — you — with a dependency graph you have to draw before you are allowed to build anything.

flowchart TD
    APP["App server\nfile data + config"]
    DB["PostgreSQL 17\nWAL archiving on"]
    K8S["Namespace\nmanifests + PVC data"]
    REPO["restic repository\nencrypted, on the NAS volume"]
    KEY["Key escrow\noutside the estate"]
    OBJ["Object store\nversioned + locked bucket"]
    SITE2["Second site\ncatalogue + keys + compute"]
    MON["Recovery capability report\nthree ages per system"]
    APP --> REPO
    DB --> REPO
    K8S --> REPO
    REPO --> OBJ
    KEY --> REPO
    OBJ --> SITE2
    KEY --> SITE2
    REPO --> MON
    OBJ --> MON

Before you build, redraw this as your own, with your own hostnames, your own retention windows and your own arrows — and add the three flows this version deliberately omits: how a failure reaches a human, which identity is permitted to traverse each arrow, and how each component is itself rebuilt if it is the thing that was lost. Your diagram is a graded deliverable and it is the first artefact a reviewer reads. An arrow you cannot name an identity for is a finding against your own design.

Stages 1 to 4 — inventory, objectives and the first proven restore

The first four stages produce the two documents the rest of the build depends on and one restore that actually happened. Estates that skip straight to configuring a backup tool fail here, retroactively, at stage thirteen.

Stage	What you build	Acceptance criterion you can verify yourself
1	A classified inventory of every system, keyed on data rather than on tool	For each system: what protects it, where the copies are, which identities can delete them, and the date and result of the last proven restore. A system with no protection appears as a row with an empty protection cell, not as an absent row
2	A dependency graph taken down to the infrastructure primitives	Every service has an edge to the storage, name resolution, identity and secret material it cannot start without; you can name every cycle and the bootstrap path that breaks it
3	Derived RPO and RTO per system, each with its architecture and assumptions written beside it	Each RPO states the worst case, not the schedule interval; each RTO is a sum over named stages, and you can point to which stage dominates
4	One complete restore of the application server’s file data onto a clean container	The restore exits zero, every file matches a manifest recorded at backup time, and you have a measured elapsed time per stage rather than a total

Stage 4 is the one to do properly. It repeats backup-dr-lab-10-restore-a-service-onto-a-clean-host, and its purpose is not the data — it is the list of things the clean host turned out to need that nobody had written down. Record that list. It is evidence for stage 3, because every item on it is a stage of your RTO that your first estimate omitted.

Stages 5 to 8 — the repository, its keys and its independence

These four stages build the thing that holds your recovery points and then attack it, because a repository you have not attacked is a repository whose failure mode you are guessing at.

Stage	What you build	Acceptance criterion you can verify yourself
5	An encrypted restic repository with per-client identities, none of which can delete	Attempting a destructive operation with a client identity is refused; retention is enforced by a separate identity that runs from the repository side and exists on no production host
6	Key escrow: a second unlock path plus exported key material held outside the estate	You open the oldest retained recovery point using only the escrowed material, on a host with no access to the production environment or its passphrase file
7	Corruption detection: damage a data pack and detect it	`restic check` returns `no errors were found` and exit 0 on the damaged repository; `restic check --read-data` returns exit 1 with `ciphertext verification failed`; the restore then reports `Restored 6 / 7 files/dirs`, `data.bin: FAILED` and `Fatal: There were 1 errors`
8	An independent second copy in the object store, with versioning and object lock	The lock is configured at bucket creation because the platform refuses `Object Lock configuration cannot be enabled on existing buckets` on an existing one; a delete attempt against a locked object is refused for the production identity

Stage 7 is the stage that reframes the course, and it is worth doing exactly rather than approximately. Corrupt a data pack, not a metadata pack: metadata damage is caught by the cheap structural check and produces the comfortable, wrong conclusion that the nightly check is sufficient.

Under the hood

The structural check walks the repository’s index and tree objects and confirms that every reference resolves: each snapshot points at a tree that exists, each tree entry points at blobs the index claims to hold, and each pack file is the size the index recorded. None of that reads the bulk data, which is why it is cheap enough to run nightly and why it passes on a repository whose contents have rotted.

--read-data performs a different operation entirely. It downloads every pack, recomputes the cryptographic hash of each blob, and compares it against the identifier the blob was stored under — which, in a content-addressed repository, is its hash. That is why the failure message names ciphertext verification rather than a checksum mismatch: the blob is encrypted at rest, and the integrity tag on the ciphertext fails before anything is decrypted. It costs approximately what a restore costs, because it performs the same I/O, and that cost is the entire reason estates run the cheap check nightly and the expensive one never.

Stages 9 to 12 — databases, cluster state, infrastructure and the second site

Now the parts of the estate whose recovery is not a file copy. Each of these has a failure mode that produces a running system with wrong data, which is the outcome this course exists to prevent.

Stage	What you build	Acceptance criterion you can verify yourself
9	PostgreSQL with WAL archiving to a destination outside the data directory, plus base backups	A point-in-time recovery to a target immediately before a deliberate mistake reports `recovery stopping before commit of transaction <id>` and `selected new timeline ID: 2`, and the recovered database matches both the recorded row count and the recorded sum
10	The naive comparison, run deliberately	A file-level `cp -a` of the live data directory starts and returns every row; the same directory copied without `pg_wal` fails with `PANIC: could not locate a valid checkpoint record`. You can state why the first result is the dangerous one
11	Kubernetes: manifests in version control, cluster state snapshotted off-node, volume data backed up separately	Deleting and re-applying the namespace returns every object with an empty volume (`cat: can't open /data/orders.csv`); restoring the node-level archive returns byte-identical contents, and your validation refuses to pass on two empty checksums
12	A second site holding the catalogue, the escrowed keys and an independent bootstrap path	You can open a repository, locate a recovery point and start a service using only material present at the second site, with the primary site unreachable

Stage 12 is where most capstone builds are found to be incomplete, and the discovery is always the same shape: the second site has compute and storage and no way to find out what recovery points exist or to open them. The catalogue and the key material are small, unglamorous and irreplaceable, and they are classified as infrastructure rather than as data, so nothing backs them up.

Stages 13 to 16 — assurance, measurement and the exercise

The last four stages convert the estate from a collection of mechanisms into something whose capability is observable.

Stage	What you build	Acceptance criterion you can verify yourself
13	Automated restore verification that restores to an isolated target and compares against a manifest	The verification exits non-zero on the damaged repository from stage 7, and your comparison cannot pass on two empty strings
14	A recovery capability report: three ages per system	Age of the newest recovery point, age of the newest completed data verification, age of the newest successful restore. No job-success percentage appears anywhere on it
15	A measured end-to-end RTO for one tier-one service	A stage table in seconds — detection, decision, provisioning, retrieval, transfer, decryption, restore, dependency startup, data validation, business validation — and a named dominant stage. Your stage 3 estimate is corrected from this, in writing
16	A failover to the second site, operation there, and a failback	Data written at the second site during the outage is present after failback; you can state which site was authoritative at each moment and how the divergence was reconciled

Stage 16 is the half nobody rehearses. Failover is exciting and gets practised; failback happens under exhaustion, after the incident is notionally over, with data that now exists in two places and diverged while nobody was watching. Treat it as a second incident with its own plan, because that is what it is.

The twelve injected incidents

With the build complete, work through these in order. Each is answered with the evidence your own estate produced, and several are unrecoverable if you skipped an earlier stage — which is the point.

A file is deleted and noticed four days later. Recover it, and state the retention property that made that possible.
The nightly structural check has been green for the whole build. Prove or disprove the integrity of one repository.
The primary site is gone. Open a repository using only what exists elsewhere.
A client host is compromised. State exactly what the attacker can do to your recovery points with the credential on it.
Ransomware encrypted the application server two days ago and was noticed today. Choose a recovery point and defend the choice.
Someone enabled a lifecycle rule expiring non-current versions after seven days. Establish what you still have.
The database returns rows after a restore. Prove the data is correct.
A mistaken DELETE ran at a known time. Recover past it and state the RPO you actually paid.
The Kubernetes namespace is rebuilt from manifests and the application starts empty. Say what is missing and recover it.
The backup platform itself is the thing that failed. Recover a single file without it.
The recovery takes longer than your committed RTO. Name the stage and the architectural change that fixes it.
Your director asks for three findings this quarter and the rest next year. Separate containment from architecture and defend the split.

How the build is assessed

A reviewer reads four things, in this order, and can reach a verdict from them alone: your dependency graph, your recovery capability report, your stage-15 timing table, and the transcript of incident 3. If the graph has an unbroken cycle, the report contains a job-success percentage, the timing table has a total instead of stages, or incident 3 ended with an intact repository nobody could open, the build does not pass regardless of how much of it works.

The pass condition is not that nothing failed. Stage 7 is supposed to fail, stage 10’s wrong answer is supposed to succeed, and a capstone in which nothing surprised you is a capstone in which you built the estate you already believed in.

Production discipline

Draw the dependency graph before the architecture diagram. The graph finds the cycles and the unowned items; the architecture diagram only records what you already decided.
Never remove a protection before its replacement has produced a verified restore. Every migration in this estate is additive first and subtractive second, and the subtraction gets its own dated change.
Report three ages per system, never a success rate. Newest recovery point, newest completed data verification, newest proven restore. A percentage aggregates away the one system that matters.
Contain before you clean up. A credential that can delete backups is revoked or de-scoped tonight; the identity redesign it implies takes weeks and does not protect you during them.
Publish no objective without the measurement that produced it. An RPO or an RTO stated without its architecture and its assumptions is a wish, and it will be quoted back to you by someone who was not in the room.

Cross-course references

PostgreSQL for Production Sysadmins, Part XIII (Backup, Archiving and Point-in-Time Recovery) covers the database-side mechanics that stages 9 and 10 exercise here — this capstone treats the same operations purely as recovery evidence, and asks what the resulting database proves rather than how it was produced.
Secrets, PKI & Certificate Management for Infrastructure Engineers, Part XVIII (Incidents and Recovery) is the direct dependency of stage 12: the second site cannot start its secret manager without the certificate authority whose key that manager holds, and this capstone requires you to have broken that cycle and written the bootstrap path down before failover, not during it.
Kubernetes for Production Sysadmins, Part XCIX (Complete Cluster Loss) is where the control-plane mechanics behind stage 11 are taught in depth; here they are reduced to the single distinction stage 11 proves by experiment, that desired state and application data are separate protection problems with separate failure modes.