Backup & DR · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 127
- Parts covered
- 19
- Of all lessons
- 100%
Part I
Recovery Objectives, Vocabulary and Failure Models
6 checks
- Backup is not the objective — recovery isA repository whose integrity check reported no errors and exited 0 lost a file on the very next restore, which is why job success is not a measure of recovery capability.→
- Seven words that are not synonymsAn LVM snapshot and its origin were extents in one volume group on one physical volume, and destroying that volume destroyed both — which is why snapshot, replica, mirror and backup are not four names for the same arrangement.→
- The failure model: naming what you protect againstA nightly rsync 3.4.1 mirror is a complete answer to disk failure and no answer at all to an operator deletion, which is why every protection claim has to name the failure class it is making the claim about.→
- Classifying data by what it costs to loseEvery item in an estate answers one of three questions when it is gone — would we rebuild it, re-derive it, or stop — and the items that end recoveries are almost always the ones nobody classified at all.→
- What a backup system owes you, and what it never promisedOn a healthy restic 0.19.1 repository the cheap integrity check and the expensive one printed the same words and the same exit code while guaranteeing different things, which is what a protection boundary looks like before anyone crosses it.→
- Reading an estate: inventory and the first dependency mapMost estates own an asset inventory and a monitoring target list, and neither answers the recovery question; this lesson builds the artefact that does, service-first, and shows why a recovery order falls out of the map rather than the schedule.→
Part II
RPO, RTO and Recovery Sequencing
6 checks
- RPO: engineering an acceptable loss windowA nightly backup at 01:00 with nothing continuous does not give you a recovery point of one day; it gives you a window that grows to 24 hours, and only what reaches the archive narrows it.→
- RTO: everything between the incident and a working serviceThe four-hour RTO in the policy is usually the runtime of a restore command, while the real interval contains a dozen more stages, several of them serial and paced by people rather than by hardware.→
- Estimating restore duration from evidenceA restore estimate starts as volume divided by measured throughput and is then multiplied by CPU, metadata, concurrency, rehydration, target writes and validation — which is why a backup that stored 1.370 KiB says nothing about the 60.000 MiB a restore must read back.→
- Service tiers and what a tier actually buysA tier is real only when it changes backup frequency, retention, storage placement, replication, rehearsal cadence or recovery order — and a Tier 1 service standing on a Tier 3 identity provider recovers at Tier 3.→
- The recovery dependency graphRecovery order is derived from a graph of "cannot start without" edges rather than chosen from a tier list, and the circular dependencies hidden in that graph are why a correct-looking plan stalls in the middle of the night.→
- Recovery SLOs: measuring the promise you madeAn RPO and an RTO that are never measured are aspirations with units attached, and backup job success is the one metric that cannot fall as recovery capability degrades — a repository that reported "no errors were found" with exit code 0 then returned 6 of 7 files.→
Part III
Backup Architecture: Copies, Chains, Retention and Capacity
7 checks
- 3-2-1 read criticallyThree copies, two media and one offsite copy are counts, and an estate can satisfy every clause while one destroyed device, one credential or one scheduled run removes everything it counted.→
- Independence: media, failure domain, security domain, geographyCopy count means nothing until you can name the single event that destroys two copies at once, and that event can arrive along four axes which fail separately: media, failure domain, security domain and geography.→
- Full, differential, incremental and incremental-foreverMeasured on restic 0.19.1, a second backup of a 60 MiB tree added 2.062 KiB and still lists as a 60.000 MiB recovery point, which is why the scheme decides how many stored objects a restore depends on.→
- Restore dependency chains and the risk of chain lengthA recovery point is the transitive closure of every object a restore must read, which is why a snapshot listed at 60.000 MiB returned 6 of 7 files once a single pack in its chain became unreadable.→
- Retention schemes as patterns, not prescriptionsRetention is the only scheduled operation in a backup system whose purpose is to destroy recovery points, which is why both restic and borg print the policy decision before they act on it.→
- Deduplication and compression: the benefit and the billA second backup of a 60.000 MiB tree stored 1.370 KiB, and the same mechanism that produced that number puts the chunk index in memory, scatters the restore across packs and lets one damaged pack cost several snapshots at once.→
- Capacity planning for a recovery estateRepository size is a build-up — protected data, copy count, retention depth, change rate, measured reduction, an immutability window and a restore workspace — and one captured restic run stored 60.008 MiB for 60.005 MiB added, while a COMPLIANCE-locked object refused deletion by the account administrator.→
Part IV
Consistency, Integrity and Proof of Restorability
6 checks
- Crash-consistent, application-consistent, and the gap between themA cp -a of a live PostgreSQL data directory produced a copy that started, ran crash recovery and returned all 45000 rows, which is exactly why the practice survives and exactly what it failed to prove.→
- Filesystem consistency: open files, buffered writes and quiescingBetween an application write and durable storage sit the page cache, the filesystem journal and the device write cache, and a snapshot taken at the block layer captures only what has already reached the block layer.→
- Checksums, verification, and what a completed job does not meanIntegrity is asserted at four layers — in transport, at rest, per object, and end to end — and only the last one proves recovery, which is why plain restic check reported "no errors were found" with exit code 0 on a repository that restic check --read-data then rejected with "ciphertext verification failed".→
- Repository verification and the cost of proving data intactStructural verification is priced by metadata and full data verification is priced by the entire repository, which is why `restic check` and `restic check --read-data` both reported "no errors were found" on the same healthy repository while only one of them read 7 / 7 packs.→
- Detecting corruption before a restore needs the dataSilent corruption is defined by the absence of a signal, so detection has to be designed rather than observed: ten bytes overwritten in the middle of a 17374653-byte restic pack left the name, the size and the inode unchanged, and only a check that read the data found it.→
- Proving a restore: validation with teethValidation strength is a ladder, and a recovery is only proved by a property recorded before the incident — which is why the measured point-in-time recovery was accepted only when 50000 rows and sum(amount)=825025000 both matched the pre-DELETE state.→
Part V
Linux File-Level Backup and Restore
6 checks
- What actually has to be backed up on a Linux hostEvery path on a host is either state or a recipe, and the list written from memory is always shorter than the list a restore onto a clean machine produces.→
- rsync: what it is, and what it is notrsync -a produced a copy whose contents would have passed a checksum comparison while dropping an ACL, an extended attribute and a file capability, expanding a sparse file to 200 MiB allocated and splitting two paths that shared an inode.→
- Mirror propagation: deletion, corruption and ransomwareA nightly rsync 3.4.1 mirror reproduced an operator deletion and an in-place encryption on the very next scheduled run, with exit code 0 both times, which is why a destination without history answers only one class of failure.→
- Archive semantics: permissions, ACLs, xattrs, sparse files and hard linksOne source tree was restored four ways, every restore returned identical file contents, and two of them produced a tree whose ACL, extended attribute, file capability, sparseness and hard link were gone.→
- Restoring a single file under pressureThe most frequent recovery request is the one most often done badly, because it is done in a hurry and without a sequence: establish when the damage happened, choose the recovery point deliberately, restore into staging, compare, and only then put the file back.→
- Restoring a complete service onto a clean hostA volume archive restored the application data byte-identical while the file sitting beside it came back as No such file or directory, because only the named volume had ever been captured — which is the shape of every restore onto infrastructure that has never run the service.→
Part VI
Snapshots: LVM, Btrfs and ZFS
7 checks
- Copy-on-write: the mechanism under every snapshotA snapshot copies nothing at the instant it is taken, and the bill arrives later charged to writes against the origin: a 100 MiB LVM snapshot reached 100.00% after 100 MiB of a 600 MiB origin was rewritten, with nothing written to the snapshot.→
- LVM snapshots: sizing, growth and invalidationA 100 MiB LVM snapshot of a 600 MiB origin reached 100.00% after 100 MiB of writes to the origin and was then destroyed by the kernel rather than merely filled, which is why LVM snapshots are sized from the origin write volume and monitored on data_percent.→
- Btrfs subvolumes and snapshotsA read-only Btrfs snapshot still held the 09:00 ledger after the live copy was truncated, refused a write with exit code 2, and only survived the loss of its filesystem because it had been sent to a different one.→
- ZFS snapshots, clones and space accountingA ZFS snapshot reported 0B USED at the moment it was taken and 47.0M USED thirty minutes later, with nothing ever written to it, which is the arithmetic behind most surprise capacity incidents.→
- Replication with send/receive: when a snapshot becomes a copyAfter the production device was overwritten and zpool import reported no pools available to import, the backup pool still listed both rbdrbkp/ledger@0900 and rbdrbkp/ledger@0930, because those two points in time had been sent across a device boundary instead of retained behind one.→
- The snapshot test: why hourly snapshots may protect nothingOne destroyed device, three snapshot implementations: LVM returned nothing from vgs and lvs, ZFS reported "no pools available to import", and Btrfs failed to mount at exit code 32 — while the independent copies restored byte-identical in every run.→
- Rollback: the operation that destroys newer dataRolling a dataset back is not a restore but a replacement of its current contents, and zfs-rollback(8) will not reach a snapshot other than the most recent one until every newer snapshot and bookmark has been destroyed.→
Part VII
Block Images, Bare-Metal Recovery and Reconstruction
6 checks
- Block-level images: use cases and limitsAn image copies a device without understanding what is on it, which is the one thing it does better than a file-level backup and the source of every limit it carries — including the 200M region with 0 blocks allocated that became 200M of real blocks in a naive copy.→
- Bare-metal recovery as an architectureRebuilding a server from nothing is nine dependent stages, each with a precondition that has to exist before it runs, and an intact 11 MiB repository that answered "Fatal: wrong password or no key found" at exit code 12 is what an unmet precondition looks like.→
- Boot, partitioning and the parts nobody backs upA file-level archive restores files, and a bootable machine is a file tree plus a partition table, an EFI system partition, firmware variables in NVRAM, assembly metadata and a LUKS header — none of which has a path a backup tool would walk.→
- Rebuild versus restore: the central choiceRe-applying a complete set of Kubernetes manifests recreated every object in the namespace and left the orders file missing, because the business data was never part of the desired state.→
- What configuration management can and cannot reconstructA committed container image produced a directory holding nothing but `.` and `..`, which is the same boundary a configuration management run meets on a clean host: it rebuilds what was declared and cannot rebuild what merely accumulated.→
- Golden images, drift and the reconstruction gapA golden image plus configuration code is a promise to rebuild, and every change made to production but not to the definition is interest on it — which is why re-applying a manifest to a deleted namespace returned the same four objects with exit code 0 and nothing else.→
Part VIII
Backup Repositories: restic, Borg and Repository Failure
7 checks
- The repository model: chunks, indexes and snapshotsA backup repository stores content-addressed chunks and trees that reference them, which is why a second backup of a 60.000 MiB tree added 2.062 KiB, why forget and prune are separate commands, and why one damaged pack cost a restore six of seven files.→
- restic in production: backup, retention and pruneThe repository holds one key file, `forget` and `prune` are two commands for a reason, and a restore that reported "Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB)" exited 1 — which is why the exit code is the result of a restore and the directory listing is not.→
- BorgBackup: a different set of trade-offsBorg stored a second archive of the same 41.94 MB tree for 613 B, keeps the repository key inside the repository under repokey, and let a compromised client delete every archive with append_only set — three decisions restic made differently.→
- Repository integrity: what each level of check provesStructural verification reads bookkeeping, full data verification reads every stored object, and end-to-end verification reads the source — which is why one damaged pack produced "no errors were found" with exit code 0 from `restic check` and exit code 1 from `restic check --read-data`.→
- Repository corruption and the partial restoreA damaged repository rarely refuses to restore — measured on restic 0.19.1 it delivered "Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB)" at exit code 1 with one file wrong, and a tar chain missing its middle archive delivered a plausible directory with no error at all.→
- Losing the catalogue: metadata recoveryIf the index vanished and every stored object survived, could the data be recovered and how long would it take — a Borg repository whose index, hints and integrity files were deleted still listed day1, day2 and day3 and extracted at exit code 0, while an etcd snapshot describes itself in four values and nothing else.→
- Choosing a backup tool honestlyThe decision is eight questions about your estate rather than a comparison of two products: restic 0.19.1 and borg 1.4.0 both read every stored object during a full verification and both print a retention decision before acting on it, so what separates them is everything a feature matrix has no column for.→
Part IX
Encryption, Keys and Key Recovery
6 checks
- Encryption in transit and at rest for backup dataA restic repository survived the loss of its host completely intact — 11M on disk, six files, the key file still present — and answered every read with "wrong password or no key found" and exit code 12, which is what client-side encryption looks like from the wrong side of the passphrase.→
- The encryption paradox: the key that died with productionA repository that survived the disaster completely intact — 11 MiB, six files, every byte readable — answered `Fatal: wrong password or no key found` with exit code 12, because the passphrase it needed had lived inside the directory it was protecting.→
- Key custody, escrow and split knowledgeThe recovery team was given its own passphrase on the production repository with `restic key add`; after production was destroyed a second time, that passphrase restored the data byte-identical — 9eb4e2ad8e08e1dcaaf87ababab964b0 in and out.→
- KMS and HSM dependencies inside a recovery pathEnvelope encryption stores a wrapped data key next to the ciphertext and moves the ability to unwrap it into a service, which turns one storage problem into five availability questions that nothing in the backup system will ever ask on your behalf.→
- Rotating backup encryption without orphaning historyAdding and removing a restic passphrase changes who can open a repository without rewriting a single stored chunk, which is why `key add` followed by `key remove` is a safe rotation and replacing the master key is a different operation with a different bill.→
- Proving you can still decryptA drill that withheld everything the disaster would have taken carried an escrowed passphrase to a byte-identical restore, while the arrangement it replaced answered an intact 11 MiB repository with exit code 12.→
Part X
Object Storage, Versioning and Retention
6 checks
- Object storage as a backup target: durability is not backupA durability figure prices the platform losing an object it was asked to keep, which is not the failure that destroys backups — measured on MinIO RELEASE.2025-09-07T16-13-09Z, a delete issued with production credentials exited 0, emptied the ordinary listing, and left the 38 B object in place as version v1.→
- Versioning and delete markers: what a deletion actually didA production identity holding the built-in readwrite policy deleted a backup object at exit code 0, the object left every ordinary listing, and the 38-byte version was still stored underneath a zero-byte delete marker.→
- Lifecycle rules and the backups they quietly removeA lifecycle rule removes data correctly, on schedule, with a successful result and no alert, which is why the most dangerous configuration on a backup bucket is the one nobody reviews as a retention change.→
- Retention modes: governance, compliance, and what each promisesGovernance and compliance retention are configured through the same command and differ entirely in their threat model: measured on MinIO, an administrator removed a governance-protected version with --bypass at exit code 0 and could not remove the compliance-protected one at all.→
- Credentials, least privilege and the delete permissionThe identity that writes the nightly backup is usually the identity that can destroy it, and the MinIO capture shows why: a production credential holding the built-in readwrite policy deleted the current object at exit code 0 and was stopped only by retention the policy could not reach.→
- What varies between S3-compatible implementationsS3 compatibility is a claim about which requests an endpoint accepts, and every property a backup design depends on — enforcement, consistency, lifecycle coverage, retrieval latency — lives underneath that claim and has to be measured on the endpoint you intend to trust.→
Part XI
Immutability, Air Gap and Ransomware Resilience
8 checks
- Can the attacker delete the backups?The question is answerable by measurement rather than opinion: a production identity holding the built-in readwrite policy hid a backup at exit code 0 and a Borg client emptied an append-only repository at exit code 0, while the same estate refused every version deletion at exit code 1 once compliance retention was in force.→
- Object lock in practice: what it stops, and what it does notA COMPLIANCE-retained version refused the production identity, that identity passing --bypass, and the full administrator, all at exit code 1 — and none of those refusals stopped anyone from writing new objects into the same bucket.→
- Logical air gap and physical air gapA compromised client deleted three Borg archives from a repository with append_only = 1 and exited 0 each time, while a COMPLIANCE-retained object version refused the production identity and the full administrator alike — two logical separations of very different strength.→
- Separation of duties and the backup administratorIf one identity can both compromise production and destroy every backup, the copy count is decorative — and the only separation worth relying on is the one a storage mechanism enforces rather than an administrator observes.→
- Ransomware: determining scope and the compromise timelineThe encryption time is loud and late, the intrusion time is quiet and early, and only the second one bounds a defensible recovery point — which makes the timeline an evidence exercise conducted before any restore is authorised.→
- Selecting a clean recovery pointThe newest recovery point is the one everyone reaches for and the one most likely to carry the intrusion, so the choice has to be evidenced in an isolated restore rather than assumed from a timestamp.→
- Recovering without restoring the persistence an attacker leftA restore is faithful to whatever was in the copy, including the attacker foothold, which is why ransomware recovery rebuilds the operating system and application layers from code and restores only data into them.→
- Preserving evidence while restoring serviceRolling a Borg repository back to transaction 13 removed segments 14 through 25 — the segments that recorded the attacker deletes — and the only reason that record survived was a hard-link copy taken one command earlier.→
Part XII
Virtual Machine and Hypervisor Recovery
6 checks
- VM snapshot, VM backup, and the difference that mattersA hypervisor snapshot preserves a point-in-time view of a VM inside the storage the VM already occupies, while a VM backup writes an independent copy somewhere else — and the copy-on-write price, measured on LVM 2.03.31(2) where 10 MiB of origin rewrite consumed 10.12% of a 100 MiB snapshot, is paid by the running guest.→
- Guest quiescing and application consistency inside a VMA hypervisor snapshot holds only what the guest storage stack had already written, which is why an uncoordinated copy of a live PostgreSQL 18.6 data directory started, ran crash recovery and returned all 45000 rows while establishing nothing about which instant it came from.→
- Backing up a virtualisation estate: what else is stateA restored guest definition is almost entirely references — a storage name, a bridge name, a pool — and a VM backup preserves the references without preserving anything they refer to.→
- Restoring a VM: the parts that are not the disk imageA restored disk image is a restored disk, not a restored service: the definition around it — identity, sizing, machine type, disk controller, bridge and VLAN, firmware mode — decides whether the guest boots at all and whether anything can reach it.→
- Hypervisor loss with the repository intactA hypervisor is permanently gone and the backup repository is healthy, which makes the recovery a sequencing problem rather than a restore problem — decided by what the host held, what the survivors can carry, and the order and pace of the restores.→
- Losing an entire virtualisation clusterAn intact 11 MiB restic 0.19.1 repository answered `Fatal: wrong password or no key found` with exit code 12 once the host holding its passphrase was gone — and that repository is only the fifth of seven inputs a whole-cluster recovery needs before the first guest starts.→
Part XIII
Container and Kubernetes Recovery
8 checks
- A container image is not a backupA file written into a container layer was captured by docker commit and a file written into a mounted volume was not, which is why an image registry full of images is not evidence that any container state is protected.→
- Volumes, bind mounts, and where the state actually isA named volume reported Mountpoint /var/lib/docker/volumes/rbdr-data/_data and Driver local, which is a fact about one engine rather than the interface a backup should be built on.→
- Backing up container data with consistencyCopying a volume while the workload writes to it reproduces every Part IV consistency problem inside a smaller box: measured on PostgreSQL 18.6, a live copy of a data directory started, ran crash recovery and returned all 45000 rows, while the same directory without pg_wal panicked with "could not locate a valid checkpoint record at 0/2F20158".→
- The Kubernetes backup model: four separate thingsRe-applying a complete set of manifests after a namespace deletion recreated every object in six seconds and left the volume empty, which is why a Kubernetes estate needs four protections rather than one backup.→
- etcd: snapshot and restoreMeasured on etcd 3.7.1: `snapshot status` and `snapshot restore` are no longer etcdctl subcommands, a restored snapshot reverted a Deployment from replicas=12 to replicas=6 without reporting anything, and the cluster id turned out to be derived from --initial-cluster rather than carried inside the file.→
- Persistent volumes and CSI snapshotsOn k3s v1.36.3+k3s1 the claim came back Bound to a different volume and the orders file did not come back at all, because the reclaim policy — not the delete — decided what happened to the bytes.→
- Velero: what it covers and what it does notVelero uploads a tarball of API objects to a bucket and moves volume data by a road its storage provider has to supply, which is why "we run Velero" and "Kubernetes is backed up" are different statements.→
- Recovering a cluster onto clean infrastructureA full-loss Kubernetes recovery is five ordered steps, and the capture behind this lesson shows why: re-applied manifests rebuilt every object and the Pod reached Ready after 6s against an empty volume, and only a separate data restore made the service correct.→
Part XIV
Database Backup and Point-in-Time Recovery
6 checks
- Why copying live database files is not a backupMeasured on PostgreSQL 18.6: a cp -a of a live data directory taken mid-workload started, ran crash recovery and returned all 45000 rows, while the same directory without pg_wal died with "PANIC: could not locate a valid checkpoint record at 0/2F20158".→
- Logical and physical backups comparedA base backup that exited 0 and contained 45000 rows was replayed forward to a chosen instant and returned 50000 rows; a logical dump of the same data would instead restore one table at a time, and onto a newer major version. Neither family does the other job.→
- WAL archiving and the continuity of the archivePoint-in-time recovery is bounded by the archive rather than by the base backup, and the measured recovery reached its target over an archive that pg_stat_archiver reported as archived=6 failed=0 with an unbroken sequence of segments behind it.→
- Performing a point-in-time recoveryA base backup holding 45000 rows was replayed forward over 5000 later inserts and stopped before an unqualified DELETE, and the recovery log named the stopping point — the commit of transaction 836 — before a single row was counted.→
- Recovering from a logical mistakeAn unqualified DELETE is a valid transaction, so every replica applies it faithfully, and the recovery that keeps the rest of the day is a side-by-side copy recovered to just before the statement rather than a rewind of production.→
- Validating a recovered databaseA recovered cluster that accepts connections has established almost nothing, which is why the measured point-in-time recovery was accepted only once 50000 rows and sum(amount)=825025000 both matched figures recorded before the DELETE.→
Part XV
Infrastructure Reconstruction: IaC, Config, Network and Identity
8 checks
- Code rebuilds infrastructure; backup restores stateRe-applying a complete declarative repository after a deletion returned every object and left the volume empty, which is the general result that splits an estate into what code rebuilds and what only a backup returns.→
- Terraform state as critical recovery materialA state file maps declared resources to the real identifiers a provider issued, so losing it leaves the infrastructure running and unmanageable while the next apply proposes to create everything that already exists.→
- Ansible and the boundary of reconstructionA playbook reconstructs the packages, users, directories, unit files and templates that some task names, and nothing else; run in the wrong order against a restored host, the same idempotent tasks overwrite the data the restore just returned.→
- Git, pipelines and artifacts in a recoveryEvery clone is a genuine copy of history and of nothing else the forge holds, so a recovery that has to build depends on a pipeline, a runner, a registry and a set of credentials that all live inside the estate being recovered.→
- Network device configuration recoveryA few hundred kilobytes of switch, router and firewall configuration is the smallest irreplaceable thing in the estate, covered by no backup agent and needed before anything that recovers over the network.→
- Firewall and routing recoveryA firewall configuration is a set of statements about a topology, so restoring it into a different topology produces a device that boots, reports healthy and forwards nothing — or forwards the wrong thing.→
- DNS, DHCP and IPAM as recovery dependenciesName resolution is a prerequisite of nearly every service and is nearly always absent from the recovery order, and the time-to-live that governs how fast a repointed name takes effect is a parameter you can only set before the incident.→
- Identity, secrets and PKI: the bootstrap problemThe secret manager presents a certificate issued by a CA whose signing key is sealed in the secret manager, and a graph containing that pair has no first node — the only remedy is material held outside the estate, which is why an intact repository answered `Fatal: wrong password or no key found` with exit code 12 while an escrowed passphrase opened the same repository and exited 0.→
Part XVI
Monitoring, Restore Testing and Recovery Assurance
8 checks
- Monitoring the backup estate: the signals that matterA backup estate has to emit seven classes of signal, and the one almost every estate omits is the liveness of the checks themselves — which is why a repository that reported "no errors were found" at exit code 0 while holding an unrecoverable file is only the second-worst monitoring failure available.→
- Restore-point age, not job successA wrapper script reports the exit status of its last command; the repository reports when it last accepted a snapshot for a host and a path. Only the second number has a threshold that can be derived from an objective rather than tuned until it stops paging.→
- Alerting on recovery capabilityThree ages — newest recovery point, newest completed data verification, newest proven restore — each decay for different reasons and each demand a different repair, which is why one alert per backup job produces volume and three age alerts produce signal.→
- Automated restore verificationA check that restores a recovery point into an isolated target, compares it against a manifest recorded at backup time and exits non-zero on any mismatch — plus the two wrapper defects that would have reported the measured partial restore as a pass.→
- Recovery testing: a maturity progressionA staged progression from reading one file out of a recovery point to a full exercise with failback, where the measured k3s capture shows level four finding a backup nobody had taken while every object came back Running.→
- Restoring onto clean infrastructure to expose hidden dependenciesA rebuilt namespace reached Ready after 6s with its claim Bound and its data directory holding nothing but . and .., which is what a clean target is for — it is the only environment in which an omitted dependency has to announce itself.→
- Business-level validationA recovered system that runs is not yet a recovered service, which is why the measured point-in-time recovery was accepted on sum(amount)=825025000 recorded before the incident, and why the matching row count on its own would have accepted a plausible fake.→
- A failed restore is a production incidentA restore test that fails has discovered that a production system is currently unrecoverable, which is an exposure running now rather than a defect to schedule, so it needs an owner, a clock, a blast radius and a successful restore before it closes.→
Part XVII
Disaster Recovery: Failover, Failback and the Recovery Estate
10 checks
- Disaster recovery is not high availabilityHigh availability keeps a failure out of the user experience with capacity that is already running; disaster recovery rebuilds a service from a state captured before the failure — and the test that separates them is whether the mechanism holds an earlier state or only the current one.→
- Cold, warm and hot: what each actually costsCold, warm and hot describe how much of the recovery estate is already built, not how long recovery takes — and each carries a cost that never appears on the infrastructure invoice.→
- Active/passive and active/active for disaster recoveryActive/passive keeps one authoritative site and a cutover you can rehearse; active/active keeps two and buys a reconciliation problem that runs continuously — and neither one holds a state from before the mistake.→
- DNS, routing and certificates during a failoverThree mechanisms actually move a client from one site to another, each bounded by a cache or a router you do not operate, and every one of them can be defeated by a certificate the recovery site cannot serve.→
- Secrets, identity and third-party dependencies in DRA secret manager that comes up sealed at the recovery site, an identity provider that cannot authenticate the people who would unseal it, and third parties whose allowlists name only the primary addresses are dependencies no rehearsal finds while the primary is still reachable — and on restic 0.19.1 an intact repository answered `Fatal: wrong password or no key found` with exit code 12 until a passphrase held outside the estate opened it and exited 0.→
- Executing a failoverA failover declared thirty minutes late costs the whole failover duration on top of the time already lost, which is why the declaration belongs to a named person working against a clock rather than to whoever is closest to the repair estimate.→
- Operating in DR, and the data you create thereA failover moves the workload and leaves the protection behind, so from the moment services answer at the recovery site the data being created there exists in exactly one copy — which is why the first task after the estate is up is not tuning, not re-enabling deferred jobs and not the timeline, but a backup the recovery site can take on its own.→
- Failback: the second outageThe primary holds the state it had at the instant it failed, the recovery site holds everything written since, and reconciling the two is a data problem before it is an infrastructure one — which is why failback, run by a tired team after the incident is notionally over, is where the second outage happens.→
- DR exercises from tabletop to fullFive exercise stages from tabletop to full failover with failback, the boundary each one crosses, and the two conditions — an unavailable primary and an acceptance test that compares data — without which the whole programme is a demonstration.→
- Incident command and human factors during recoveryA recovery has four jobs — deciding, executing, communicating and recording — and the common default of one person holding all four produces a failback nobody can reconstruct, a handover that transfers tasks instead of state, and an RPO reported as a reassurance rather than a measurement.→
Part XVIII
Backup Platform DR, Media, Cost and Compliance
9 checks
- When the backup platform is the thing that failedEvery service in the estate has a recovery procedure except the system that holds the recovery material for all of them, and the first move in a platform outage is not a rebuild — it is recovering one file from the storage with nothing but the tool and the key, which on restic 0.19.1 returned `Restored 7 files/dirs (60.000 MiB)` at exit code 0 with no server, catalogue database or client registration involved.→
- Rebuilding a backup catalogueA repository whose index is derived can be scanned back into a listable state, and the scan still returns nothing about retention intent, client mapping or what a name meant — measured on borg 1.4.0, a repository with its index, hints and integrity files deleted listed day1, day2 and day3 again, while the rule that kept day2 arrived on the prune command line.→
- Backup network design and restore throughputThe network is a stage of the RTO with a duration nobody times, and a link sized for a nightly delta is not a link sized for a full restore — the measured capture backed up 400 MiB in 1.49s, repeated it unchanged in .73s, and needed 1.07s to read it back.→
- Tape: what it is still good atA cartridge on a shelf has no interface for a credential to authenticate to, which is the strongest air gap available, and the LTO Program states that LTO-9 drives read and write LTO-8 and LTO-9 media only, which turns any retention longer than the read window into a scheduled migration.→
- Cloud archive tiers and the retrieval that breaks RTOAn archive storage class is cheap because retrieval is slow and charged, and a documented retrieval latency measured in hours is a serial stage that sits in front of every byte a restore will move.→
- Media lifecycle, bit rot and decommissioningTen bytes overwritten in the middle of a 17374653-byte pack left the name, the size, the inode and the link count unchanged, and the integrity check that never reads the data still exited 0 — which is why long retention needs a verification rotation, a migration date and a destruction record.→
- Cost engineering for a recovery estateEvery control in this course is a line item somebody will eventually ask to remove, and an engineer who cannot price the control loses the argument to someone who was never told which failure it was holding back.→
- Retention, legal hold and data sovereigntyThree obligations argue for three different retention numbers, and an estate that records only the number cannot say later which one it was answering — while the storage that refuses an attacker also refuses an erasure request.→
- A defensible reference architectureA recovery estate is defended in a review as a list of properties, each attached to the failure it removes and to a measurement that shows it is present — not as a product list, a diagram, or a job success rate.→
Part Capstone
Production Capstone
1 check