Backup, Restore & Disaster Recovery for Production Infrastructure

Part I

Recovery Objectives, Vocabulary and Failure Models

Why recovery rather than backup is the objective, the seven words that are not synonyms, naming the failures you are actually protecting against, classifying data by whether it can be reconstructed, and building the first inventory and dependency map.

6 lessons

Part II

RPO, RTO and Recovery Sequencing

Engineering an acceptable loss window, counting everything that happens between the incident and a working service, estimating restore duration from evidence, service tiers, the recovery dependency graph, and measuring the promise you made.

6 lessons

Part III

Backup Architecture: Copies, Chains, Retention and Capacity

3-2-1 read critically, independence of media and failure and security domains, full and differential and incremental and incremental-forever, restore dependency chains, retention as a pattern rather than a prescription, deduplication and compression, and capacity planning.

7 lessons

Part IV

Consistency, Integrity and Proof of Restorability

Crash-consistent versus application-consistent and the gap between them, open files and buffered writes and quiescing, checksums and what a completed job does not mean, repository verification, detecting corruption before a restore needs the data, and validation with teeth.

6 lessons

Part V

Linux File-Level Backup and Restore

What actually has to be backed up on a Linux host, rsync and what it is not, mirrors that propagate deletion and corruption and ransomware, archive semantics for permissions and ACLs and extended attributes and sparse files, and restoring both a single file and a complete service.

6 lessons

Part VI

Snapshots: LVM, Btrfs and ZFS

Copy-on-write as the mechanism under every snapshot, LVM snapshot sizing and growth and invalidation, Btrfs subvolumes, ZFS snapshots and clones and space accounting, replication that turns a snapshot into a copy, the snapshot test, and rollback as an operation that destroys newer data.

7 lessons

Part VII

Block Images, Bare-Metal Recovery and Reconstruction

Block-level images and their limits, bare-metal recovery as an architecture, the boot and partitioning state nobody backs up, the choice between rebuilding and restoring, what configuration management can and cannot reconstruct, and the drift that opens the reconstruction gap.

6 lessons

Part VIII

Backup Repositories: restic, Borg and Repository Failure

The chunk and index and snapshot model, restic and BorgBackup as two different sets of trade-offs, what each level of repository verification actually proves, corruption that survives a shallow check, losing the catalogue, and choosing a tool honestly.

7 lessons

Part IX

Encryption, Keys and Key Recovery

Encryption in transit and at rest for backup data, the paradox of a key that died with the environment it protected, key custody and escrow and split knowledge, KMS and HSM dependencies inside a recovery path, rotating without orphaning history, and proving you can still decrypt.

6 lessons

Part X

Object Storage, Versioning and Retention

Why durability is not backup, versioning and delete markers and what a deletion actually did, lifecycle rules that quietly remove recovery points, governance and compliance retention modes, the delete permission as the control that matters, and what varies between S3-compatible implementations.

6 lessons

Part XI

Immutability, Air Gap and Ransomware Resilience

Whether an attacker who owns production can destroy the backups, object lock in practice, logical versus physical air gaps, separation of duties for backup administration, establishing a compromise timeline, selecting a clean recovery point, avoiding the restoration of persistence, and preserving evidence.

8 lessons

Part XII

Virtual Machine and Hypervisor Recovery

VM snapshot against VM backup, guest quiescing and application consistency inside a guest, the estate state that is not a disk image, restoring a VM into a working network, hypervisor loss with the repository intact, and losing an entire virtualisation cluster.

6 lessons

Part XIII

Container and Kubernetes Recovery

Why an image is not a backup, where container state actually lives, backing up volumes with consistency, the four separate things a Kubernetes backup has to cover, etcd snapshot and restore, persistent volumes and CSI snapshots, the boundary of Velero, and recovery onto a clean cluster.

8 lessons

Part XIV

Database Backup and Point-in-Time Recovery

Why copying live database files is not a backup even when it appears to work, logical against physical backups, WAL archiving and the continuity of the archive, performing a point-in-time recovery to a chosen target, recovering from a logical mistake, and validating the result.

6 lessons

Part XV

Infrastructure Reconstruction: IaC, Config, Network and Identity

Code rebuilds infrastructure while backup restores state, Terraform state as critical recovery material, the boundary of configuration management, Git and pipelines and artifacts, network device and firewall configuration, DNS and DHCP and IPAM, and the identity and PKI bootstrap problem.

8 lessons

Part XVI

Monitoring, Restore Testing and Recovery Assurance

The signals that indicate recovery capability, restore-point age instead of job success, alerts worth waking someone for, automated restore verification, a testing maturity progression, restoring onto clean infrastructure to expose hidden dependencies, business-level validation, and treating a failed restore as an incident.

8 lessons

Part XVII

Disaster Recovery: Failover, Failback and the Recovery Estate

Why disaster recovery is not high availability, what cold and warm and hot actually cost, active/passive against active/active, DNS and routing and certificates during a failover, secrets and identity and third-party dependencies, executing a failover, the data created in the recovery site, failback as a second outage, exercises, and human factors.

10 lessons

Part XVIII

Backup Platform DR, Media, Cost and Compliance

Recovering when the backup platform is what failed, rebuilding a catalogue, backup network design and restore throughput, what tape is still good at, cloud archive tiers whose retrieval breaks an RTO, media lifecycle and decommissioning, cost engineering, retention and sovereignty, and a defensible reference architecture.

9 lessons

Part Labs

Labs

Disposable-environment labs covering classification and RPO/RTO calculation, file restore, rsync propagation, archive semantics, LVM and ZFS and Btrfs snapshots, repository integrity and corruption, object storage versioning and object lock, container and Kubernetes and etcd recovery, database point-in-time recovery, key recovery, automated restore verification, a measured RTO, and a failover with failback.

0 lessons

Part Runbooks

Runbooks

Operational procedures for restoring files, services, virtual machines, container and Kubernetes data and databases, for investigating failed backups and failed restores and repository corruption and capacity exhaustion, for recovering encryption keys and the backup control plane, and for ransomware recovery, failover, validation, failback and total site loss.

0 lessons

Part Checklists

Checklists

Production-readiness reviews for backup and restore readiness, RPO and RTO, ransomware resilience, immutability, repository security, database and virtual machine and Kubernetes and network device recovery, secrets and PKI recovery, pre-restore and post-restore validation, and disaster-recovery site and exercise readiness.

0 lessons

Part Breakfix

Break/Fix Scenarios

Evidence-first diagnosis of green jobs that cannot restore, exhausted repositories, unavailable keys, broken incremental chains, corruption that passes a shallow check, lifecycle rules that deleted the recovery points, inconsistent backups, unreachable recovery targets, and the dependency failures that stop a technically successful recovery from restoring the service.

0 lessons

Part Capstone

Production Capstone

Design, build and defend a complete recovery estate for a realistic production site: classified data, engineered RPO and RTO, immutable and offsite copies, recoverable keys, proven restores, monitored recovery capability, a rehearsed failover and failback, and twelve injected incidents to survive.

1 lesson

Part Final

Final Assessment

Final theory assessment of operational judgement, plus a final practical assessment of an inherited backup and disaster-recovery estate carrying realistic coverage, consistency, immutability, key-custody, retention and recovery-time defects.

0 lessons