Losing an entire virtualisation cluster · Backup & DR

Losing one hypervisor with the repository intact left most of the estate standing, and that is what made it a bounded problem: there was still a management platform to log into, a cluster for a replacement node to join, credentials that worked, and a name that still resolved. Remove all of it at once — the room, the rack, the site, the account — and none of those remain. What is left is a set of files somewhere else and a set of people who have to turn them back into a running estate, in an order that nothing in the estate is still around to remind them of.

The inputs, and the order they are needed in

The instinct in a total-loss exercise is to reach for the backup repository first. That is the wrong end of the sequence. A restore command needs somewhere to write, a platform to write into, and permission to read the source, and none of those exist yet. Written as an ordered list of inputs rather than an ordered list of systems, a cluster recovery needs:

Hardware, or replacement capacity. Physical machines, a rented rack, or an account with a provider that will sell you instances today.
A management platform, installed and configured. Bare hypervisors on bare hosts do not constitute a cluster and cannot start a guest that the estate defined.
The cluster configuration: storage definitions and network definitions. Which storage exists, under which name, with which credentials; which bridges and VLANs the guests expect to be attached to.
The authentication configuration and certificates, so that the platform can be administered at all — accounts, realms, the trust that the API and the web interface present, and whatever the automation authenticates with.
The backup repository, reachable, and its credentials. Reachable is a network claim and a name-resolution claim; credentials is a separate claim from both.
The encryption key for that repository. Distinct from the credentials that reach it, and, as measured below, not implied by them.
The guests, in dependency order. Only now, and not in the order the inventory happens to list them.

The ordering is not a preference. Each item is a precondition for the next, and skipping forward produces the familiar recovery-day discovery that the step you are on cannot begin. Two of the items deserve emphasis because they are the ones most often assumed rather than held.

Item three is where the estate state that is not a disk image lives. A guest’s definition — its memory size, its disks and where they are, its network interfaces and their bridges, its boot order — is platform configuration, not guest content, so no restore of the guest’s filesystem produces it. In Proxmox VE the documentation is explicit about where that configuration sits: the Proxmox Cluster File System, pmxcfs, is a database-driven filesystem mounted at /etc/pve whose contents are replicated to all cluster nodes using Corosync, backed on each node by a database file at /var/lib/pve-cluster/config.db. The files inside it include corosync.conf, storage.cfg, user.cfg and the per-node guest configuration under nodes/. Replication across nodes is exactly the property that stops helping when every node is gone at the same time, which is why that filesystem belongs in a backup that leaves the cluster.

Item four is the one people forget is a recovery input at all, because during normal operation nobody thinks of “being able to log in” as a dependency. The documented behaviour of a Proxmox VE cluster makes the point sharply in another direction: pmxcfs becomes read-only when the node loses quorum, so a partially rebuilt cluster can be a cluster you can look at and cannot change. Access, quorum and trust are conditions on the platform being usable, and they are recovered before data, not alongside it.

The dependencies that pointed back at the cluster

An estate diagram drawn during normal operation has arrows running from services towards the cluster. A recovery needs the arrows drawn the other way: for each input above, what did obtaining it depend on? The uncomfortable answers are the ones where the arrow points back into the thing that has been lost.

Recovery input	Where it commonly lives	What its loss does to the sequence
Object storage or repository credentials	A secret manager, frequently a guest on the cluster	Step 5 cannot start; the repository is reachable and refuses you
The repository’s DNS name	A resolver that was a guest on the cluster	The name does not resolve, and the address was never written down
Platform administrative accounts	An identity provider integrated into the platform, often on the cluster	Step 4 stalls with no local account known to work
Certificates and the CA that issued them	An internal CA, often a guest on the cluster	Trust must be re-established before the API is usable
The written recovery procedure	A wiki on the cluster	The sequence must be reconstructed from memory under pressure
The repository encryption passphrase	A file on a protected host, or a password manager on the cluster	Step 6 fails on data that is perfectly intact

Every row is ordinary engineering. A secret manager exists so that credentials are not scattered across hosts, a wiki so that procedures are written down, an internal CA so that certificates are issued centrally. Each is good practice, and each, run on the cluster it helps operate, becomes a circular dependency that is invisible until the cycle has to be broken from outside.

The test is mechanical, and it is worth applying literally rather than generally. Take the seven inputs, and for each one ask which system you would have used to obtain it. If that system was a guest on the cluster, the input is not available in the scenario you are planning for, whatever the runbook says. This is uncomfortable because the honest answer for most estates is that several rows fail at once, and because the fix is not technical cleverness but the discipline of holding a small number of things somewhere else.

`Fatal: wrong password or no key found` on an intact repository

One of those rows was measured rather than argued, and it is the cleanest demonstration of the category. A restic repository was created in the ordinary way, with the passphrase held in a file on the protected host — in this capture /work/prod/.restic-pass, inside the very directory being backed up. A backup was taken and completed. Then the host was destroyed, as it would be in a site loss, and the repository, which lived elsewhere, survived exactly as designed.

Data-loss riskan intact repository, and the command that could not open it— EXECUTION VALIDATED on restic 0.19.1 compiled with go1.26.4 on linux/amd64, from the key loss and escrow capture recorded for this course.

$ restic snapshots

--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

--- attempting a restore with no passphrase ---
$ restic snapshots
Fatal: wrong password or no key found
>>> exit code: 12

Read what that exit code is not. It is not corruption: no pack was damaged, and the 11 MiB of repository content is exactly what the backup wrote. It is not a media failure, and it is not a truncated transfer — the capture records six files in the repository, all present. The bytes survived the disaster; the ability to read them did not, because the one input never copied off the host was the input that turns those 11 MiB back into data.

That is the general shape of the whole-cluster problem in miniature. The backup strategy was sound in the dimension it was designed for — a second copy, in a second place, surviving the loss of the first. It failed in a dimension nobody had listed as part of the strategy, which is that a recovery input stored inside the failure domain is not a recovery input.

Why possessing a repository is not possessing its contents

The repository in the capture was not missing its key. Listing the repository’s keys/ directory returned exactly one file, named d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e, and it was there throughout the failure above.

That file is the master key, stored encrypted under a key derived from the passphrase. The data packs are encrypted with the master key; the master key is encrypted with what the passphrase derives; the passphrase itself is stored nowhere in the repository, which is the entire point of the design. So the repository is self-contained in the sense that everything needed to decrypt it is inside it except the one thing that must not be — and copying the repository offsite faithfully copies the lock along with the door.

Generalise past restic and the principle survives. Every recovery input is either data, which replication is designed to carry, or an unlock — a passphrase, a private key, a credential, a token — which replication is deliberately designed not to carry. A copy strategy therefore cannot, by construction, deliver the unlocks. They travel by a separate mechanism or they do not travel at all, and the same reasoning applies to the credentials a platform holds for its backing storage, to its own certificate private keys, and to the API tokens the automation uses.

That is why a whole-cluster recovery has two inventories and not one: what has been copied, and what can open it.

Two arrangements that only look like they are outside the failure domain

“Store the key on the backed-up server” is the anti-pattern the transcript measures, and it rarely looks like a decision. It looks like a passphrase file next to the script that uses it, on the host the script protects, because that is the arrangement the script needs to run unattended. The capture above put the file at /work/prod/.restic-pass for exactly that reason and the result was exit code 12 on undamaged data.

Its close relative is keeping the credentials, keys and procedures in a secret manager or wiki that runs as a guest on the cluster. That looks like the opposite of the first anti-pattern, and in the whole-cluster scenario it fails identically, because the platform holding the unlocks is inside the failure domain that has just been lost.

An independent recovery kit, and the test that makes it real

The mitigation is small, unglamorous, and has to be assembled deliberately because nothing in normal operation produces it as a by-product. Call it a recovery kit: the minimum set of things that lets a competent stranger begin the sequence from item one, held somewhere that is not the estate.

It contains four categories. Credentials, meaning the accounts and API keys that reach the backup repository and the replacement capacity, kept as recovery-specific credentials rather than copies of production ones. Keys, meaning the passphrases and private keys that open what those credentials reach. The network plan, meaning addresses and not only names — the repository endpoint by IP as well as by hostname, the address ranges the guests expect, the VLAN identifiers, and the upstream that must be re-pointed — because the resolver that translated names was itself a guest. And the procedure, meaning the ordered sequence above written out concretely for this estate, since the wiki that used to hold it is gone.

For the key category, the measured mechanism is a second repository key rather than a second copy of the same passphrase. Registering one is a single command: in the capture, restic key add --new-password-file /work/recovery-pass reported saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90 and exited 0, after which restic key list showed two keys against one repository. Neither passphrase can derive the other; both decrypt the same master key. The recovery passphrase is then held by people who do not operate production, and the same disaster plays out differently.

Configuration changethe same disaster, with a passphrase held outside the estate— EXECUTION VALIDATED on restic 0.19.1, from the same capture, after a second key was registered against the repository and production was destroyed again.

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

$ restic --password-file /work/recovery-pass snapshots
ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots
>>> exit code: 0

$ restic --password-file /work/recovery-pass restore latest --target /work/rec
restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

The same capture shows the other mechanism on borg 1.4.0: exporting the key material itself as a printable block with borg key export --paper, for a safe in a different building. Its instructive property is stated there: the printed block is still the encrypted key material, so it remains useless without the passphrase, and the two must be escrowed separately and to different custodians.

None of this counts until it has been used. A kit is a claim, and the evidence for it is a restore performed using only the kit, by someone not allowed to look anything up in the estate, onto hardware that is not part of the cluster. That exercise is where you discover that the runbook references a hostname, that the recovery credential expired, or that the network plan records VLAN numbers and not the gateway address.

The guests come last, in an order the estate imposes

Only after the first six inputs are satisfied does the restore of guests begin, and the order matters more here than in any single-host recovery, because nothing is running to serve the dependencies. Directory and identity services come first because the platform and the applications authenticate against them. Name resolution comes next because almost everything else is configured by name. The certificate authority follows if internal TLS is expected. Then the databases, and only then the application tiers that connect to them, and last the reporting and batch systems that nobody notices for a day.

Two consequences follow. This order is a property of your estate and cannot be derived on the night: it comes from a dependency graph that has to have been written down while the estate was healthy. And early guests must be startable without the services that normally support them — a database VM whose configuration resolves its storage by a name nothing yet resolves is not first in any usable order, however high it sits on the priority list. Recovery is the one occasion when that graph is walked from its roots, and roots are the part a running estate never asks you to identify.

Production discipline

Write the recovery sequence as an ordered list of inputs, not of systems. Capacity, platform, cluster configuration, authentication and certificates, repository reachability and credentials, encryption key, then guests — and confirm for each input which system you would use to obtain it.
Hold no recovery input inside the failure domain it protects. In the measured case the passphrase file sat at /work/prod/.restic-pass, inside the backed-up directory, and an entirely intact repository — 11M, 6 files — returned Fatal: wrong password or no key found at exit code 12.
Register a second repository key held by people who do not operate production. restic key add --new-password-file /work/recovery-pass returned saved new key with ID 66c34166... and exited 0, and the later restore with that passphrase exited 0 with recovered md5 matching original md5 at 9eb4e2ad8e08e1dcaaf87ababab964b0.
Record the network plan by address as well as by name. The resolver that turned the repository endpoint into an address was itself a guest, so the kit carries endpoints, address ranges, VLAN identifiers and gateways, not only hostnames.
Test the kit from outside the estate, with the estate treated as absent. A recovery kit that has never been used to complete a restore on unrelated hardware is an unverified claim, and the whole-cluster scenario is the one where unverified claims are discovered simultaneously.

Cross-course references

Proxmox VE for Production Operators — Part XIV (Disaster Recovery) is where the platform-side half of this sequence is performed on a real product: reinstalling the management platform and restoring the cluster configuration, which are items two and three above and are the prerequisites for the repository work this course concentrates on.
Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) covers the platform that holds the credentials and keys enumerated in the recovery kit, and therefore also the circular dependency measured here, where the platform that stores the unlocks runs on the estate those unlocks are needed to recover.
Ceph & Distributed Storage for Production Sysadmins — Part LXXXII (Hyper-Converged Ceph) describes the arrangement in which the storage layer and the compute layer are the same machines, which collapses items one and three of this sequence into a single loss and makes the independent recovery kit correspondingly more important.