Hypervisor loss with the repository intact

Restoring a single VM ended with a guest that boots, holds the data it should and answers on its own address, and the whole procedure fitted inside one person’s attention. This lesson multiplies that by thirty and removes the machine they all ran on. A hypervisor has failed in a way it will not come back from — a board, a backplane, a suppression discharge — and every guest it carried is down. The repository is healthy and reachable, which is the good news and also what makes the next hour dangerous: with the copies right there, the reflex is to start restoring immediately, and the order and pace of those restores decide whether the estate returns inside its documented recovery time or several hours outside it. No hypervisor was available to this course, so nothing below quotes hypervisor output; the sequence and the two Proxmox VE behaviours it names come from the documentation cited above.

Establish what the host held before you restore anything

The first difficulty is that the authoritative list of what ran on the failed host was, until an hour ago, held on the failed host. Assembling it means combining what survived: the cluster’s view from a surviving node, the configuration management inventory, the address management records, and — most importantly — the backup catalogue, the only source that describes what can be recovered rather than what merely existed.

Reconciling those lists produces three groups, treated very differently.

The first group is guests whose disks live on shared storage that is still healthy. Nothing about their data was lost. The host loss was a compute event for them, and the correct action is to start them somewhere else, which is minutes of work rather than hours. Restoring these from the repository is the most expensive mistake available here: it spends the scarce read path on data that was never in danger and replaces a current disk with an older copy. Confirm the storage location from the surviving cluster’s configuration, not from memory.

The second group is guests whose disks lived on the host’s local storage. These are genuinely gone and are the real restore workload. Their recovery point is whatever the catalogue holds — a fact to establish now rather than to discover per guest during the restore.

The third group is state that belonged to the host itself and is not a guest at all: its network configuration, its bridges and VLAN trunks, an ISO and template store, a local staging area for backups, a share it exported, a hardware licence, and — on a converged design — its contribution to the storage cluster. Almost none of this appears in a guest backup catalogue, and the bridge configuration is the item most reliably forgotten.

Two further findings belong in the same pass. The first is the gap between inventory and catalogue: a guest that was never in a backup job, or whose newest capture predates a change that mattered, is a loss to report now rather than at hour three. The second is quorum. A permanently lost node also removes a vote, and Proxmox VE documents removing one with pvecm delnode run from a surviving node, warning that the removed node must not power on again in the existing cluster network with its current configuration. Do that when quorum actually requires it, not as a reflex in the first ten minutes.

Capacity decides what runs; the tier register decides what runs first

Before any restore is started, one arithmetic question has to be answered: can the surviving hosts carry the workload the lost host was carrying? Memory is usually the binding constraint, because it cannot be oversubscribed without paying for it in swap; CPU is softer, and a fleet at moderate utilisation will tolerate a period of contention. Storage has two separate answers — whether the target has free capacity for the restored disks, which thin provisioning makes easy to get wrong, and whether it has the write throughput to absorb them while still serving what survived.

If the answer is yes with headroom, the capacity question closes and the work becomes ordering. If it is no, there are exactly three responses, and each is a decision with an owner rather than a technical detail. Provision replacement capacity: a spare host, one borrowed from a lower environment, or rented capacity, each costing lead time that starts when somebody agrees to it. Run a subset: everything above some line comes back now and the rest waits. Run degraded: everything comes back with reduced memory and CPU, which works for stateless front-ends and fails badly for anything that sizes a cache or a connection pool from what it was given.

The subset option is where the service tiers from Part II stop being a document and become an instruction. A tier register is a pre-made decision about which systems get scarce recovery capacity first, taken with the people who carry the consequence and written down while nobody was under pressure. If it exists, “which twelve of these thirty come back first” is answered by reading, and the answer carries the authority of whoever agreed it. If it does not, the decision is taken live, at speed, in front of whichever stakeholder is most present rather than the one whose service matters most.

One check belongs here, before the ordering work begins: find out what is already running. Where high availability is configured and the disks are on shared storage, guests may have restarted on surviving nodes without anybody asking, so restoring one of them would produce a second, identical machine. Read the surviving cluster’s guest list and subtract it from the work.

Restore in waves, in dependency order

Two orderings feel natural during an incident and both are wrong. The first is whatever order the inventory printed in — by identifier, by name, alphabetically — which correlates with nothing but when the guests were created. The second is by size: smallest first, which maximises green rows on a status update and can restore a dozen machines nobody is waiting for, or largest first, which hands the entire path to one job while everything else queues behind it.

The order that works comes from the recovery dependency graph. Time synchronisation, name resolution, identity, certificate issuance and the secret store precede the databases; the databases precede the applications that open connections to them; and the monitoring you intend to judge the recovery with has to be running before it can tell you anything. An application started before its database does not wait politely. It crash-loops, its restart backoff grows so that it is slow to recover even once the database appears, it fills the logs people are trying to read, and it pages a team that is already busy. Some do worse: a service that initialises against an empty or unreachable store can register a failed state, or begin serving a page announcing that the data is gone — a customer-visible event generated by the recovery itself.

Two mechanics make wave ordering practical. The first is that restore and power-on are separate acts: leaving recovered guests powered off until their wave’s prerequisites are validated turns a race into a sequence. The second is concurrency, and restoring a whole wave at once is the trap this scenario is famous for.

A restored guest with a live twin is the second incident

A recovered guest carries the network identity it had when the backup was taken: the same MAC address, the same bridge and VLAN tag in its configuration, and inside the guest either a static address or a DHCP client holding a reservation. That fidelity is what you want when the original is gone, and what makes the failure mode dangerous when it is not.

Start with the mundane version. The bridge or VLAN the guest expects does not exist on the replacement host, so the guest boots perfectly, reports healthy to itself, and is unreachable. The host’s network configuration is not in any guest catalogue, so provisioning replacement capacity means reproducing the bridges and trunked VLANs before the first guest is attached.

The serious version is the duplicate. If the guest is already running somewhere, powering on a copy with the same address puts two machines with one identity on one network. The symptoms are ugly and slow to diagnose: address conflicts, a switch forwarding table that flaps as the same MAC appears behind two ports, half the client sessions landing on a machine with stale data, and — where both reach a shared database or object store — two writers with different views of the world committing into it. That last one outlives the outage.

There are more ways for the original to be alive than the scenario suggests. High availability may have restarted it on a surviving node. A colleague may have restored it in another terminal. It may have been on shared storage and never gone down. Or the host may be merely unreachable — no management network, no console — while its guests run on a data path that never failed. The rule covering all four is that you prove the original is not running before attaching the copy: query the address, look for its MAC on the switch, read the guest list on every surviving node, check the load balancer’s backend health. An unreachable host is not a dead host, and it is the case where restoring feels most obviously correct.

Elapsed time is the deliverable, not the restore count

Record wall-clock timestamps throughout, because the deliverable is a number: incident declared, inventory reconciled, capacity decision taken, first restore started, first service validated, last Tier 1 service validated, last guest validated. Those seven marks, against the recovery times the tier register commits to, are the only honest report of how the estate performs.

They almost always disappoint the first time, and the reason is structural. Where hypervisor loss is rehearsed at all, it is usually rehearsed by restoring one guest and multiplying by the guest count. That multiplication silently drops everything this lesson has been about: the inventory reconciliation, the capacity decision and the conversation it requires, the contention that makes concurrent streams slower than the single measured stream, the waves that must complete before the next begins, and the coordination across a busy team. Recovery time for thirty guests is not thirty times the recovery time for one; it is longer, by a factor the single-guest measurement cannot predict.

Be equally precise about where the clock stops. A guest that boots is not a restored service. The clock stops when the service answers on its own address, its dependants report healthy, and a value recorded before the incident — a row count, a checksum, a last-known transaction identifier — matches what the recovered system returns. Validation converts a restore into evidence, and it is the step most often cut when the elapsed figure is already embarrassing.

Then publish the figure, including the embarrassing one. The response to a recovery that overran its documented target is to change the architecture — capacity held ready, copies closer to the targets, fewer guests that must be restored rather than restarted — or to change the commitment. Continuing to publish the target while quietly knowing the measurement is how the gap survives to the next incident.

Production discipline

1. Reconcile the inventory against the catalogue before starting anything. Separate guests on healthy shared storage, which need a start rather than a restore, from guests on the lost host’s local storage, and report any guest that was never in a backup job as a loss immediately.
2. Take the capacity decision explicitly, and name who owns it. Memory is usually the binding constraint; the answers are provision more, run a subset or run degraded, and the subset order comes from the tier register rather than from whoever is loudest on the bridge call.
3. Restore in waves that follow the dependency graph, and keep restore and power-on as two separate acts. Time, name resolution, identity, certificates, secrets and databases precede the applications that need them, and a guest left powered off after its restore cannot start early.
4. Set the concurrency deliberately, from a rehearsal measurement. One repository, one link and one array do not become twelve because twelve jobs were started; cap the rate with a documented control such as the per-storage restore limit and --bwlimit override Proxmox VE documents, so recovery does not degrade what survived.
5. Prove the original guest is not running before attaching the copy, and stop the clock only on a validated service. An unreachable host is not a dead host, and a guest that boots has not proved anything until it answers on its address and matches a value recorded before the incident.

Cross-course references

• Proxmox VE for Production Operators — Part XIV (Disaster Recovery) is where this sequence becomes product-specific, covering the node removal, guest restore and cluster repair steps on a real host, so it is where the wave plan described here is turned into the commands an operator runs.
• Ceph & Distributed Storage for Production Sysadmins — Part CXII (OSD Host Loss) matters directly whenever the lost hypervisor was also a storage node, because the cluster’s own recovery traffic then competes for the same network and disks as your restores, and the concurrency decision in this lesson has to account for both.
• Observability for Production Sysadmins — Part CX (Observability During Major Incidents) covers keeping the measurement working while the estate is broken, which is what keeps the seven timestamps and the bottleneck readings this lesson depends on available during the recovery rather than reconstructed after it.

Quiz