Backing up a virtualisation estate: what else is state

Guest quiescing was a question about one machine — what the guest was doing when the capture fired, and whether anything inside it was asked to participate. Answer that correctly for every VM you own and you still do not have a recoverable estate. A virtualisation platform holds state that is inside no guest, that no VM backup job has captured, and that has to exist before a single restored guest can be placed, started or reached. This lesson enumerates that state, what each omission costs, and how to protect it.

Every guest recovered, and nothing runs

Take the failure this part exists for. The site is gone, the hardware is new, the hypervisor has been installed from media, and every VM backup is present and verified. Restore all of them. What lands on disk is a set of guest disk images, each with the guest definition its backup carried alongside it: so many virtual CPUs, so much memory, a disk on storage fast-nvme, an interface on bridge vmbr1 with VLAN tag 40, membership of a resource pool called retail-prod.

Read that definition again, because almost every field in it is a reference. fast-nvme is not a disk; it is the name of a storage definition telling the platform which pool, export or LUN to use, with which options, presenting which credential. vmbr1 is not a network; it is the name of a bridge that has to exist on this host, attached to the right physical interface, carrying the right VLANs. retail-prod is not a permission model but a label one attaches rules to. The VM backup preserved the references perfectly and preserved none of the things they refer to, because none of those things live inside the guest.

That is the whole lesson in one observation: backing up every VM is not backing up the estate. The guests are the part everyone counts, because they are the part with a backup job and a row on a dashboard. The platform underneath them is configuration somebody typed once, possibly years ago, possibly in a web interface that left no artefact anywhere — and it is what a recovery needs first, before the guests are addressable.

The register of state that never appears in a guest

The enumeration below is a register rather than a checklist of commands, because the items recur across platforms while the commands do not.

Three rows are routinely assumed to be covered by something else. Storage credentials are held by the platform, not the guest, so a definition without its credential is a name pointing at a door that will not open. API tokens are the credential class most likely to have been created ad hoc and least likely to have been written down anywhere. And backup job definitions are the recursive case: the configuration of the thing that protects the estate is itself estate state, and cannot be protected by the job it defines. The last row has no technical home at all; it gets its own section below.

What each omission costs, in the order it bites

At recovery time the register is not flat: the items fall into bands, and the bands impose a sequence.

The platform has to exist before anything else is meaningful. Cluster membership and the quorum arrangement come first, because a platform that has not formed a cluster cannot present cluster-wide storage, cannot evaluate HA rules and in many designs cannot accept a configuration change at all. Rebuilding this from a document saying “there were three nodes” is not the same as restoring it: the addresses, identifiers and voting weights are precise, and getting them wrong yields a cluster that forms and then behaves oddly.

A guest has to be placeable before it can be started. The restored definition says the disk lives on fast-nvme; the platform is asked to resolve that name and has no entry for it. Recreating the entry mid-incident requires the backend, the export path or pool name, the options and the credential — four facts the guest does not carry, all of which somebody knew on the day they configured it.

A guest that starts still has to be reachable. This band produces the most confusing incidents, because everything looks successful. A restored VM with no matching network definition does not connect: the guest boots, configures its interface exactly as before, and the packets go nowhere, because the bridge it expected does not exist on this host or does not carry the VLAN named. Where a software-defined layer is involved the same failure has more places to hide — a zone, a subnet, a controller, none of them guest content.

People and machines have to be able to operate what has been recovered. The identity, permission and token rows rarely stop a recovery outright; they degrade it into a recovery performed by one person with one credential, which is the condition under which recoveries go wrong. The management TLS certificate belongs here too: automation that verifies the endpoint refuses the rebuilt platform until a trusted certificate is in place, so issuance lands on the critical path at the worst moment.

Protection and safety have to resume. An estate running without its backup jobs accrues a second outage silently, and one running with half its fencing configuration is worse than one with none, because partial fencing produces the case where two nodes each believe they may run the same guest.

The method: locate it, schedule it apart, put it where the platform is not

The product-specific commands for exporting a platform’s configuration belong with the product, and this course does not restate them; the Proxmox specifics are cited below. What belongs here is the method: three steps that survive a change of hypervisor.

Step one: identify where the platform stores its own configuration, from the platform’s documentation rather than by inspecting the filesystem. Inspecting the filesystem finds the view; the documentation names the authority, gives the supported way to obtain a consistent copy, and says what is deliberately excluded from it. Where it distinguishes node-local from cluster-wide configuration, the answer has two parts and both need capturing. The written answer is itself a recovery artefact.

Step two: back that configuration up on a schedule independent of the VM backups. Independent in cadence, because configuration changes rarely but changes at precisely the moments that matter — a maintenance window, a new storage backend, a new VLAN, a node replacement — and a daily copy at 02:00 can miss an entire change window. Independent in mechanism, because the backup job definition is one of the objects being protected and cannot sensibly be the thing that protects it. Independent in retention, because needing the configuration as it was three changes ago is a different question from needing yesterday’s data. Exports are small, which makes both affordable in a way neither would be for guest data.

Step three: store the copy somewhere reachable without the platform. This is the circular dependency above, stated as a rule. The test is a sentence you should be able to say out loud: with every node powered off and the shared storage unavailable, here is who fetches the configuration and from where. If the answer needs the management interface, the cluster filesystem, or a share defined inside the configuration itself, the copy fails.

set -euo pipefail

# The three things the export has to name.
CONFIG_PATH=/etc/platform             # the location the documentation gives
STAGING=/var/tmp/platform-config      # a plain directory, outside the platform
OFFSITE=backup@offsite.example.net:/srv/estate-config

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
ARCHIVE="$STAGING/config-$STAMP.tar.gz"

mkdir -p "$STAGING"
tar -czf "$ARCHIVE" -C "$CONFIG_PATH" .
sha256sum "$ARCHIVE" >"$ARCHIVE.sha256"
scp "$ARCHIVE" "$ARCHIVE.sha256" "$OFFSITE/"

The shape matters, not the paths: produce a self-describing artefact, checksum it, and move it off the estate by a route that does not traverse the estate. An export a plain file restore can read, on a machine that never joined the cluster, beats an elegant mechanism that needs the platform healthy.

vm-142 is not a service name

That last row has no technical home, which is why it is missing almost everywhere. A platform knows a guest by an identifier and a name field. It does not know that vm-142 is the payments API, that the payments API cannot serve traffic until the identity provider on vm-107 is up, or that reporting can wait until Monday. That knowledge is the mapping between VM identifiers and business services, and it is state in exactly the sense used throughout: needed at recovery, not inside any guest, captured by no backup job.

Its absence does not stop a recovery, which is what makes it dangerous. It removes the recovery’s order and its stopping condition: you restore in whatever order the console lists things, you cannot tell the business which services are back, and you cannot tell whether the guest that took forty minutes was the one that mattered. The map is what turns a technical restore into a statement about the business, and that statement is what the incident is waiting for.

It also has the unhappy distinction of being the piece most likely to be stored on the estate it describes: a wiki page, a spreadsheet on a file server, a diagram in the ticketing system. Each of those is a guest, and each is down in the incident where the map is needed. Keep it with the configuration export, under the same rule.

Production discipline

1. Write down where this platform keeps its own configuration, citing the documentation you took it from. The filesystem shows a view; the documentation names the authority and the supported way to extract a consistent copy.
2. Give that configuration its own job, cadence and retention, separate from the VM backups. It changes in maintenance windows rather than overnight, and the VM backup job definition is one of the objects to be protected.
3. Place the copy where it can be read with every node powered off. Say aloud who fetches it and from where; if the answer needs the management interface, the cluster filesystem, or a share defined inside the copy, the destination fails.
4. Restore platform configuration before the first guest, and record that order in the plan. Cluster membership, then storage definitions and their credentials, then networks, then identity, permissions and tokens, then HA, fencing and the backup jobs.
5. Own the map from VM identifier to business service, and keep it off the estate it describes. Without it a recovery has no order and no stopping condition, and the wiki page holding it is usually a guest that is down.

Cross-course references

• Proxmox VE for Production Operators — Part XI (Clustering) covers the cluster filesystem and membership configuration treated abstractly here, and Part XIV (Disaster Recovery) gives the product commands for exporting and restoring it. This course deliberately does not restate those specifics: the register above is the platform-independent question, and the Proxmox material is where it becomes a command on a named host.
• VyOS for Production Network Engineers — Part LV (Backup, Restore, Disaster Recovery) applies this same method to a routing platform, which matters here because the VLANs a restored guest attaches to usually terminate on a router whose configuration is a separate estate artefact needing its own copy, schedule and off-box destination.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) is where the storage credentials and API tokens in the register belong so that they survive the platform, and Part IX (Certificate Lifecycle and Revocation) covers the management certificate whose reissue would otherwise land on the critical path of the recovery above.

Quiz