Restoring a VM: the parts that are not the disk image

Cataloguing the state that lives outside any single guest ended with a list of things the hypervisor holds that no VM backup contains. Narrow the frame to one machine and the same shape appears a level down. Restore a disk image and you have a disk; what you do not yet have is a machine, because a machine is a disk image plus a definition, and the definition is a separate object that the restore reproduces exactly, reproduces approximately, or does not reproduce at all.

No hypervisor was available to this course, so nothing below quotes hypervisor output: the attributes and the failure modes are taken from the documentation cited at the head of this lesson, and the single transcript that appears is a container capture used as an analogy for the principle, not evidence about VMs.

The definition is the other half of the machine

Every platform stores the definition somewhere different — a configuration file per VM, a database row, a domain XML document — but the contents converge, because they describe the same emulated hardware. It is worth walking the list explicitly, because a restore that gets nine of these right and one wrong fails in a way that looks nothing like a backup problem.

Identity. The name, the unique identifier the platform assigns, and the MAC address of each virtual interface. The name matters to anything that addresses the VM by name, including your own runbooks. The unique identifier matters to the platform: restoring under a new one produces a machine that is, from the management layer down, a different VM with the same contents. The MAC address matters far more often than people expect, because DHCP reservations key on it, some licence managers bind to it, and a guest whose interface naming rules were written against a specific MAC will present the restored interface under a different name.

Sizing. Virtual CPU count and topology, memory, and any reservation or share the host was honouring. A guest usually starts on less than it had, which is precisely the danger: it starts, it is accepted as recovered, and it fails under load hours later when the caller has stopped watching.

Virtual hardware. The machine type or chipset the platform emulates and the model of each device attached to it. Paravirtualised devices need a driver in the guest; emulated ones are chosen because they need no driver at all. Moving a guest between the two changes what the guest must already contain in order to use the device, and changes the device the guest sees even when the underlying bytes are identical.

Disk attachment. The bus each disk is presented on, the controller model behind that bus, and the order the disks appear in. This is the attribute most likely to stop a boot outright, and it is discussed in full below.

Network attachment. The bridge or virtual switch each interface connects to, and the VLAN tag applied to it. Nothing inside the guest can detect that this is wrong. The interface comes up, the driver reports a link, and the frames go somewhere other than where they used to go.

Boot. The boot order, and the firmware mode — legacy BIOS or UEFI. A guest installed under one and started under the other does not fall back gracefully. Where the platform presents UEFI, the firmware variable store is itself a small per-VM volume, distinct from the data disks, and losing it loses the recorded boot entries.

Guest integration. Whether the guest agent is installed inside the guest and enabled on the VM. The previous lessons in this part showed what the agent buys during a capture; on the restore side it is what lets the platform report the guest address, run a clean shutdown, and confirm the guest is actually up rather than merely powered on.

Host-level placement. High-availability group membership, affinity and anti-affinity rules, start order and start delay. These live above the VM and are frequently not carried in a VM-level backup at all. A restored pair of cluster members that used to be pinned apart, and now both land on one host, has lost the property the pair existed to provide.

The two failure signatures that dominate

Almost every unhappy VM restore that is not a storage problem lands in one of two buckets, and each points at a specific part of the list above.

The first is the VM boots and has no network. The console works, the guest logs in, the interface exists and reports a carrier, and nothing reaches it or is reachable from it. This is a network attachment problem in almost every instance: the interface is attached to a bridge that exists on the recovery host but leads somewhere different, or the VLAN tag was not carried across, or the tag was carried but the recovery host’s uplink does not trunk that VLAN. From inside the guest the evidence is systematically misleading. The link is up, because the link is virtual and terminates at the bridge. Address configuration either fails silently, because the DHCP server for that segment never sees the request, or succeeds against the wrong segment and hands out an address from the wrong subnet. A static configuration comes up cleanly and reaches nothing.

The MAC address belongs to this signature too. A new MAC means a DHCP reservation misses, so a machine that was always at a known address arrives somewhere else; and on a Linux guest it can change the interface name, at which point per-interface configuration that names the old device is orphaned and the machine boots with an unconfigured NIC.

The two causes inside this signature separate without touching the guest. Put a machine you already trust on the bridge the restored VM claims to be attached to, give it a static address in the expected subnet, and see whether the two can exchange traffic at layer two. If they can, the attachment is sound and the fault is above it — the tag that was not carried, or an uplink that does not trunk it — because an attachment that works locally and reaches nothing beyond the host is what an untrunked VLAN looks like. If they cannot, the interface is on a different bridge from the one you configured. Neither test depends on the guest cooperating, which matters because reading DHCP server logs produces nothing when the request never reached the segment the server is on.

The second is the VM does not boot at all. The firmware screen appears and the guest never takes over, or the bootloader starts and then cannot find its root filesystem. Two attributes cause the overwhelming majority of these. A disk controller or bus change means the guest is being offered its disk through a device it may have no driver for in its early boot image, and through a different kernel device path once it does. A firmware mode mismatch means the firmware is looking for a bootloader in a place the installed guest never put one. Both produce a machine that looks broken and a disk image that is perfectly intact, which is why the instinct to re-run the restore is so strong and so useless: the second restore reproduces the same definition and fails identically.

These two also separate on the console, and the discriminator is how far the machine gets before it stops. A firmware mode mismatch stops before any code belonging to the guest runs: the firmware works through its boot entries, finds nothing it can load, and drops into its own shell or boot menu with no bootloader on screen. A controller or bus mismatch gets visibly further, because the bootloader was found and did run — a menu appears, a kernel loads, and the failure surfaces afterwards as a kernel that enumerated no disk and cannot mount the root device. Where the sequence stopped names the attribute to compare against the exported definition first.

The container capture makes the principle concrete

No hypervisor transcript exists for this course, but the same relationship was measured one abstraction away, on containers, where the parts separate cleanly. A container’s persistent data lived in a named volume; the container itself was described by an image plus a set of runtime settings. The volume was archived, the container and the volume were destroyed, and the archive was unpacked into a fresh volume on clean infrastructure.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

The data recovery is unambiguous: the md5 sums match, so every byte the volume held came back. And a file the workload had written outside the volume did not, because nothing was protecting it. Map the analogy onto a VM and the volume is the disk image, while the container layer and the runtime settings — the network it joined, the ports published, the environment handed to it — are the machine definition. A VM restore has the same shape, with one difference that makes it worse rather than better: a container that is missing part of its definition usually fails immediately and loudly, while a VM that is missing part of its definition often boots.

Validation that ends in a completed transaction

A restored VM that reaches a login prompt has proved that the firmware found a bootloader and the kernel found its root filesystem. That is the beginning of the validation, not the end of it, and every stage after it tests something the boot could not.

The service answers, on the network its clients are actually on. A check run from the hypervisor console, or from the recovery host, or over a management interface, tests a path the clients do not use. The test that means something originates where the callers originate and crosses the same segments.

Its dependencies resolve and accept it. DNS returns the records the service expects, the database or directory it authenticates against accepts the restored credentials, the message broker is reachable, and the clock is close enough to correct that anything time-sensitive works. A restored VM can come back with a clock far behind, which breaks certificate validation and ticket-based authentication before it breaks anything obvious.

Its certificate is valid. Not merely present: valid for the name clients use, inside its validity window, with the chain the clients need and with the private key restored alongside it. A certificate and key pair sitting in a restored filesystem proves nothing until something completes a handshake with them.

A real transaction completes, and is still there afterwards. Write something, read it back through the same path a user would, and confirm it survives a restart of the service. This is what separates a machine that is running from a service that is working, and it is the only stage that exercises storage, application, dependency and identity together.

VM_FQDN=app01.example.internal
VM_IP=$(getent hosts "$VM_FQDN" | awk '{ print $1 }')

# reachability from where the callers live, not from the hypervisor console
ping -c 1 -W 2 "$VM_IP"

# dependencies and clock, checked inside the guest
ssh "$VM_FQDN" 'systemctl is-system-running --wait; timedatectl status'

# the certificate the clients will actually be offered
openssl s_client -connect "$VM_IP:443" -servername "$VM_FQDN" </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates

# a transaction, end to end, through the client-facing path
curl -fsS "https://$VM_FQDN/healthz"

Record what each stage returned. A restore that stopped at “it booted” and a restore that ended with a completed transaction are described identically in most incident timelines, and only one of them established anything.

Production discipline

1. Export every VM definition on the same schedule as the VM data, and store it where the platform’s own loss cannot take it. The definition is a small text artefact; there is no capacity argument for not having it, and no way to reconstruct the values nobody wrote down.
2. Compare identity, disk bus, firmware mode and network attachment before the first power-on, not after the first failure. These four produce the two dominant failure signatures, and all four are cheap to read off the exported definition while the VM is still stopped.
3. Treat “boots but unreachable” as a network attachment question and “does not boot” as a firmware or controller question. Both look like storage failures from inside the incident and neither is; re-running the restore reproduces the same definition and fails the same way.
4. Carry the MAC address deliberately. A DHCP reservation, an interface name derived from it, and any licence bound to it all break together and present as three unrelated problems on a machine whose disk is perfect.
5. End every restore validation with a transaction that a user would recognise. The container capture is the reminder of why: the data came back byte-identical, md5 for md5, and the workload was still missing something it needed, because the missing thing was never in the part that was restored.

Cross-course references

• Proxmox VE for Production Operators — Part IX (Virtual Machines) is where the definition described here is actually built and edited: machine type, disk bus and controller, firmware mode, network device and bridge. Read it as the concrete form of the checklist above, since it names the settings a restore has to be compared against on that platform.
• VyOS for Production Network Engineers — Part VIII (VLANs) covers the segmentation that decides whether a restored interface lands in the broadcast domain it used to occupy, which is the half of the “boots but no network” signature that lives outside the hypervisor and cannot be diagnosed from inside the guest.
• Linux for Production Sysadmins — Part IX (Boot Process) explains the firmware, bootloader and early userspace handover that this lesson traces to explain why a controller or firmware change stops an intact image, and it is where the repair techniques for a guest that cannot find its root device are covered in detail.

Quiz