Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

intermediatebdr-vm-restore~45 min

The restored machine boots, reports healthy and reaches nothing

Reported symptoms

  • ●A hypervisor host is lost at 03:20 and the order-entry VM rbdr-vm-orders is restored to the recovery cluster at the second site, reaching a login prompt 41 minutes later
  • ●The guest reports healthy in every way it is able to: systemd finishes booting, the order service is listening on 8443, the guest agent answers the platform, and the console session behaves normally
  • ●No client reaches the service, the load balancer marks the pool member down, and the guest cannot reach either the database or the directory it authenticates against
  • ●From inside the guest the interface is up and reports a carrier, but the machine cannot reach its own default gateway and its ARP table holds no entry for it
  • ●The disk image passed repository verification before the restore, and the guest mounted every filesystem it expected without complaint
  • ●The recovery call spends its first fifty minutes on the storage path, on the theory that the restore produced a damaged image, and a second restore is started and then abandoned
  • ●The quarterly restore test for this VM has recorded PASS for six consecutive quarters, the most recent one nineteen days before the incident

Evidence

  • · The exported production definition attaches net0 to bridge vmbr1 with VLAN tag 41, and the restored definition on the recovery host names the same bridge and the same tag, so no field visibly changed during the restore
  • · The MAC address on the restored interface is identical to the production one, which removes both the interface-rename and the missed-DHCP-reservation variants of this signature
  • · Inside the guest, the interface is up with a carrier and holds the same static 10.20.41.24/24 the machine has always had, with its default route via 10.20.41.1
  • · The guest runs no DHCP client, holds no lease file, and has never been issued an address by the IPAM system, so no address assignment took place at restore time to go wrong
  • · A helper VM placed on vmbr1 with tag 41 on the same recovery host exchanges ARP and ICMP with the restored guest, which establishes that frames move inside the host
  • · Neither the helper nor the restored guest reaches anything beyond the recovery host on that segment, and the recovery host has no address in 10.20.41.0/24 of its own
  • · The recovery host bridge named vmbr1 has a single uplink on a switch port that carries one untagged segment, while the production host bridge of the same name is a bond to a trunk carrying VLANs 40 to 49
  • · The quarterly restore-test procedure has three steps - restore the VM, power it on, confirm a login prompt on the hypervisor console - and records the result on the third; it names no client, no dependency and no transaction
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first put the machine in the wrong place; the second is why nobody knew that the wrong place was where the recovery would put it. **The immediate cause: the segment was not carried at the recovery target.** The restored definition asked for bridge `vmbr1`, VLAN tag 41, and the recovery host has a bridge called `vmbr1`, so the restore matched the name and reported nothing. The two bridges are not the same network. On the production host `vmbr1` is a bond to a switch trunk carrying VLANs 40 to 49; on the recovery host it is a single link to a port that carries one untagged segment and does not trunk VLAN 41. The guest was therefore attached correctly to a bridge that goes nowhere useful, which is why every reading available inside the guest was both accurate and worthless: the virtual link terminates at the bridge, so the carrier was real, the static address was the right address, and the frames leaving it were discarded above the host. **The control that should have caught it: the restore test ended at the login prompt.** The quarterly test restored this VM into this same recovery cluster six times and passed six times, because reaching a login prompt on the hypervisor console is the whole of what it asserted. That check crosses no network the clients use. It exercises firmware, bootloader, disk controller and filesystem, all of which were fine on the night, and it exercises nothing about where the interface lands. The defect it was blind to had been present in the recovery cluster the entire time the test was passing.

Remediation

Do not re-run the restore. The second restore reproduces the same definition against the same host and fails identically, and the forty minutes it costs are forty minutes of the outage. Read the attachment on both sides before changing anything, because the fix differs depending on which of the two faults you have. Compare the bridge named in the restored definition against what that bridge actually is on the recovery host - its uplink, and the VLANs that uplink carries. Then restore the segment rather than the setting. Either make the recovery host's uplink carry VLAN 41 and leave the definition alone, which is the change that makes every future restore of every VM on that segment land correctly, or move the interface to a bridge on the recovery host whose uplink already trunks it. Prefer the first: it repairs the recovery site, while the second repairs one VM and leaves the next restore to rediscover the fault. Once the frames leave the host, confirm the guest needed no change at all. Its address, route and MAC were correct throughout, and any edit made to them during the incident is now a second fault to unwind.

Verification

Verification originates where the callers originate. A check run from the hypervisor console, from the recovery host, or over a management interface traverses a path no client uses and can pass while the incident is still running. Establish, in order: that a machine on the client subnet exchanges traffic with the restored guest at layer two and layer three; that the guest resolves and reaches its database and its directory; that the certificate offered on the client-facing port is valid for the name clients use; and that a transaction a user would recognise completes end to end and is still present after the service restarts. Then verify the control rather than the machine. Re-run the quarterly restore test against its rewritten procedure and confirm it now fails on a recovery cluster with the uplink change reverted. A control that cannot fail on the defect it exists to catch has not been fixed.

Prevention

**Test the segment, not the bridge name.** Matching names are what let this restore report success. Record, per protected VM, the bridge, the VLAN tag and the uplink that carries it at both sites, and treat a name that resolves to a different segment as the defect it is. **End every restore test with a completed transaction.** A login prompt proves that firmware found a bootloader and the kernel found its root filesystem. It is the first stage of a validation, not the result of one, and a test that stops there is blind to everything the definition controls. **Run the validating traffic from the client subnet.** The hypervisor console is the one vantage point guaranteed not to exercise the fault. **Keep a layer-two probe in the recovery site.** A small machine you can attach to any bridge with any tag turns "is this attachment sound?" into a two-minute answer that does not depend on the restored guest cooperating. **Diff the recovery site against production on network attachment, not only on capacity.** DR readiness reviews count CPU, memory and storage. The attribute that decided this incident was which VLANs an uplink trunks, and no capacity review has a column for it.

Reported symptoms

A hypervisor host dies at 03:20. The order-entry VM rbdr-vm-orders is restored to the recovery cluster at the second site and reaches a login prompt at 04:01.

The guest then reports healthy in every way it can: the boot completes, the order service binds 8443, the guest agent answers the platform, and the interface is up with a carrier. Nothing reaches it. The load balancer marks the pool member down, and the guest reaches neither its database, nor its directory, nor its own default gateway, for which its ARP table holds no entry.

The call spends fifty minutes on storage, because a machine that answers on the console and reaches nothing feels like a damaged image. The quarterly restore test for this VM has recorded PASS six quarters running, into this same cluster.

Evidence provided

First, the definitions. Production export and restored configuration name the same bridge, vmbr1, and the same tag, 41, and the MAC is unchanged — which removes the interface rename and the missed DHCP reservation.

Read-only / Safewhat the definition claims the interface is attached to
$ grep -E '^net[0-9]+:' /etc/pve/qemu-server/8121.conf

Second, the guest’s address configuration: the static 10.20.41.24/24 it has always held, default route via 10.20.41.1, no DHCP client, no lease file, no IPAM record of this address.

Third, the layer-two probe, the test that does not depend on the guest: a helper VM on vmbr1 with tag 41 on the same recovery host, statically addressed in 10.20.41.0/24.

Read-only / Safefrom the helper, then from the recovery host
$ arping -c 3 -I eth0 10.20.41.24; ip -d link show vmbr1; bridge vlan show dev vmbr1

Helper and guest exchange ARP and ICMP; neither reaches anything beyond the recovery host on that segment. And the two bridges sharing the name are not the same thing: on the recovery host, a single uplink to a switch port carrying one untagged segment; on the production host, a bond to a trunk carrying VLANs 40 to 49.

Fourth, the control: a three-step procedure — restore, power on, confirm a login prompt on the hypervisor console — recording its result on the third. It names no client, no dependency, no transaction.

Work the evidence before reading on

  1. The helper on the same bridge and tag reaches the guest; nothing beyond the host does. Which attachment fault does that confirm, and which does it rule out?
  2. The guest holds a static address and runs no DHCP client. What family of explanations does that remove?
  3. The guest booted, mounted its filesystems and bound its port. What does that establish about the disk image, and what does it not?
  4. The restore test passed six times here. What did each pass actually assert?

Root cause

The segment was not carried at the recovery target

The definition asked for vmbr1, tag 41. The recovery host has a bridge called vmbr1, so the name matched, the restore reported nothing, and the interface was attached exactly as instructed — to a bridge whose uplink does not trunk 41.

Every reading inside the guest was accurate and useless. The virtual link terminates at the bridge, so the carrier was genuine and the address right for the segment the machine belongs on; the frames crossed the bridge and were discarded above the host.

The probe separates the two faults in this signature without the guest’s cooperation. A helper on the same bridge and tag that cannot reach the guest means the interface sits on a different bridge from the one configured. A helper that can reach it, while nothing beyond the host can, means the attachment is sound and the fault is above it — the tag not carried, or the uplink not trunking it. That is this incident.

The restore test asserted that the guest booted

Reaching a login prompt on the hypervisor console is the whole of what the quarterly test asserted, and it was true on all six occasions.

Resolution

Do not re-run the restore. Read the attachment on both sides first, since the repair differs by which fault the probe identified.

set -u
VMID=8121
BRIDGE=vmbr1
grep -E '^net[0-9]+:' "/etc/pve/qemu-server/${VMID}.conf"
ip -d link show "$BRIDGE"
bridge vlan show dev "$BRIDGE"

Then repair the segment rather than the setting. Making the recovery host’s uplink carry VLAN 41 fixes this VM and every future restore onto that segment; moving the interface to a bridge that already trunks 41 fixes this VM and leaves the next restore to rediscover the fault. Prefer the first; take the second only as a temporary measure inside the outage.

The guest needs no change. Its address, route and MAC were correct throughout, and any edit made to them during the incident is a second fault to unwind.

Verification

Verification originates where the callers originate, because the hypervisor console can report success while the incident continues.

set -u
VM_IP=10.20.41.24
VM_FQDN=rbdr-orders.example.internal
PROBE=rbdr-client-probe.example.internal
ssh "$PROBE" "ping -c 3 -W 2 $VM_IP"
ssh "$PROBE" "curl -fsS https://$VM_FQDN/healthz"
ssh "$PROBE" "curl -fsS -X POST https://$VM_FQDN/rbdr-selftest/order"

Establish in order: a machine on the client subnet exchanges traffic with the guest; the guest reaches its database and directory; the certificate offered on the client-facing port is valid for the name clients use; and a transaction a user would recognise completes and survives a restart.

Then verify the control. Re-run the rewritten quarterly test against a cluster with the uplink change reverted and confirm it now fails. A control that cannot fail on the defect it exists to catch has not been fixed.

Prevention

Test the segment, not the bridge name. Matching names are what let this restore report success. Record per VM the bridge, the tag and the uplink that carries it at both sites, and treat a name resolving to a different segment as a defect.

End every restore test with a completed transaction, issued from the client subnet. A login prompt is the first stage of a validation, not its result.

Keep a layer-two probe in the recovery site. Attaching it to any bridge and tag answers the wrong-bridge question without the guest’s cooperation.

Diff the recovery site against production on network attachment. Readiness reviews count CPU, memory and storage. Which VLANs an uplink trunks decided this incident, and no capacity review has a column for it.