high riskservice affecting~90 min

Restore a virtual machine

1 · Prerequisites

Confirm every item is in place before any state change.

The exported machine definition as it stood before the incident: name and unique identifier, MAC address of each interface, virtual CPU and memory, machine type, disk bus and controller model, firmware mode, the full disk list, bridge and VLAN tag, boot order. Without it, the restore is a reconstruction from memory.
Access to the backup catalogue for this machine, showing for every recovery point both when it was taken and which class of consistency it holds: application-consistent, filesystem-consistent, or crash-consistent.
A written incident timeline in UTC recording the earliest evidence of the fault and who observed it. The recovery point is selected against this timeline, not against the catalogue.
Authority to decide the fate of the original machine, and the means to power it off or fence it.
A recovery host or cluster with capacity for the guest, offering the firmware mode and disk controller the guest was installed against, and carrying the bridge and VLAN the guest expects — or a recorded decision about each difference.
A way to bring the guest up with its virtual interface disconnected, or attached to a bridge that reaches nothing, on the recovery platform.
The machine's address, DNS name and DHCP reservation, plus any licence, monitoring identity or firewall rule bound to its MAC address or its address.
A named business transaction that a caller would recognise, and somebody able to run it and confirm the result.
Somewhere to record the start time, the stage timings and the outcome. The elapsed time of this procedure is the recovery time this estate actually has.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Fix the cut-off before you open the backup catalogue. Take the earliest evidence of the fault from the incident timeline, subtract a margin for the gap between the fault starting and anyone seeing it, and write down one timestamp in UTC. Choosing while looking at the list invites "the newest one", which is the candidate most likely to contain the fault.
· Read the consistency class of every candidate, not only its timestamp. A crash-consistent copy is an image of the disks as they would be after a power cut; whether the guest comes back from it is a property of the applications inside it, not of the backup. An application-consistent copy required the guest to be quiesced while it was taken, and the catalogue records whether that quiescing actually succeeded or was skipped.
· Confirm the chosen recovery point predates the cut-off, and that at least one older eligible candidate exists behind it. A single eligible point means an abort has nowhere to go.
· Count the disks in the recovery point against the disks in the exported definition. A backup job scoped to the boot disk produces a recovery point that restores, boots, and is missing the volume the application actually writes to. The count is free to check now and expensive to discover after the guest is running.
· Establish whether the original machine is still running, and who can stop it. Every decision below turns on this answer, and it is the one most often assumed rather than checked.
· Compare the recovery host against the exported definition for firmware mode, disk bus and controller model, machine type, and virtual CPU and memory. These are read while the machine is stopped and cost nothing; discovering them after a failed boot costs the whole cycle.
· Confirm the bridge exists on the recovery host and leads where you think it leads, and that the VLAN tag is carried on that host's uplink. Nothing inside the guest can distinguish a working attachment from one that terminates in the wrong broadcast domain.
· Probe the intended address from a host already on the target segment, before anything is powered on. arping -D -c 3 -I "$IFACE" "$VM_IP" exits 0 when nothing answered. ABORT CRITERION: if it exits non-zero and the responder is a running production machine, stop here — do not connect the restored interface and do not power the original off to make room. Either change the restored machine's address or complete the decision to retire the original first.
· Note what is bound to the MAC address: DHCP reservation, licence, monitoring identity, firewall rule. A regenerated MAC breaks all of them at once and presents as several unrelated faults on a machine whose disk image is perfect.
· Record the start time in UTC, and start the stage clock.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Select the recovery point against the cut-off, and against its consistency class. The correct choice is the newest candidate that predates the cut-off AND holds a consistency class the applications in this guest can come back from. It is never "the latest available", and the two are rarely the same recovery point.
2Record the identity of the chosen recovery point before restoring anything: its identifier, its timestamp in UTC, and its consistency class. Every later question about what was lost is answered from those three values, and none of them are easy to reconstruct afterwards.
3Decide restore in place or restore beside, and write the reason down. Restoring beside builds a new machine from the recovery point and leaves the original untouched, which keeps the original as both evidence and fallback at the cost of capacity and of an identity collision you must then manage. Restoring in place is faster and discards the only copy of the current state. Choose beside whenever the cause is unknown, the guest may be compromised, or the original is still running.
4If you restore in place, copy the current disks first or record in writing that the decision is one-way. There is no third option, and the moment the restore starts writing there is no way back to the state you are about to overwrite.
5**If you restore beside, give the new machine a scoped name such as rbdr-app01-restore,** so that everything created by this procedure can be found and removed later without guessing which objects belong to it.
6Treat every identity attribute as a decision, not a default. Keeping the MAC address preserves the DHCP reservation and any licence bound to it, and makes a collision with the original possible. Regenerating it removes the collision and breaks the reservation, the licence and possibly the guest's interface names. You cannot have both; pick one deliberately and record which.
7Restore the definition, not only the disk image. Reproduce machine type, firmware mode, disk bus and controller model, virtual CPU and memory, and the full interface list from the exported definition. A guest given less memory than it had still starts, is accepted as recovered, and fails under load after the incident call has ended.
8Restore every disk the definition lists, from the same recovery point, and attach each on the bus it was on. A guest whose second volume is absent or was taken from a different point in time boots, and then either fails to mount the path or serves data that never coexisted.
9Map the interfaces explicitly, one at a time. For each virtual interface, record which bridge it lands on and which VLAN tag applies, and compare both against the original. Do not accept the platform's default bridge merely because it was first in the list.
10Decide what the address will be, rather than assuming it returns with the machine. A guest holding a static address brings it back inside the disk image. A guest that used DHCP returns to its old address only if its MAC survived and the reservation is still present. The two cases give the conflict check completely different meanings.
11Power on with the virtual interface disconnected, and treat that as the default. Two conditions make an isolated first boot mandatory, and neither can be ruled out by inspecting the disk image: the guest may be compromised, in which case a copy that reaches the network resumes whatever the original was doing before anyone has looked at it; and the original may still be running, in which case connecting produces two machines answering for one identity.
12While the guest is isolated, establish what it is, what it has, and when it thinks it is. From the console, confirm the hostname, the block devices and their mount points, the interface names and their configuration, the clock and its synchronisation state, and whether the applications started or are waiting on something they cannot reach. A restored guest can come back with a clock far behind, which breaks certificate validation and ticket-based authentication before it breaks anything obvious.
13Repeat the address conflict probe from the target segment, then reconnect one interface at a time. The first probe was taken before the restore began and may be minutes old. Reconnect only after the second probe is clean, and only the interface you intend to test.
14Run the named business transaction from where the callers are. Not from the hypervisor console, not from the recovery host, and not over a management interface — those paths test something the callers do not use.
15Record the timings, the decisions and the gap. Stage timings, the recovery point identifier and class, the identity decisions, and the work that existed between the recovery point and the incident. That last item is what the service owner has actually lost, and it is nobody else's job to write it down.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The chosen recovery point's identifier, UTC timestamp and consistency class are recorded in the incident note, and [ "$RP_EPOCH" -le "$CUTOFF" ] returns exit 0 — the point predates the cut-off derived from the incident timeline rather than merely being the newest offered.
✓A field-by-field comparison of the exported definition against the restored one — machine type, firmware mode, disk bus and controller model, virtual CPU and memory, disk count, interface count — either produces no differences, or every difference it prints is recorded with the reason it was accepted.
✓Every disk in the exported definition is attached to the restored machine and came from the same recovery point: inside the guest, lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS exits 0 and lists each expected device mounted at the path /etc/fstab names, with no entry left unmounted.
✓findmnt --verify --verbose inside the guest exits 0, proving no fstab entry refers to a device or UUID that the restore did not bring back.
✓Each interface's bridge and VLAN tag are recorded next to the original values, and ip -brief link show inside the guest exits 0 reporting the expected device names in state UP.
✓Before power-on, arping -D -c 3 -I "$IFACE" "$VM_IP" from a host on the target segment exited 0 — nothing answered for the address. After power-on the same command exits non-zero and ip neigh show "$VM_IP" reports exactly one MAC address, the restored machine's.
✓hostnamectl --static returns the expected hostname with exit 0, and timedatectl status reports System clock synchronized: yes before any certificate or ticket-based authentication is attempted.
✓openssl s_client -connect "$VM_IP:443" -servername "$VM_FQDN" </dev/null completes the handshake and exits 0 — a handshake the server cannot complete without the matching private key — and openssl x509 -noout -checkend 0 on the certificate it offered exits 0, with the subject and SAN covering the name clients use.
✓curl -fsS --resolve "$VM_FQDN:443:$VM_IP" "https://$VM_FQDN/rbdr-transaction" exits 0 from a client on the network the callers use and prints the expected record identifier, and the identical request repeated after the service is restarted prints it again. A result that does not survive the restart was served from memory.
✓Each dependency answered a real request rather than being assumed to work: getent hosts "$VM_FQDN" exits 0 with the intended address, and the directory, database, broker or licence server the service needs accepted the restored machine.
✓The elapsed time from the start of this procedure to the completed transaction is recorded and compared against the recovery time the service owner expects, and reported whether or not it met it.
✓The work that existed between the recovery point and the incident is written down and communicated to the service owner as a loss, not discovered by a user later.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the restore was beside the original, rollback is to disconnect the restored machine, power it off, and delete it and its disks once the recovery point identifier has been recorded. Scoping every created object to the rbdr- prefix is what makes this a clean removal rather than a hunt.
↶If the restore was in place, there is no rollback to the previous state unless a copy of the pre-restore disks was taken first. Take that copy before starting, or accept in writing that the decision is one-way.
↶Undo the reflexes: alert silences, monitor threshold edits, disabled health checks, paused backup jobs. A silence created during the incident and left in place hides the next fault on the same machine.
↶Revert DNS records, DHCP reservations, load-balancer members and firewall rules that were pointed at the restored machine during validation. A record still pointing at a machine you then delete is a second outage with a new cause.
↶If an address conflict occurred after reconnection, disconnect the restored machine first — never the production one — and then clear the neighbour caches on the affected segment so that clients stop using the wrong hardware address.
↶If the guest was booted as part of a compromise assessment, do not roll back by deleting it until whoever owns the investigation has taken what they need from it.
↶Record what was rolled back, when, and why. A restored machine removed without a note reappears in the next incident as an unexplained gap in the timeline.

6 · Escalation

When the runbook isn't enough, contact:

· The intended address is already answering and the owner cannot be identified: escalate to the network owner before power-on. Do not proceed on the assumption that the reply is stale.
· The original machine cannot be powered off or fenced and the restore needs its identity: escalate immediately. Two machines claiming one identity is not recoverable by continuing.
· No recovery point exists that predates the cut-off: escalate to the service owner. The choice is now between a recovery point that may contain the fault and a longer loss window, and that is a business decision rather than a responder's.
· The recovery point contains fewer disks than the definition lists: escalate to the backup platform owner before booting the guest. A guest started with a missing volume writes into the mount point underneath it, and those writes have to be found and unpicked afterwards.
· The only eligible recovery point is crash-consistent and the application does not come back from it: escalate to the application owner. The next step is application recovery inside the guest, which is a different procedure with a different owner.
· The recovery host cannot offer the firmware mode, machine type or disk controller the guest was installed against: escalate to the platform owner. Repeating the restore reproduces the same definition and fails identically.
· The guest may be compromised: escalate to whoever owns the security incident before any interface is connected, and preserve the isolated machine rather than deleting it.
· Elapsed time is exceeding what the service owner expects: escalate with the measured remaining work so that they can decide about partial service instead of waiting.

Restoring a machine is four decisions and one copy operation. The copy is the part everybody plans for. The decisions are which recovery point, in place or beside, isolated or connected, and what counts as recovered — and each of those is where a restore goes wrong in a way the disk image cannot explain.

No hypervisor was available to this course, so nothing below quotes hypervisor output. The sequence, the attribute list and the failure modes come from the vendor documentation cited above; the commands shown are the platform-neutral ones a responder runs around the restore, and they are given without captured output rather than with invented output. The one capture quoted here was taken one layer down, on a filesystem, and is labelled as such.

Recovery point selection

The order matters more than it looks. Derive one timestamp from the incident timeline first, then filter the catalogue against it. Reversing the order turns recovery point selection into a glance at a list, and the newest entry always wins a glance.

# The earliest evidence of the fault, plus a margin for the gap between the
# fault starting and somebody noticing it.
FIRST_EVIDENCE='2026-08-27 21:40:00 UTC'
MARGIN_MINUTES=90

START_EPOCH=$(date -u +%s)
CUTOFF=$(date -u -d "$FIRST_EVIDENCE - $MARGIN_MINUTES minutes" +%s)
date -u -d "@$CUTOFF" '+cut-off: %Y-%m-%dT%H:%M:%SZ'

Now read the catalogue with two columns, not one. A recovery point has a time AND a consistency class, and the second is what decides whether the guest comes back from it.

# One line per candidate: epoch seconds, consistency class, identifier.
CANDIDATES=/tmp/rbdr-recovery-points.tsv

while IFS=$'\t' read -r taken class id; do
  when=$(date -u -d "@$taken" '+%Y-%m-%dT%H:%M:%SZ')
  if [ "$taken" -le "$CUTOFF" ]; then
    printf 'ELIGIBLE  %s  %-22s %s\n' "$when" "$class" "$id"
  else
    printf 'TOO LATE  %s  %-22s %s\n' "$when" "$class" "$id"
  fi
done < "$CANDIDATES"

Application-consistent means the guest was quiesced when the copy was taken: the applications flushed and the filesystems were frozen, usually through an in-guest agent, and the platform records whether that step succeeded or was skipped. Crash-consistent means the disks were captured as they would be after a power cut. A crash-consistent copy of a database machine boots and runs the application’s own crash recovery; whether that recovery succeeds is a property of the application, not of the backup that produced the copy.

Decision point: in place, or beside

	Restore in place	Restore beside
Original machine	Overwritten	Kept, powered off
Fallback if the recovery point is wrong	None, unless the current disks were copied first	The original
Identity collision	Not possible	Must be managed
Capacity needed	The machine’s own	A second machine’s worth
Right when	The cause is known, the original is stopped, and its current state is worthless	The cause is unknown, the guest may be compromised, or the original is still running

The default is beside. It is reversible, it preserves the evidence, and its costs are capacity and one identity decision — both of which you can pay while the machine is stopped.

Reproduce the definition, and every disk in it

A restored disk image is a restored disk. The machine is that image plus a definition, and the definition is a separate artefact that a restore reproduces exactly, approximately, or not at all.

VM=rbdr-app01
DEF=/tmp/rbdr-${VM}-definition.xml

virsh dumpxml "$VM" > "$DEF"

# The attributes that decide whether it boots, and whether anything reaches it.
grep -E 'mac address|bridge|vlan|loader|nvram|target dev|bus=|machine=' "$DEF"

# The disk count the recovery point has to match.
grep -c "<disk device='disk'" "$DEF"

Restoring to a different host or cluster is where these diverge quietly. The unique identifier the platform assigns may be regenerated, producing a machine that is — from the management layer down — a different machine with the same contents, which matters to anything holding a per-machine licence or an inventory record. The MAC address may be regenerated with it, which misses the DHCP reservation, breaks a licence bound to the interface, and, under naming policies that derive a device name from the hardware address, can rename the interface inside a Linux guest so that per-interface configuration is orphaned. And the bridge names on the recovery host may match the originals while leading somewhere entirely different.

The failure that costs the most is quieter still: a backup job scoped to the boot disk. The guest restores, boots, and is missing the volume the application writes to. This course had no hypervisor to demonstrate that on, but it did measure the same shape one layer down — a data directory restored without its second volume:

Configuration changeone volume restored, the other absent— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2). Application consistency and point-in-time recovery capture, Part B: the data directory present, its separate write-ahead-log volume emptied, modelling a snapshot that covered one volume of two.

$ pg_ctl -D /work/nowal start

  waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
>>> exit code: 1

2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/archive_status"
2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/summaries"
2026-08-28 13:34:37.879 UTC [132] LOG:  invalid checkpoint record
2026-08-28 13:34:37.879 UTC [132] PANIC:  could not locate a valid checkpoint record at 0/2F20158
2026-08-28 13:34:37.941 UTC [126] LOG:  startup process (PID 132) was terminated by signal 6: Aborted
2026-08-28 13:34:37.941 UTC [126] LOG:  terminating any other active server processes
2026-08-28 13:34:37.942 UTC [126] LOG:  shutting down due to startup process failure
2026-08-28 13:34:37.943 UTC [126] LOG:  database system is shut down

That one is the lucky version, because it exits 1 and says why. A guest whose missing volume carries application data rather than a redo log usually starts cleanly and serves an empty or stale directory instead, and the writes it then accepts land underneath the mount point. Count the disks before you power on.

Decision point: isolated, or connected

The conflict check belongs before power-on, from a host already on the segment the restored machine will join. arping -D is the duplicate address detection mode described on the arping(8) manual page cited above: it exits 0 when no reply is received, and non-zero when something answered.

VM_IP=192.0.2.41
IFACE=eth0

arping -D -c 3 -I "$IFACE" "$VM_IP"
echo "arping exit status: $?"

ip neigh show "$VM_IP"
ping -c 2 -W 2 "$VM_IP" || echo 'no ICMP reply'

Blast radius

Action	Reversible?	What it costs if wrong
Restoring beside, powered off	Yes — delete the machine	Capacity and time
Restoring in place without copying the current disks	No	The only copy of the current state
Booting with a disk from the definition missing	Sometimes	Writes land under an empty mount point and have to be unpicked
First boot with the interface connected	Sometimes	A compromised guest back on the network, or two machines on one identity
Reusing the MAC while the original runs	No	Clients split between two machines; neighbour caches to clear on every segment
Regenerating the MAC	Yes, but noisy	A missed DHCP reservation, a broken licence, renamed interfaces inside the guest
Declaring recovery at the login prompt	No	An outage that is reported as resolved and reopens hours later

When the guest boots and the service does not

The disk image is almost never the fault. Read the signature and go to the attribute it implicates, rather than restoring again and getting the same machine.

Signature	Most likely cause	Go and check
Firmware drops to a shell or a boot menu; no bootloader is reached	Firmware mode differs from the one the guest was installed against, or the boot disk landed on a different bus	Firmware mode and disk bus in the restored definition against the exported one
Kernel starts, then waits for a root device or drops to an emergency shell	The disk controller model changed and the guest’s initramfs carries no driver for the device it was given	Controller model in both definitions; whether the initramfs contains that driver
Boots, but an `/etc/fstab` entry fails to mount	That disk was not in the recovery point, or was restored but never attached	`findmnt --verify`; disk count in the definition against the restored machine
Interface up, carrier present, nothing reachable	Wrong bridge, or a VLAN tag not carried on the recovery host’s uplink	Layer-two test from a trusted machine on the same bridge
The interface exists under an unexpected name and its configuration is not applied	The MAC was regenerated and the naming policy derives the device name from it	MAC address against the exported definition
Reachable by address, but no address was assigned by DHCP	The MAC changed, so the reservation no longer matches	MAC address; the reservation on the DHCP server
Reachable from some segments and not others, intermittently	Two machines answering for one address	Whether the original is still running; neighbour tables on the affected segments
TLS handshakes fail or authentication tickets are rejected while the service is up	The restored guest’s clock is behind	`timedatectl status` inside the guest
The service runs and the data is older than the recovery point implies	A second disk was restored from a different recovery point	The recovery point identifier recorded for each disk

Abort criteria

Stop and escalate rather than continuing when any of these is true: no recovery point predates the cut-off; the only eligible recovery point is the sole one, so an abort has nowhere to go; the recovery point holds fewer disks than the definition lists; the intended address answers from the target segment and the responder is a running production machine; the original cannot be stopped or fenced and the restore needs its identity; the guest may be compromised and its interface is not yet isolated; or an in-place restore is about to begin and no copy of the current disks exists. Each of these makes the next step actively harmful rather than merely slow, because from the moment the guest boots connected it starts answering for an identity and accepting writes.

Business validation

Reaching a login prompt establishes that the firmware found a bootloader and the kernel found its root filesystem. Everything the callers care about is still unproven.

VM_FQDN=rbdr-app01.example.internal
VM_IP=192.0.2.41
SSH_TARGET="rbdr-ops@$VM_IP"

# Identity, storage and clock, from inside the guest.
ssh "$SSH_TARGET" 'hostnamectl --static; lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS'
ssh "$SSH_TARGET" 'findmnt --verify --verbose; timedatectl status'

# The certificate the clients will actually be offered. Completing the
# handshake is what proves the private key is present.
openssl s_client -connect "$VM_IP:443" -servername "$VM_FQDN" </dev/null 2>/dev/null |
  openssl x509 -noout -subject -dates -checkend 0

# A transaction, end to end, through the path a caller uses.
curl -fsS --resolve "$VM_FQDN:443:$VM_IP" "https://$VM_FQDN/rbdr-transaction"
TXN_RESULT=$?
echo "transaction exit status: $TXN_RESULT"

# Restart, then read the same record back. A result that does not survive a
# restart was served from memory and the storage layer is still unproven.
ssh "$SSH_TARGET" 'sudo systemctl restart rbdr-app01.service'
curl -fsS --resolve "$VM_FQDN:443:$VM_IP" "https://$VM_FQDN/rbdr-transaction"

What to record

RECORD=/tmp/rbdr-restore-record.txt

# Each of these is a decision made above, not a default. The shell refuses to
# write the record until the ones that cannot be guessed have been supplied.
: "${RP_ID:?record the recovery point identifier}"
: "${RP_TIME_UTC:?record the recovery point timestamp in UTC}"
: "${RP_CLASS:?record the consistency class of the recovery point}"
: "${START_EPOCH:?record the start time before the first change}"
RESTORE_MODE=${RESTORE_MODE:-beside}
MAC_DECISION=${MAC_DECISION:-kept}
FIRST_BOOT_MODE=${FIRST_BOOT_MODE:-isolated}

{
  printf 'recovery point id   : %s\n' "$RP_ID"
  printf 'recovery point time : %s\n' "$RP_TIME_UTC"
  printf 'consistency class   : %s\n' "$RP_CLASS"
  printf 'cut-off used        : %s\n' "$(date -u -d "@$CUTOFF" '+%Y-%m-%dT%H:%M:%SZ')"
  printf 'in place or beside  : %s\n' "$RESTORE_MODE"
  printf 'mac decision        : %s\n' "$MAC_DECISION"
  printf 'first boot          : %s\n' "$FIRST_BOOT_MODE"
  printf 'transaction status  : %s\n' "$TXN_RESULT"
  printf 'elapsed seconds     : %s\n' "$(( $(date -u +%s) - START_EPOCH ))"
} | tee "$RECORD"

The last two lines are the ones that change future decisions. The elapsed time is the recovery time this estate has, as distinct from the one in the recovery plan. And the interval between the recovery point and the incident is the work the service owner lost — a number they are entitled to hear from you rather than from a user.