Skip to main content
RunBook Academy

← All labs in VyOS

Lab · advanced · ~150 min

Lab: Backup and Restore a VyOS Router

B · Nested virtualisationC · Simulation

Objectives

  • Build a two-chassis lab in which the replacement router has different MAC addresses and a different NIC attachment order, because those two differences are what a cross-appliance restore actually fails on
  • Separate the four backups a VyOS router produces — the boot file, the commit archive, an on-box snapshot, an off-box copy — and state the failure each one does and does not survive
  • Reproduce the two ways a backup lies: a change that was committed and never saved, and a `save <path>` that the operator believed updated `/config/config.boot`
  • Use `compare` as the change record for a restore, and identify from it the colleague work the restore is about to discard
  • Run a same-appliance restore under `commit-confirm` and verify it in dependency order: interfaces, addresses, adjacencies, routes, traffic
  • Restore onto the replacement chassis and watch the commit fail because `hw-id` pins an interface to a MAC that is not on this hardware
  • Produce the failure that has no error message at all: correct names, correct addresses, correct adjacency count, and cabling that does not match the configuration
  • Measure a recovery time for the full-router-loss drill and compare it against the stage table in Part LV-06

Prerequisites

Objective

Taking a backup on VyOS is one command and it is not the subject of this lab. The subject is the gap between the four things people call “the backup” and what each of them can actually recover, because that gap is invisible until the day you need it and then it is the whole incident.

By the end you will have produced all four — /config/config.boot, the commit archive, a labelled on-box snapshot, and an off-box copy — and you will have watched each of them fail at something. You will also have run the restore that is genuinely hard: onto a different chassis, where the configuration is correct, the commit is clean, every interface is up, the adjacency count is right, and no traffic passes.

That last failure is the one worth the lab time. It has no error message, no counter, and no log line. The only thing that separates it from a successful recovery is a table of MAC addresses somebody wrote down before the outage.

Architecture

Four VyOS VMs on four isolated bridges. edge-01 is the router in service; peer is the OSPF neighbour and holds the far service; ops is the LAN host that runs the end-to-end test and doubles as the backup target; edge-02 is the replacement chassis, and it exists from the first task even though nothing uses it until Task 6.

                            peer
                     +--------------------+
                     | eth0  MGMT   .13   |
                     | eth1  203.0.113.2  |
                     | dum0  203.0.113.200|
                     +--------------------+
                               |
                    TRANSIT 203.0.113.0/30
                               |
                     +--------------------+
                     |      edge-01       |
                     | eth0  MGMT   .11   |
                     | eth1  203.0.113.1  |
                     | eth2  198.51.100.1 |
                     +--------------------+
                               |
                     LAN 198.51.100.0/24
                               |
                     +--------------------+
                     |        ops         |
                     | eth0  MGMT   .20   |
                     | eth1  198.51.100.20|
                     +--------------------+

                     +--------------------+
                     |      edge-02       |   spare chassis
                     | eth0  MGMT   .12   |   NICs attached in a
                     | eth1  (LAN bridge) |   DIFFERENT order --
                     | eth2  (TRANSIT)    |   this is deliberate
                     +--------------------+
SegmentPrefixedge-01edge-02peerops
MGMT192.0.2.0/24eth0 · .11eth0 · .12eth0 · .13eth0 · .20
TRANSIT203.0.113.0/30eth1 · .1eth2eth1 · .2
LAN198.51.100.0/24eth2 · .1eth1eth1 · .20
Far service203.0.113.200/32dum0

Every address comes from a range RFC 5737 reserves for documentation, so nothing here can collide with a real network if a bridge is attached to something it should not be.

Read the edge-02 row again, because it is the design decision the whole second half of the lab rests on. On edge-02 the vNIC that lands on the LAN bridge is eth1, and on edge-01 the LAN is eth2. That is not a mistake in the table. It is the thing a replacement chassis does to you when the ports are cabled by their labels rather than by their bus positions, and it is what Part LV-04 means by “the interface names may differ”.

Requirements

  • A hypervisor with roughly 4 GiB of free RAM and 40 GiB of free disk — four VMs at 1 GiB and 8 GiB each is the floor, and edge-02 gets rebuilt from scratch in Task 8.
  • The VyOS 1.5 LTS ISO. compare saved, commit-confirm and the system config-management tree are all written against 1.5.
  • Four isolated layer-2 bridges with no physical port: MGMT, TRANSIT, LAN, and one spare you will re-attach in Task 7.
  • Console access to all four VMs, per the callout above.
  • The ability to attach and detach vNICs and choose their order on the hypervisor. Task 7 needs it.
  • No Internet access. Nothing here resolves DNS or reaches a public address.
  • Roughly two and a half hours, of which perhaps twenty minutes is waiting on reboots and adjacency timers.

Scenario

edge-01 has been in service for two years. It carries one OSPF adjacency, one LAN, and a forward filter that a security review added eight months ago. The person who built it has left. What exists is a nightly copy of its configuration on the ops host, a commit archive on the router itself, and a general belief that “we have backups”.

This week the chassis has started logging correctable memory errors and a replacement has arrived. Your job is to prove the belief before you need it: establish which backup can rebuild this router, on which hardware, losing how much recent work — and then actually do the rebuild while nothing is on fire.

Tasks

Configuration blocks are written for the [edit] prompt. Where a block opens with configure and you are already at [edit], skip that line. Blocks opening with set or delete continue the session you have open. Blocks tagged bash are the operational-mode shell or the hypervisor host, as their comments say. Blocks made up of show commands are operational mode; from inside configure, prefix each one with run.

Task 1 — Build the estate, and write down the chassis identity table

Install four routers from the ISO. Give each one a hostname and its MGMT address on eth0, and stop there for a moment — before any lab configuration, capture what each chassis physically is.

# Run from operational mode on edge-01, then on edge-02.
JOURNAL="$HOME/lab22"
mkdir -p "$JOURNAL"

for i in eth0 eth1 eth2; do
  printf '%-6s %s\n' "$i" "$(cat /sys/class/net/$i/address)"
done | tee "$JOURNAL/chassis-identity.txt"

That file is the deliverable this lab is built around. Three lines per chassis, name and MAC, and the bridge each vNIC is attached to — which the hypervisor knows and the router does not, so write it in by hand.

Everything after Task 5 is either easy or impossible depending on whether that table exists. In production it is the single cheapest artefact in a disaster-recovery plan and the one most reliably missing.

Now build edge-01 as the router in service:

configure
set system host-name edge-01
set system name-server 192.0.2.13
set system time-zone UTC
set system config-management commit-revisions 20

set interfaces ethernet eth0 address 192.0.2.11/24
set interfaces ethernet eth0 description 'MGMT'
set interfaces ethernet eth1 address 203.0.113.1/30
set interfaces ethernet eth1 description 'TRANSIT to peer'
set interfaces ethernet eth2 address 198.51.100.1/24
set interfaces ethernet eth2 description 'LAN'

set protocols ospf parameters router-id 192.0.2.11
set protocols ospf area 0 network 203.0.113.0/30
set protocols ospf area 0 network 198.51.100.0/24
set protocols ospf interface eth2 passive

set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall ipv4 forward filter default-action drop
set firewall ipv4 forward filter rule 10 action accept
set firewall ipv4 forward filter rule 10 description 'LAN to the far service'
set firewall ipv4 forward filter rule 10 inbound-interface name eth2
set firewall ipv4 forward filter rule 10 outbound-interface name eth1
set firewall ipv4 forward filter rule 10 state new
commit
save

peer holds the far service and the other end of the adjacency:

configure
set system host-name peer
set interfaces ethernet eth0 address 192.0.2.13/24
set interfaces ethernet eth1 address 203.0.113.2/30
set interfaces dummy dum0 address 203.0.113.200/32
set interfaces dummy dum0 description 'far service'
set protocols ospf parameters router-id 192.0.2.13
set protocols ospf area 0 network 203.0.113.0/30
set protocols ospf area 0 network 203.0.113.200/32
commit
save

ops is the LAN host, the end-to-end test source, and the backup target:

configure
set system host-name ops
set interfaces ethernet eth0 address 192.0.2.20/24
set interfaces ethernet eth1 address 198.51.100.20/24
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1
commit
save

edge-02 gets a hostname and a MGMT address and nothing else. It is a bare chassis, which is exactly what a replacement is.

Now establish the baseline from ops, not from edge-01. This distinction runs through the whole lab: a router can reach its own neighbour while forwarding nothing.

Read-only / Safeops
$ ping 203.0.113.200 count 3
PING 203.0.113.200 (203.0.113.200) 56(84) bytes of data.
64 bytes from 203.0.113.200: icmp_seq=1 ttl=63 time=1.02 ms
64 bytes from 203.0.113.200: icmp_seq=2 ttl=63 time=0.88 ms
64 bytes from 203.0.113.200: icmp_seq=3 ttl=63 time=0.91 ms

Illustrative output

On edge-01, record the four facts every later task compares against: the adjacency, the route, the interface set, and the configuration itself.

show ip ospf neighbor
show ip route
show interfaces
show configuration commands

Task 2 — What save persists, and the two ways a backup can lie

Both of these are one-line mistakes with a delayed consequence, which is why they survive code review and turn up during a recovery instead.

The first lie: committed, never saved. Add a route on edge-01, commit it, and deliberately do not save.

configure
set protocols static route 192.0.2.128/25 next-hop 203.0.113.2
set protocols static route 192.0.2.128/25 description 'committed, not saved'
commit
exit

The route is live. show ip route finds it, the FIB has it, traffic would use it. Now ask the only question that matters:

Read-only / Safeedge-01
[edit] vyos@edge-01# compare saved
[edit]
+protocols {
+    static {
+        route 192.0.2.128/25 {
+            next-hop 203.0.113.2 {
+            }
+        }
+    }
+}

Illustrative output

Any output at all from compare saved means the running configuration and the boot file disagree, and the boot file wins at the next reboot. Reboot edge-01 and confirm the route is gone. Two minutes of your time, and it is the failure mode Part LV-01 opens with.

The second lie: save <path> is not save. Re-apply the route, then take what looks like a backup:

configure
set protocols static route 192.0.2.128/25 next-hop 203.0.113.2
commit
save /config/backups/task2.boot
exit

The directory has to exist first — mkdir -p /config/backups from the operational shell — and the command reports success in the same shape as a plain save, which is the trap.

Read-only / Safeedge-01
[edit] vyos@edge-01# compare saved
[edit]
+protocols {
+    static {
+        route 192.0.2.128/25 {
+            next-hop 203.0.113.2 {
+            }
+        }
+    }
+}

Illustrative output

Identical output to the first case, from a completely different mistake. An operator who has typed save once that day and seen “Done” will not type it again, and the router is one unplanned reboot away from reverting a change everybody believes is in place.

Now save properly, and confirm with the check that understands both syntaxes:

configure
save
compare saved
exit

Empty output is the proof. Record in the journal that compare saved is the verification and a file timestamp is not.

Task 3 — Four backups, four different failures

You now have four artefacts, or you will by the end of this task. Produce each one, and write next to it in the journal the failure it survives and the failure it does not.

The boot file. /config/config.boot, written by save. Survives a reboot. Does not survive a disk failure, a reinstall, or delete system image.

The commit archive. Written automatically on every successful commit.

Read-only / Safeedge-01
$ show system commit
0   2026-08-19 11:04:22 by vyos via cli
1   2026-08-19 10:58:07 by vyos via cli
2   2026-08-19 10:41:55 by vyos via cli
3   2026-08-19 10:39:12 by vyos via cli

Illustrative output

Revision 0 is the running configuration and the numbers count backwards in time. The archive is as fresh as the last commit, which makes it the best source you have — and it lives on the router’s own disk, which makes it worthless in exactly the scenario the rest of this lab is about. Retention defaults to 20 entries, so on a busy router it can be a single afternoon.

A labelled on-box snapshot. Independent of archive pruning, and it holds states the archive cannot — an uncommitted repair, or a state you want to compare against rather than run.

configure
save /config/backups/pre-lab22.boot
exit

The off-box copy. Push it from ops; do not ask the router to fetch it.

# On ops. Substitute your own values before running:
ROUTER=192.0.2.11
LABEL=edge-01
STAMP=$(date +%Y%m%d-%H%M%S)

mkdir -p /home/vyos/configs
scp "vyos@$ROUTER:/config/config.boot" "/home/vyos/configs/$LABEL-$STAMP.boot"
ls -la /home/vyos/configs/

The timestamp in the filename is doing the job a version-control repository does in production. Part LV-02 is unambiguous that the off-box copy belongs in Git and that a single overwritten file is not a history; the lab uses stamped filenames because the point being made here is about lag, not about the version-control tool. Write the lag next to each one: the boot file is as fresh as your last save, the archive as fresh as your last commit, this copy as fresh as the last time you ran that scp.

Task 4 — Restore is not rollback, and compare is the only record of what you discard

Now simulate the thing that makes a restore expensive: somebody else’s work.

Fifteen minutes after your snapshot, a colleague adds a firewall rule. Apply it as them:

configure
set firewall ipv4 forward filter rule 20 action accept
set firewall ipv4 forward filter rule 20 description 'CHG-4471 monitoring probe to LAN'
set firewall ipv4 forward filter rule 20 inbound-interface name eth1
set firewall ipv4 forward filter rule 20 outbound-interface name eth2
set firewall ipv4 forward filter rule 20 protocol icmp
set firewall ipv4 forward filter rule 20 state new
commit
save
exit

Then break something yourself, in a way that looks like a bad change rather than a typo:

configure
delete protocols ospf area 0 network 198.51.100.0/24
commit
save
exit

The LAN prefix stops being advertised. From ops, the far service is still reachable — the outbound path is fine — but the return path has no route, so the ping stops. That asymmetry is worth noticing; it is the subject of Lab 24.

Before reaching for a file, ask the question Part LV-03 and the restore runbook both put first:

show system commit

The good state is revision 1. This is a rollback 1, not a restore, and doing it as a restore is the expensive way to get the same result. Do it as a restore anyway, this once, because the diff is the lesson.

configure
load /config/backups/pre-lab22.boot
compare

Read the whole diff, not the part you expected.

Read-only / Safeedge-01
$ compare
[edit firewall ipv4 forward filter]
-rule 20 {
-    action accept
-    description "CHG-4471 monitoring probe to LAN"
-    inbound-interface {
-        name eth1
-    }
-    outbound-interface {
-        name eth2
-    }
-    protocol icmp
-    state new
-}
[edit protocols ospf area 0]
+network 198.51.100.0/24

Illustrative output

Two hunks, two completely different meanings. The + line is the restore doing its job. The - block is a change somebody else made, for a ticket you have never read, which this commit will silently delete. Nothing else in this lab — and nothing in the router’s own output afterwards — will ever mention it again. The commit produces one archive entry and no summary of what it reverted.

discard here, then take the cheap path:

discard
rollback 1
compare
commit
save
exit

Paste the compare output into the journal and annotate every line as either “the restore” or “work being discarded”. That annotated diff is the change record for a restore, and in production it is the artefact that goes in the ticket.

Task 5 — The same-appliance restore, under a timer, verified in dependency order

Now do the restore properly, on a state the archive cannot reach. First take a fresh off-box copy, so the source is the state you validated at the end of Task 4 rather than the Task 3 one — otherwise this restore discards the colleague’s rule a second time and you will be reading a diff about that instead of about the fault:

# On ops. Substitute your own values before running:
ROUTER=192.0.2.11
STAMP=$(date +%Y%m%d-%H%M%S)

scp "vyos@$ROUTER:/config/config.boot" "/home/vyos/configs/edge-01-$STAMP.boot"

Now wreck edge-01 the way a bad change does — several nodes at once — and recover from that copy rather than from anything on the box.

configure
delete interfaces ethernet eth2 address
delete protocols ospf
delete firewall ipv4 forward filter rule 10
commit
save
exit

That is a router with a LAN interface holding no address, no routing protocol and a forward filter that drops everything. It is up, reachable on MGMT, and completely out of service — which is the realistic shape of a bad change.

Push the off-box copy back and load it:

# On ops. Substitute the filename you just captured:
ROUTER=192.0.2.11
FILE=/home/vyos/configs/edge-01-20260819-110800.boot

scp "$FILE" "vyos@$ROUTER:/tmp/restore.boot"
ssh "vyos@$ROUTER" 'ls -la /tmp/restore.boot'
configure
load /tmp/restore.boot
compare
commit-confirm 15
exit

commit-confirm is not a formality here. A full restore replaces the interface addressing, the management binding and the firewall in one commit, and any of those can carry the session you are typing into. The timer returns the router to a reachable state without requiring you to still be connected to it.

Size the timer for the verification you intend to run, not for how long the commit takes. An OSPF adjacency takes tens of seconds to reach Full; a timer that fires while you are still watching it revert a restore that was working.

Verify from operational mode, in dependency order. Order matters more than the individual commands:

show interfaces
show ip route
show ip ospf neighbor
show firewall ipv4 forward filter

Interfaces before addresses, addresses before adjacencies, adjacencies before routes, routes before traffic. Out of order, you get confident wrong answers: an adjacency stuck in Init because an interface never came up looks identical to one stuck because the network statement is missing, and only the ordering separates them.

Then the test that is not on the router at all:

Read-only / Safeops
$ ping 203.0.113.200 count 3
PING 203.0.113.200 (203.0.113.200) 56(84) bytes of data.
64 bytes from 203.0.113.200: icmp_seq=1 ttl=63 time=0.95 ms
64 bytes from 203.0.113.200: icmp_seq=2 ttl=63 time=0.87 ms
64 bytes from 203.0.113.200: icmp_seq=3 ttl=63 time=0.90 ms

Illustrative output

Only now keep it and persist it:

configure
confirm
save
compare saved
exit

Task 6 — The replacement chassis: hw-id pins a MAC that does not exist here

Everything so far has been the same appliance. Now pin the interface names, the way a production configuration for real hardware does, and take the file next door.

On edge-01, using the MAC addresses you measured in Task 1:

configure
set interfaces ethernet eth0 hw-id 52:54:00:11:aa:01
set interfaces ethernet eth1 hw-id 52:54:00:11:aa:02
set interfaces ethernet eth2 hw-id 52:54:00:11:aa:03
compare
commit
save
exit

Substitute your own measured values — those three are the shape of a QEMU MAC, not yours. hw-id binds a name to a specific NIC so that eth1 is always the NIC in a known socket regardless of the order the kernel enumerated them. It is the correct thing to have on a production router, and it is the thing that makes the file non-portable.

Take a fresh off-box copy, then load it onto edge-02:

# On ops.
ROUTER=192.0.2.11
STAMP=$(date +%Y%m%d-%H%M%S)

scp "vyos@$ROUTER:/config/config.boot" "/home/vyos/configs/edge-01-$STAMP.boot"
scp "/home/vyos/configs/edge-01-$STAMP.boot" "vyos@192.0.2.12:/tmp/restore.boot"

Before loading, do what Part LV-04 puts first and the incident always puts last: compare what the file expects against what the box has.

show interfaces
show configuration commands | match "hw-id"

Now load and commit, and let it fail:

configure
load /tmp/restore.boot
compare
commit
Service impact possibleedge-02
$ commit
[ interfaces ethernet eth1 ]
Error: Interface "eth1" hardware ID does not match any interface on this system

[[interfaces ethernet eth1]] failed
Commit failed

Illustrative output

Read what just happened carefully, because the good news is easy to miss. The commit engine is transactional with respect to the running configuration: the validator rejected the load and edge-02 is exactly as it was. A failed commit is not an outage.

The candidate, however, still holds the rejected content. The recovery is to fix the offending nodes or discard — not to retry the same commit.

Fix the hardware-bound nodes in the candidate, before committing, using edge-02’s own row of the identity table:

delete interfaces ethernet eth0 hw-id
delete interfaces ethernet eth1 hw-id
delete interfaces ethernet eth2 hw-id
set system host-name edge-02
set interfaces ethernet eth0 address 192.0.2.12/24
delete interfaces ethernet eth0 address 192.0.2.11/24
set protocols ospf parameters router-id 192.0.2.12
compare
commit
save
exit

Three separate corrections in that block, and only the first one had an error message.

The address change is the second failure mode from Part LV-04, and it is the one that reaches production. The restored file carried 192.0.2.11 — the address edge-01 is still using, because edge-01 is still powered on. Commit it unchanged and two routers claim one address: ARP flapping on the MGMT segment, and a management path that works intermittently for reasons that look like anything except a restore.

The router-id change is the third. It came in from the file too, and two OSPF routers sharing a router-id is its own class of incident.

Task 7 — The restore that commits cleanly and routes nothing

Delete the hw-id lines and the commit succeeds. That is what makes this the most dangerous shape a restore can take, and it is exactly the shortcut the break/fix scenario vyos-bf-restore-wrong-interface is built on.

You have already deleted them. Now finish the job the way a rushed engineer would: move edge-02 into edge-01’s place.

# On the hypervisor. Substitute your own VM and bridge names before running:
VM_OLD=edge-01
VM_NEW=edge-02

virsh destroy "$VM_OLD"
# Attach VM_NEW's vNICs to MGMT, LAN and TRANSIT --  in the order the
# chassis presents them, which for this VM puts LAN on eth1.

edge-02 now has the restored configuration and is cabled by socket label rather than by bus position. Commit and look at it:

show interfaces
show ip ospf neighbor

Everything is green. Three interfaces up, three addresses, one OSPF adjacency in Full. There is no error anywhere on the box. And from ops:

Read-only / Safeops
$ ping 203.0.113.200 count 3
PING 203.0.113.200 (203.0.113.200) 56(84) bytes of data.
From 198.51.100.20 icmp_seq=1 Destination Host Unreachable
From 198.51.100.20 icmp_seq=2 Destination Host Unreachable

--- 203.0.113.200 ping statistics ---
3 packets transmitted, 0 received, +2 errors, 100% packet loss

Illustrative output

Now find it. The configuration is correct, so no configuration command will show you anything wrong. The evidence is one layer down: which MAC is currently answering to each name.

# On edge-02, operational mode.
for i in eth0 eth1 eth2; do
  printf '%-6s %s\n' "$i" "$(cat /sys/class/net/$i/address)"
done

# And compare against the row you wrote in Task 1.
cat "$HOME/lab22/chassis-identity.txt"

eth1 holds the address 203.0.113.1/30 and is attached to the LAN bridge; eth2 holds 198.51.100.1/24 and is attached to TRANSIT. The adjacency formed because peer is on the segment eth2 is now cabled to, and OSPF does not care which interface name it runs over. Every single indicator is consistent with a working router.

There are two fixes and they are not equivalent.

Re-cable. Move the vNICs so the names land where the configuration expects. Immediate, no reboot, and leaves the configuration byte-identical to the off-box copy. It also makes this the one chassis in the estate whose socket labels disagree with the standard, and hands the same confusion to the next person in reverse.

Re-derive hw-id from this chassis. Set each name to the MAC that is in the socket you want it to be, from the measured table:

configure
set interfaces ethernet eth1 hw-id 52:54:00:22:bb:03
set interfaces ethernet eth2 hw-id 52:54:00:22:bb:02
compare
commit
save
exit

This keeps the cabling standard true for everyone and puts the chassis-specific data back where the design always had it. It needs a reboot for the renaming to settle, and it means the off-box copy for this router now differs from the one you restored — which has to be pushed back, or the next backup records it as an unexplained rewrite.

Choose one deliberately and write down why in the journal. “Deleted the hw-id lines to get a clean commit” is the sentence that appears in the break/fix ticket for this exact failure, and it is a decision made by nobody.

Task 8 — Time the drill

Everything above was untimed. Do it once more against a clock, because Part LV-06 is clear that a recovery procedure with no measured time is a hypothesis.

Rebuild edge-02 from the ISO — a genuinely bare chassis, empty commit archive, nothing on disk — and recover it from the off-box copy alone. Record each stage:

StageYour timePart LV-06 reference
Confirm the failure (console, not ping)5 min
Obtain the replacement (here: install from ISO)60 min
Transfer and load the configuration15 min
Adapt the hardware-bound nodesnot in the table
Adjacency verification10 min
Traffic verification, from ops10 min
Total130 min

Two things to notice in your own numbers.

The reference table has no row for adapting the hardware-bound nodes, and in this lab that stage is where you lost the most time — twice, once to an error message and once to no error message at all. A stage table that does not name it is a stage table written by somebody who has only ever restored onto the same appliance.

And the bottleneck will not be the restore. It is the install, exactly as Part LV-06 predicts, which is why the recommendation is a pre-configured spare rather than a faster recovery procedure. Write down which stage you would actually attack, and what it would cost.

Validation

  • The chassis identity table exists for both routers, with interface name, measured MAC, and the bridge each vNIC is attached to.
  • In Task 2, compare saved produced output in both cases and empty output after a plain save, and the reboot removed the unsaved route.
  • Four backup artefacts exist, each labelled with the moment it was taken and the failure it survives.
  • The Task 4 compare output is in the journal, annotated line by line as either the restore or discarded work, and it names the colleague’s rule.
  • The Task 5 restore was committed with commit-confirm, verified in dependency order, confirmed, saved, and compare saved afterwards returned nothing.
  • The Task 6 commit was refused by the validator with a hardware-ID error, and show interfaces on edge-02 was unchanged after the failed commit.
  • After the Task 6 corrections, edge-02 holds 192.0.2.12 and a router-id of 192.0.2.12, and no address appears twice in the estate.
  • In Task 7, show interfaces and show ip ospf neighbor on edge-02 both reported a healthy router while ping from ops failed, and the measured MAC table is what identified the cause.
  • A stage-by-stage recovery time exists for Task 8, with a named bottleneck.

Expected Outcome

A two-chassis lab in which you have produced every backup a VyOS router produces and made each of them fail at something specific: the boot file at a reinstall, the archive at a dead disk, the on-box snapshot at a chassis replacement, and the off-box copy at nothing except its own lag — which is the reason it is the one that matters.

More usefully, you have seen the two shapes a cross-appliance restore takes. One of them is loud: the validator refuses the commit, names the interface, and leaves the running configuration untouched. The other is silent: it commits, it comes up, every operational command agrees the router is healthy, and no traffic passes. You have a written procedure that catches the second one, and it is a table of MAC addresses.

Troubleshooting

save /config/backups/pre-lab22.boot reports that the directory does not exist. It is not created for you. mkdir -p /config/backups from the operational shell, once per router. /config is on the persistent partition, so it survives a reboot — but not a reinstall.

compare saved prints the whole configuration rather than a diff. You are comparing against an empty or absent boot file, which on a freshly installed router is the correct answer. Run a plain save and try again.

The Task 6 commit succeeds when you expected it to fail. The hw-id values in the file happen to match edge-02, which means you captured the file before the Task 6 set interfaces ... hw-id commit, or you measured edge-01’s MACs on the wrong VM. Check with show configuration commands | match "hw-id" on the candidate.

The OSPF adjacency will not form after a restore. Work outward and in order. show interfaces first — an adjacency cannot form over an interface that has no address. Then check that both ends agree on the network statement covering the transit prefix, and that the router-ids are distinct. show ip ospf interface names the area and the timers each side is using.

ops can reach 203.0.113.200 but the ping from edge-01 itself was already working. That is the normal state and it is why the test is run from ops. A router reaching its own neighbour proves the local interface works; it proves nothing about forwarding, the firewall, or the return path.

The restore committed and then reverted itself a few minutes later. The commit-confirm timer expired before you ran confirm. Reload and use a timer sized for adjacency formation rather than for the commit.

Two routers are fighting over 192.0.2.11 on the MGMT segment. Task 6’s address correction was skipped, or edge-01 was never powered off. Power it off, then clear the neighbour entry on the other hosts with ip neighbour flush dev eth0 before deciding whether the problem is fixed.

Everything is up on edge-02 and nothing routes, and the MAC table matches. Then the cabling is right and something else is wrong. Re-check the forward filter: rule 10 names inbound-interface eth2 and outbound-interface eth1, and if you fixed the naming by re-cabling rather than by hw-id, those two are now reversed. Read the rule’s packet counter in show firewall ipv4 forward filter.

Cleanup

Everything here is on isolated bridges, so cleanup means restoring a known state rather than restoring service.

Step 1. If you are keeping the topology for the backup and restore break/fix scenarios, stop here and save on each router. Note in the journal which chassis is currently in service, whether its interface names are pinned with hw-id, and which bridge each vNIC is on — the same table Task 1 produced. The next person to open this lab needs it for the same reason you did.

Step 2. To return the estate to its Task 1 state, power off edge-02, restore edge-01 from the on-box snapshot, and remove the lab-only nodes:

configure
load /config/backups/pre-lab22.boot
compare
commit
save
exit

Step 3. Remove the artefacts the lab left behind on edge-01:

# On edge-01, operational mode.
ls -la /config/backups/
rm -f /config/backups/task2.boot
rm -f /tmp/restore.boot

Leave /config/backups/pre-lab22.boot until you have confirmed the router is where you want it. It is the only thing that gets you back.

Step 4. On ops, decide what happens to the off-box copies. They contain the full configuration of both routers including any credentials, which in this lab is a lab password and in production is not:

# On ops.
ls -la /home/vyos/configs/

Step 5. To remove the topology entirely, delete the four VMs and the four bridges on the hypervisor. Confirm with ip -brief link show on the host that no lab bridge survives; an orphaned bridge with no members is the kind of thing somebody reuses six months later assuming it means something.

What You Learned

  • “The backup” is four different artefacts with four different lags. The boot file is as fresh as your last save, the archive as fresh as your last commit, the snapshot as fresh as whenever somebody typed it, and the off-box copy as fresh as its schedule. Picking one is choosing how much recent work to discard, and nothing in the restore output will tell you what that was.
  • compare saved is the verification; a timestamp is not. Two entirely different mistakes — a missing save, and a save <path> that did not touch the boot file — produce the same symptom and are both invisible until a reboot.
  • compare before a restore is the change record. It is the only place the work you are about to delete becomes visible, and it deserves to be pasted into the ticket rather than skimmed.
  • A failed commit is not an outage. The validator refusing a load leaves the running configuration untouched. The candidate keeps the rejected content, so the recovery is to fix or discard — never to retry the same commit and hope.
  • A cross-appliance restore is a migration. Interface names, hw-id, addressing and router-id are all bound to the chassis the file was written for, and only the first of them produces an error message. Fix them in the candidate, before the commit.
  • The dangerous restore is the one that commits cleanly. Correct names, correct addresses, adjacency Full, no traffic. Nothing the router can observe is wrong, because the fault is in the mapping between the configuration and the room.
  • The cheapest artefact in a disaster-recovery plan is a table of MAC addresses. Three lines per chassis, written while everything works. It is the difference between a fifteen-minute recovery and an afternoon.

Deliverables

  • · A lab journal with, for each task, the command run and the output observed
  • · A chassis identity table for both routers: interface name, measured MAC address, and the bridge each vNIC is attached to
  • · The four backup artefacts, each labelled with the moment it was taken and the failure it survives
  • · The `compare` output from Task 4, annotated with which lines are the restore and which lines are the work being discarded
  • · A measured recovery time for the Task 8 drill, broken down by stage, with the bottleneck named
  • · A written answer to the closing question: which of your four backups would have been useless in Task 6, and why did it still look fine?

Verification status

Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.