critical riskservice affecting~120 min

Runbook: Recover a Failed Router

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Death confirmed from the console or the power state, not from a failed ping. A router that is unreachable because the path to it broke is a network incident; rebuilding it is the most expensive possible response to a cable fault.
· Coverage established before anything else: is a VRRP peer, a second uplink or an ECMP sibling currently carrying this router's traffic? The answer sets the clock. Covered means hours and a change window; uncovered means the outage is live and every step below is on a stopwatch.
· The failed unit confirmed to be fully off, not half-alive. A router that boots far enough to claim its addresses while you are bringing up its replacement produces duplicate addresses and flapping adjacencies on top of the original fault.
· The configuration located and its age known: the off-box copy of /config/config.boot, ideally from version control, with the commit or timestamp identified. "There is a backup somewhere" is not located.
· The KEY MATERIAL accounted for separately. /config/config.boot does not contain the contents of /config/auth — WireGuard private keys, PKI certificates and keys, and anything else referenced by file path. A configuration restore without them produces a router that commits cleanly and cannot bring up its tunnels.
· The image version the failed router was running identified, and the same version available. Restoring a configuration onto a different major version can fail at commit, because configuration trees moved between 1.3, 1.4 and 1.5.
· The interface names of the replacement hardware checked against the ones in the configuration. Different hardware names interfaces differently, and a configuration that references an interface the box does not have fails to commit.
· Console access to the replacement arranged before it is racked. The first boot has no network configuration, and the restore itself changes the addressing that any remote session would be riding.
· The peers' owners identified for every routing adjacency this router carries, and told what is happening. If the replacement comes back on a different address, their configuration has to change too, and that is their change, not yours.
· A named decision-maker for the point of no return: the moment the replacement takes over the failed router's addresses. Everything before it is reversible; nothing after it is.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm the failure at the console, not over the network: serial or IPMI, power state, indicator LEDs, and one power cycle. Record what the console showed — a box that prints kernel output and then stops is a different fault from a box that prints nothing at all, and the second one is the only one that means "hardware".
2Establish coverage. show vrrp and the routing view on the surviving peer answer whether traffic is still flowing. Write the answer, and the resulting deadline, at the top of the incident record before doing anything else.
3If a VRRP peer has taken over, check that it is actually able to carry the traffic and not merely holding the virtual address: confirm its uplinks are up and its routing table is complete. A master with a dead uplink and no tracking configured black-holes everything sent to the virtual IP.
4Make the failed unit safe: power it down and, where possible, disconnect its uplinks. This is the step that prevents the replacement and the corpse from claiming the same addresses at the same time.
5Before wiping or discarding anything, attempt to recover what only the failed unit holds. If it boots far enough for a console or a disk read, pull /config/config.boot and the whole of /config/auth. This is the only reversibility available in this runbook and it expires the moment the disk is reinstalled.
6Assemble the recovery kit off the router: the configuration file and its provenance, the contents of /config/auth, the image version, and the interface map. Do not start the install until you have all four, because discovering a missing key three hours in costs the whole window.
7Install the replacement with the SAME VyOS version the failed router ran. A configuration written for one major version can be rejected at commit by another; a version-matched restore removes an entire class of failure from an already bad day.
8Bring the replacement up on a management address only, isolated from the production addressing. Verify console and management access before it holds anything that matters.
9Compare the replacement's interface names against the configuration with show interfaces, and adapt the configuration to the hardware BEFORE loading it. A commit that fails on a missing interface leaves you editing under pressure with the window running.
10Restore the key material into /config/auth with the same paths and the same restrictive permissions the configuration expects. Files here are referenced by path, so a key in the wrong place is indistinguishable from a missing one at commit time.
11Load, then verify, then commit, then save: load the configuration, compare to read exactly what is about to be applied, commit, and save. Read the compare output rather than skimming it — a wrong or stale configuration file is one of the two most common ways this recovery goes wrong.
12Verify adjacency before verifying traffic: show ip ospf neighbor, show ip bgp summary, show ip route summary. Sessions that stay down point at the peer, the firewall or a physical connection, in that order of likelihood, and each is a different owner.
13Verify traffic end to end from a real source to a real destination through this router, not from the router's own shell. A box with every adjacency up and no forwarding is a routing or firewall problem, and it looks perfect from the console.
14Reintegrate into the redundant pair deliberately and last. A recovered higher-priority router with preemption enabled takes the master role back within seconds of coming up — before its routing has converged — and black-holes traffic while it finishes. Hold it in backup until the routing view is complete, then hand the role over on purpose.
15Rebuild the safety net before closing: the new unit has an empty commit archive and therefore no rollback target, its configuration is not yet in the off-box backup, and its monitoring may still be pointed at a dead asset. A router without a backup is one failure away from repeating this runbook from a worse starting point.
16Close with the measured recovery time, what was missing from the recovery kit, and what the next drill must test. The gap between the documented procedure and what actually happened is the most valuable output of this incident.

4 · Verification

Confirm the procedure actually fixed the problem.

✓Console access to the replacement works independently of the production network, tested before the router carries traffic
✓show interfaces on the replacement matches every interface the configuration references, with the intended addresses and admin state
✓compare output was read in full before the commit, and matched the configuration that was intended to be restored, from the intended date
✓The configuration is both committed AND saved: it survives a deliberate reboot of the replacement, tested before the router is trusted with production traffic
✓The contents of /config/auth are present with the expected paths and permissions, and every tunnel that depends on them establishes rather than merely committing
✓show ip ospf neighbor and show ip bgp summary show every expected adjacency established, with counts matching the pre-failure baseline rather than merely being non-zero
✓show ip route summary is within the expected range for this router's role; a table that is thousands of routes short is an adjacency or policy problem, not a settled router
✓Traffic is verified end to end from a real source through this router to a real destination, including a return path check, and not from the router's own shell
✓For a redundant pair: show vrrp shows the intended router in the master role, the handover was deliberate rather than automatic, and the surviving peer is healthy in its new state
✓The recovered router is back in the backup regime — its configuration is in the off-box repository, its monitoring targets the new asset, and the recovery time is recorded against the business objective

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Every step up to the moment the replacement claims the failed router's production addresses is reversible: stop, leave the replacement isolated, and the estate stays exactly as it was with the surviving path carrying traffic.
↶After that moment there is no rollback in the ordinary sense. A rebuild is not a change you can revert; the previous state was a working router that no longer exists.
↶The replacement's commit archive starts EMPTY. rollback N has nothing to roll back to on a freshly installed box, so the off-box configuration file is the only recovery target until enough commits accumulate to build a local history.
↶A wrong configuration restored is corrected the same way it was applied: load the correct file, compare, commit, save. Correct it before the router carries traffic if at all possible — a wrong configuration that has already been advertised has told the rest of the network something untrue.
↶If the recovery is going badly and the traffic is covered by a surviving peer, the correct action is to stop and leave the replacement out of service rather than to keep trying. An unfinished rebuild that is holding addresses is worse than no rebuild at all.
↶Reintegration is the one late step that IS reversible: a recovered router held in backup can be left there indefinitely while the routing is checked, and the master role handed over when the evidence supports it.
↶The failed unit is not disposed of until the replacement has been verified and its configuration and key material are safely off-box. It may still hold the only copy of something.
↶If key material could not be recovered, rotation is the only path forward and it is a coordinated change with every peer that trusts the old keys — not a step inside this runbook.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to whoever can physically reach the site the moment the failure is confirmed as hardware. Every remote step below that point is blocked on someone with hands on the equipment.
· Escalate to the peers' owners as soon as the replacement's addressing is known to differ from the failed router's. Their neighbour configuration has to change before any session can establish, and that is their change with their own approval path.
· Escalate to security immediately if key material cannot be recovered, or if the failure has any possibility of being a compromise rather than a fault. Rebuilding a compromised router from its own configuration reinstates whatever was wrong with it.
· Escalate to network engineering before reintegrating into a redundant pair during business hours. The handover is a deliberate service event and it deserves the same treatment as any other planned failover.
· Escalate to the service owner when the measured recovery time is going to exceed the business objective. That is a decision about the business, not about the router, and it needs to be made by someone who owns the consequence.
· Escalate when the off-box configuration turns out to be missing, stale or untested. Reconstructing a router configuration from memory during an outage is a last resort and the estate needs to know it is happening.
· Hold rather than force: if a surviving peer is carrying the traffic acceptably, the correct move is often to stop, sleep, and rebuild in a window with the right people awake.

Three questions come before any rebuild, and answering them in the wrong order is how a recovery turns into a longer outage than the failure.

Is it actually dead? A router you cannot reach is not the same as a router that has stopped. The path to it may be what broke, and rebuilding a healthy router is the most expensive possible response to a cable fault.

Is anything carrying its traffic? This sets the clock and therefore the whole shape of the response. If a VRRP peer or a second uplink is covering, you have hours, a window, and the right people awake. If nothing is covering, the outage is live and every decision below is made under a stopwatch. These are two different incidents and they should not be run the same way.

What does the recovery cost when it goes wrong? The two failure modes that dominate are both about identity rather than configuration: two devices claiming the same addresses, and a recovered router taking back a role it is not yet ready to perform. Neither is fixed by a better restore procedure. Both are avoided by sequencing.

The restore itself — load, compare, commit, save — is the easy part, and it is not where these incidents are lost.

When to use this runbook

A production VyOS router is unreachable and confirmed unresponsive at the console.
A router has suffered hardware failure and needs replacing with a spare or an RMA unit.
A router’s system disk or saved configuration is unrecoverable and the box must be rebuilt from off-box material.
A DR drill is exercising exactly this procedure, which is the only way anyone finds out whether it works.

When not to use it

A router that is unreachable but forwarding. If traffic is passing, this is a management-plane problem. Fix the access path.
A bad configuration on a live router. That is a rollback, and the archive on the box will do it in one command. Do not rebuild what you can revert.
A failed upgrade. Booting the previous image is a much smaller operation than reinstalling and restoring.
A suspected compromise. Rebuilding from the router’s own configuration reinstates whatever was done to it. That path starts with security, not with an ISO.

Blast radius

Everything this router carries, for as long as it is out — and, if it is part of a redundant pair, the surviving peer is now a single point of failure for all of it.

The rebuild has a point of no return: the moment the replacement takes the failed router’s production addresses. Before that moment, stopping costs nothing and the estate stays as it is. After it, the replacement is in the path whether it is finished or not.

Inputs — the recovery kit

You need all of these before the install starts, not during it:

The off-box copy of /config/config.boot, with a known provenance and date.
The contents of /config/auth for this router.
The image version the failed router was running.
The interface map for the replacement hardware.
Console access to the replacement.
The peers’ owners, and the addresses their configuration expects.

Step 1: Prove it is dead

Read-only / Safefrom the console, not from a ping

show version
show interfaces
show ip route summary

If you can run those, it is not dead and this is the wrong runbook.

If you cannot reach a console at all: check the power state, check the indicator LEDs, and power cycle it once. Record what the console printed. A box that emits kernel output and then stops has failed differently from a box that emits nothing, and only the second reliably means hardware.

The discipline here exists because “unreachable” is the most over-diagnosed condition in network operations. Every unreachable router gets a console check before anyone declares it dead.

Step 2: Establish coverage, and set the clock

Read-only / Saferun these on the surviving peer

show vrrp
show ip bgp summary
show ip ospf neighbor
show ip route summary

Write the answer at the top of the incident record, because it determines everything else: covered means a planned rebuild; uncovered means a live outage.

Step 3: Make sure the corpse stays down

Power the failed unit off, and disconnect its uplinks where you can reach them.

This is the least interesting step in the runbook and it prevents the worst outcome. A router that half-boots — enough to claim its addresses, not enough to forward — while the replacement is coming up on the same addresses produces duplicate addressing and flapping adjacencies layered on top of the original fault, and the resulting evidence is very hard to read.

Step 4: Recover what only the dead unit holds

Before anything is wiped, try to read the disk. If the failed router boots far enough for a console or the disk can be read in another machine, take /config/config.boot and the entire /config/auth directory.

This is the only genuine reversibility available in this whole procedure, and it expires at the moment the disk is reinstalled. It matters most precisely when the off-box backup turns out to be older than anyone believed — which is what DR drills exist to discover, and what unrehearsed recoveries discover instead.

Step 5: Install at the same version

Bring the replacement up on management addressing only, isolated from the production addresses. Verify console and management access now, while nothing depends on it.

Step 6: Map the interfaces before loading anything

Read-only / Safethe hardware decides the names, not the configuration

show interfaces
show configuration commands | match 'interfaces ethernet'

Interface naming depends on the platform and the kernel’s naming scheme, so a replacement — especially a different model — may present eno1 where the configuration says eth0. A configuration that references an interface the box does not have fails to commit.

Adapt the configuration to the hardware before the load, not after a failed commit. Under time pressure, editing a live candidate that has half-applied is where mistakes compound.

Step 7: Load, compare, commit, save

Configuration changefour steps, and the second one is the important one

configure
load /tmp/edge-01-config.boot
compare
commit
save
exit

load replaces the candidate configuration with the file’s contents; nothing has happened to the running router yet. compare shows the difference between that candidate and what is running — read it in full. On a freshly installed box the diff is enormous, which is exactly why people skim it, and skimming is how the wrong file or a six-month-old file gets committed.

Two specific things to look for in that output: addresses that belong to a different router, and a configuration whose date does not match the change history you expect.

commit applies the candidate to the running configuration, which lives in memory. save writes that running configuration to /config/config.boot, which is what the router reads at boot. Both are required. A recovery that was committed and never saved is a router that loses everything at its next reboot — and reboots are common in the days after a hardware replacement.

Step 8: Restore the key material

Put the /config/auth contents back at the same paths with the same restrictive permissions. The configuration references these files by path, so a key in the wrong directory behaves exactly like a missing one.

Then prove they work rather than assuming: a tunnel that commits is not a tunnel that establishes, and the difference is the whole point of this step.

Step 9: Adjacency first, then traffic

Read-only / Safein this order

show ip ospf neighbor
show ip bgp summary
show ip route summary
show ip route

Compare the counts against the pre-failure baseline rather than settling for non-zero. Sessions that stay down have three usual causes, and they belong to different people:

The peer is also affected. Common when the failure was environmental — power, cooling, a whole rack.
A firewall rule for the protocol did not come back. Check what the restored configuration actually contains rather than what it should.
The physical connection is not there. A cable moved during the replacement, or a port that was patched differently.

Only when adjacency is complete does traffic verification mean anything. Test from a real source through the router to a real destination, and check the return direction too. A router with every session established and no forwarding looks perfect from its own console.

Step 10: Reintegrate on purpose

A recovered router pre-empts before it is ready

With preemption enabled — the default — a higher-priority router that comes back online announces itself and the current master steps down within seconds. That happens as soon as VRRP is running, which is well before BGP has finished converging and the forwarding table is complete.

The result is a router that holds the virtual address and cannot yet forward what is sent to it: a second outage, caused by the recovery, minutes after everyone relaxed.

Bring the replacement up in a state where it cannot take the role — held in backup, or with its production interfaces still down — verify the routing view is complete, and then hand the role over deliberately. This is the one late step in the runbook that is fully reversible, so there is no reason to rush it.

Step 11: Rebuild the safety net before you close

The replacement is not finished when traffic flows. Three things are missing and each one costs you the next incident:

The commit archive is empty. rollback has nothing to roll back to on a freshly installed box, so the ordinary recovery mechanism for a bad change does not exist here yet.
The configuration is not backed up. The router you have just rebuilt is the one with no off-box copy of its current state.
Monitoring may still be watching the dead asset. A replacement that nobody is alerting on is a router whose next failure is discovered by a customer.

Then record the measured recovery time against the business objective, and — more valuable than anything else here — what was missing from the recovery kit. That gap is the actual finding of this incident.

Rollback

Stage	What reverting means
Before the replacement holds production addresses	Stop. Leave it isolated; the estate is unchanged.
Wrong configuration loaded, not yet committed	`discard`, or `load` the correct file and `compare` again
Wrong configuration committed	`load` the correct file, `compare`, `commit`, `save` — before it carries traffic if at all possible
After the replacement is in the path	No rollback exists. The previous state was a router that no longer exists.
Reintegration into a redundant pair	Fully reversible: hold it in backup for as long as the evidence requires
Failed unit disposal	Not until the replacement is verified and its configuration and keys are off-box

The line that matters is the third one. A rebuild is not a change with an undo; the sequencing above exists precisely because the reversibility has to be created deliberately, before it is needed.

Common patterns

Observation	Likely cause	Next step
Commit fails on an interface that does not exist	Replacement names interfaces differently	Map with `show interfaces`, adapt the configuration, reload
Commit fails on a node nobody wrote	Configuration from a different major version	Install the matching version; upgrade separately
Everything commits, tunnels do not establish	`/config/auth` key material missing	Restore the files by path; rotate if unrecoverable
Configuration gone after a reboot	Committed but never saved	`save`, then reboot again to prove it
Duplicate addresses, flapping adjacencies	The failed unit is half-alive	Power it off properly and disconnect its uplinks
Adjacencies down on one side only	Firewall rule for the protocol did not come back	Compare the restored configuration against the baseline
Traffic black-holed after the peer took over	VRRP master without tracking, uplink down	Check the survivor’s uplinks; fix tracking as follow-up
Second outage minutes after recovery	Preemption took the role back pre-convergence	Hold in backup; hand the role over deliberately
Restored configuration is months out of date	Backup pipeline was never validated	Recover from the failed disk if possible; fix the pipeline

Escalation

Escalate when:

The failure is confirmed as hardware — you need hands on site.
The replacement’s addressing differs from the failed router’s; the peers must change their configuration.
Key material cannot be recovered, or compromise is possible at all.
Reintegration into a redundant pair is due during business hours.
The recovery time is going to exceed the business objective.
The off-box configuration is missing, stale or was never tested.

Why this matters

The restore is four commands. Everything expensive about this runbook is either side of them.

Before: the recovery kit. A configuration file is not a router. The keys under /config/auth, the image version and the interface map are all required, all separate from the configuration, and all discovered to be missing at the worst possible moment by anyone who has not rehearsed the procedure. A DR drill is the only thing that finds this out cheaply, and the recovery time it measures is the number the business is actually buying.

After: identity. Two devices claiming the same addresses, and a recovered router taking back a role before it can perform it, are the two failure modes that turn a recovery into a longer outage than the failure. Neither has anything to do with the configuration being correct. Both are prevented by the same habit — decide when this router re-enters the path, and make that a step somebody performs rather than something that happens on its own while you are still checking.