medium riskservice affecting~35 min

Runbook: Configuration Error Recovery

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Out-of-band access is confirmed before any further commit - console, serial or IPMI. If the change touched the path you are on, the session you are reading this in is not a resource to spend
· The candidate has been captured, not discarded: from configure, run show configuration commands and compare into the incident record. A rejected commit writes no archive entry, so the candidate is the only copy of what failed
· The commit log has been read rather than guessed: show log commit | last 10
· The archive listing is on file: show configuration commit names the entry the router was in before the change, with the user and source address for each
· Whether the running configuration changed at all has been established. A rejected commit is not an outage and does not deserve outage-speed decisions
· If a commit-confirm timer is running, its deadline is known and somebody is watching it: show commit-confirm

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Do not discard. Capture the failed candidate first with show configuration commands and compare from configure - the archive holds successful commits only, so nothing else has a copy.
2Read show log commit | last 10 and identify which phase failed: validators, generators, or apply. The phase decides whether the running configuration is untouched or possibly mixed.
3Classify the failure into one of six states - commit rejected, commit applied partway, commit succeeded but the change is wrong, session lost, commit-confirm timer fired, or commit succeeded with no effect. The rest of the runbook branches here.
4Commit rejected: fix the leaf the message names and re-commit. Do not loop discard and commit hoping for a different result - the message identifies the directive.
5Commit applied partway: check the runtime for each subsystem the change touched before committing anything else. The configuration tree does not describe a half-applied backend.
6Commit succeeded and the change is wrong: run show configuration commit, then compare N against the target entry BEFORE any rollback, and read the whole diff.
7If the diff contains directives you cannot attribute, do not roll back. Revert your own change with targeted delete commands instead, and raise the unattributed directives with the change owner.
8Revert with rollback N, then compare, then commit-confirm 5 for anything remote. rollback N loads the candidate and does not commit.
9Leave configure mode and test from the operational shell. If the revert is correct, re-enter configure and run confirm. If it is not, do nothing and let the timer restore the previous state.
10Verify in the operational view - routes, sessions, counters, traffic - not in the configuration tree. A committed reference to a group or interface that does not exist is valid configuration and has no runtime effect.
11Run save once the running configuration is correct. A fix that is live but not saved is the same incident again after the next reboot, and the recurrence is the confusing one.
12Record the archive entry the router is in now, the entry it was in before, the phase that failed, and anything that was reverted which did not belong to this change.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show log commit | last 10 shows a successful commit as the most recent entry, with no validation, generator or apply failure after it
✓compare from configure produces no output - the candidate matches the running configuration and nothing is left pending for the next operator
✓The runtime agrees with the tree for every subsystem the change touched: show ip route, show bgp summary, show ip ospf neighbor, or the equivalent
✓The symptom that produced the page is gone, measured on traffic rather than on configuration
✓configure, load /config/config.boot, compare, discard produces no diff: the fix is saved, not only live
✓The incident record names the archive entry the router is in now, the entry it was in before, and anything reverted that belonged to another change
✓Any commit-confirm timer has been resolved deliberately with confirm or allowed to fire, not left running past the end of the incident

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶rollback N loads the N-th archive entry into the candidate and does nothing else. compare, then commit, then save. An operator who runs rollback 1 and walks away has reverted nothing
↶Run compare N BEFORE rollback N. The diff covers everything between now and that entry, including changes made by other people; a rollback that reverts three changes to undo one is a second incident
↶For a remote revert: rollback N, compare, commit-confirm 5, leave configure mode, test, then confirm. A revert is a change and inherits every risk the original change had
↶If the archive entry will not load - schema differences across a release, a corrupted file - use a different entry or load /config/backups/LABEL.boot. A failed load leaves the running configuration untouched
↶POINT OF NO RETURN: a rollback that also reverted another change does not restore it by rolling back again. Re-apply it from compare N one entry further back, and tell whoever owns it
↶POINT OF NO RETURN: withdrawals that already reached a neighbour. Restoring the local configuration re-advertises; it does not undo the withdrawal, the session reset, or any dampening the neighbour applied
↶POINT OF NO RETURN: a daemon restart used during recovery. The previous process state is gone and cannot be reconstructed from the configuration tree

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if the commit failed after the validators. A generator or apply failure can leave one backend reloaded and another not, and that is a state the configuration tree does not describe
· Escalate to the change owner before any rollback whose compare N diff contains a directive you cannot attribute
· Escalate to network engineering if this router is one of a redundant pair - reverting one half of a pair can be worse than the original fault
· Escalate to the platform team if df -h /config shows no free space or /config/archive/ is not writable: without an archive entry the next commit has no rollback point
· Escalate to security if the failing change touches authentication, key material, API access or firewall policy. Reverting a security control reopens whatever it closed, and that is a security decision rather than an availability one
· Escalate rather than improvise if the session is lost and no commit-confirm timer is running. The recovery is out-of-band; a second blind attempt from a different host usually makes the evidence worse

“Configuration error” is one phrase covering six different situations that arrive with the same page. Two of them are not emergencies. One has already been resolved by the router without anybody asking it to. One cannot be recovered from the session you are sitting in.

The runbook turns on telling them apart in the first two minutes, because the two reflexes an operator reaches for under pressure are both wrong in most of the six. discard destroys the only copy of the change that failed. rollback 1 reverts everything that happened since the last archive entry, which on a shared router is frequently more than your change.

When to use this runbook

A commit has been rejected, has applied partway, or has applied something you did not intend - and you can still reach the router.

When not to use it

You cannot reach the router at all. That is vyos-rb-recover-failed-router, and the entry point is the console, not another SSH attempt.
The change was technically correct and is being reversed for policy or scheduling reasons. That is vyos-rb-rollback-bad-config.
Routing looks wrong but no change was made. This is a configuration runbook; start at vyos-rb-troubleshoot-missing-route instead and come back only if the evidence points at a commit.

Blast radius

The diagnosis is read-only. The recovery is a commit, which means it carries the same risk as the change that caused the incident - on a router that has just proved it can surprise you. Treat the revert with the discipline of a change, not the urgency of a fix.

Step 1: capture the candidate, then stop touching it

Read-only / Safethe failed change exists in exactly one place

configure
show configuration commands
compare

The commit archive records successful commits only. A commit that was rejected writes no entry, so the candidate in configd memory is the sole copy of what you tried to do. discard deletes it, and with it the diff a reviewer needs to tell you what was wrong.

The candidate also survives exit, which cuts both ways: your work is still there when you come back, and so is everybody else’s. Read the compare output before you assume the diff is yours.

Step 2: read the commit log; it names the phase that failed

Read-only / Safewhich phase, which leaf, which user

show log commit | last 10
show configuration commit

2026-08-15T14:23:01+01:00 vyos configd[1234]: commit by vyos from 10.99.0.99 (tty/0)
2026-08-15T14:23:01+01:00 vyos configd[1234]: validation failed: value 70000 out of range for as-number
2026-08-15T14:23:01+01:00 vyos configd[1234]: rollback to previous candidate
2026-08-15T14:23:01+01:00 vyos configd[1234]: unlock

Illustrative output

A commit runs through six phases: lock, validators, generators, renderer, apply, archive and unlock. The phase that failed is the single most useful fact in the incident, because it decides whether the running configuration was touched.

Phase	Checks	Message shape	Running configuration
Validators	format, range, references, mutual exclusion	`validation failed: value 70000 out of range for as-number`	untouched
Generators	per-node rendering to the backend command	`cannot bind 192.0.2.1 to eth0: address in use`	tree untouched; backends may not be
Apply	the backend accepts or rejects	`bgpd: %AS 64512 already configured with different router-id`	tree untouched; backends may not be

Step 3: classify, then branch

Symptom	What actually happened	Go to
`commit` returned an error and nothing changed	Rejected at the validators	Step 4
`commit` errored and one subsystem behaves differently from the tree	Applied partway	Step 5
`commit` succeeded and the router is doing the wrong thing	Applied, and the change is wrong	Step 6
You are reading this from a different host because the session dropped	The change took the path you were on	Step 7
The change worked, then reverted itself several minutes later	A `commit-confirm` timer fired	Step 8
`commit` succeeded, the tree looks right, nothing happened	A dangling reference	Step 9

Step 4: rejected at the validators - the cheap case

Nothing is broken. The running configuration is what it was before you started, the router is forwarding, and you have a bad candidate and a message that names the leaf.

The failure mode here is behavioural rather than technical: an operator under pressure starts cycling discard, retype, commit, discard. That is flailing. The validator message identifies the directive and the reason, and the three families that produce most of them are known:

Wrong scope after edit. A set typed from inside edit protocols bgp builds a path under the BGP node that the schema does not have. Recovery is top, then set from the right scope.
Unquoted string values. Quote every string value; a description with a space that was not quoted is stored as something else.
A reference to something that does not exist. A firewall group, an interface, a VRF or a route map named in the change but never created. Create the referenced object in the same commit, or in an earlier one.

Fix the named leaf, compare to confirm the diff is now what you intended, and commit.

Step 5: applied partway - the expensive case

The evidence you need is in the runtime, not in the configuration.

Read-only / Safeask each subsystem directly

show log commit | last 20
show configuration commands | match protocols
show ip route
show bgp summary
show ip ospf neighbor

Read-only / Safethe daemon log is the transition timeline

# Substitute your own value before running:
PEER=192.0.2.2

journalctl -u frr --since '30 min ago' | grep "$PEER"

Step 6: applied, and the change is wrong

Read-only / Safesee what a rollback would do before doing it

show configuration commit
compare 3

compare N diffs the candidate against the N-th archive entry. Read the whole output. It is not only your change - it is every change committed between that entry and now.

Service impact possiblethe revert, with a net under it

rollback 3
compare
commit-confirm 5
exit

rollback N loads the archive entry into the candidate. It does not commit. The most common misuse of this command is to run it and walk away believing the revert is live; the running configuration is unchanged until commit.

Test from the operational shell while the timer runs, then:

Configuration changekeep it, or let the timer take it back

configure
confirm
save
exit

If the revert is wrong, do nothing. The timer restores the previous state without a second decision from an operator who has now been wrong twice.

Step 7: the session is gone

You changed the path you were connected over. This is decided before the change, not after: either a commit-confirm timer is running, in which case the correct action is to wait for it, or it is not, in which case the recovery is the console.

Two things worth knowing before you rely on the timer. show commit-confirm displays it, but only from a session you no longer have. And the timer lives in configd memory - if configd crashes or the router reboots before confirm, the timer is lost and the change persists, because it was committed and never rolled back. Commit-confirm protects against a wrong change. It does not protect against a crash.

Step 8: the timer fired and took the change with it

The change worked, and several minutes later it reverted. The archive shows the rollback and the change is gone.

This is the mechanism working. The operator was pulled into something else and never ran confirm. Re-apply the change and confirm it this time - and resist the conclusion that the change was wrong, because nothing about the revert is evidence either way.

Step 9: committed, valid, and doing nothing

A rule that references a firewall group name with a typo commits successfully. The schema is satisfied, the reference is dangling, and the rule has no effect at runtime - so the operator believes traffic is being filtered when it is being accepted.

This is the case compare cannot catch, because compare reads the tree and the tree is fine. The check is the operational view: the counters on the rule, the routes in the table, the session in the summary. After any commit that touches a filter or a protocol, verify what the subsystem is doing rather than what the configuration says it should do.

Step 10: persist it

Once the running configuration is correct, save. A fix that is live and unsaved is the same incident again after the next reboot, and the recurrence is the expensive one: the change record says it was fixed, so nobody looks at the configuration first.

Common patterns

Symptom	Likely cause	Resolution
Commit hangs with no output	Another operator holds the configuration lock	Wait; the lock releases on their commit. Find out who
`rollback 1` had no effect	`rollback` does not commit	`compare`, `commit`, `save`
Rollback fixed the fault and broke something else	The archive entry predates an unrelated change	`compare N` first; use targeted `delete` on a shared router
Commit succeeded, the rule does nothing	Reference to a group or interface that does not exist	Create the object, or correct the name
Change reverted itself minutes later	`commit-confirm` timer fired without `confirm`	Re-apply and confirm
Change gone after an unrelated reboot	Committed but never saved	Re-apply from the archive entry, then `save`
Archive entry will not load	Schema differences across releases, or a corrupt file	Use another entry, or `load /config/backups/LABEL.boot`
One subsystem behaves differently from the tree	Partial apply after a generator or apply failure	Narrow the change or hold; do not blind-commit again

Escalation

Escalate when:

The commit failed after the validators.
A compare N diff contains a directive you cannot attribute.
The router is one of a redundant pair.
/config is full or /config/archive/ is unwritable.
The change touches authentication, key material, API access or firewall policy.
The session is lost and no timer is running.

Why this matters

Almost every configuration incident that turns into two incidents does so in the first two minutes, and always for one of two reasons.

The first is discard. It feels like cleaning up. It is deleting the only record of a change that has not been diagnosed yet, on a system whose archive deliberately does not keep failed candidates.

The second is treating rollback N as an undo button. It is not an undo button for your change; it is a restore to a point in time, and everything that landed after that point goes with it. On a router that two teams touch, the reflex rollback is how a routing problem becomes a routing problem plus a firewall problem, with the second one attributed to whoever ran the command.

Both are avoided by the same habit: read before you write. show log commit names the phase, compare N names what a rollback would cost, and the operational view names whether the fix worked. None of them changes anything, and all three are faster than the second incident.