high riskservice affecting~45 min

Runbook: Roll Back a Bad Configuration

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The archive listing has been read, not assumed: show configuration commit gives every entry a timestamp, a user and a source address. The target is chosen from that list by timestamp, because the bad commit is not necessarily the most recent one
· Every commit AFTER the bad one is identified and attributed. Reverting past them reverts them too, and one of them may be an unrelated change somebody else still needs
· The commit log has been read to establish whether the bad commit applied fully or failed partway: show log commit | last 10. A commit that failed at generator or apply time can leave one subsystem reloaded and another not, and a revert from that state is not symmetric
· Out-of-band access is confirmed and tested. A revert of an interface, VRF, SSH or firewall node removes the same access path the original change did - only in the other direction
· The commit-confirm timer length is chosen before the first commit, and everyone understands which state it falls back to: after a rollback, letting the timer expire returns the router to the BAD configuration
· The archive has not been pruned past the target. commit-revisions defaults to 20 entries; on a busy router that can be a single afternoon, and a pruned entry exists only in a backup file or nowhere
· A judgement has been made that reverting is the right action at all. A forward fix is sometimes correct - notably when the change is a security control, or when something has already been built on top of it

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read show configuration commit and identify the bad commit by timestamp, user and source address. Do not assume it is the most recent entry; three commits may have landed since, and reverting only the last of them leaves the bad change in place.
2Read show log commit | last 10 to establish how the bad commit ended. A clean validator rejection changed nothing; a generator or apply failure may have left one subsystem on the new configuration and another on the old.
3Capture the current state before changing it: show configuration commands off-box, plus show interfaces, show ip route, show ip bgp summary and show ip ospf neighbor. The broken configuration is the evidence for why the revert was needed.
4Inspect the candidate target WITHOUT loading it: compare N shows the difference between the candidate and archive entry N. This is how the target is confirmed before anything touches the candidate.
5Load the target into the candidate: rollback N. This does NOT commit. The running configuration is unchanged and stays unchanged until the operator commits, which is the single most common failure of this procedure.
6Read the full compare output. The archive entry was correct for the moment it was taken; if the network has moved since, the revert reverts those changes as well - the classic case being an interface change made after the bad commit.
7Commit with a timer for any change carrying the access path: commit-confirm 10, then leave configure mode to verify. Understand the direction: if the timer expires, the router returns to the configuration you are trying to escape.
8Verify in dependency order from operational mode: interfaces, then addressing, then adjacencies, then routes, then traffic. The symptom that triggered the revert must be gone, and nothing new may have appeared.
9Run confirm to keep the revert, then save to persist it. An unconfirmed revert expires; an unsaved one survives until the next reboot and then quietly reinstates the bad configuration.
10Re-read show configuration commit. The revert wrote a NEW archive entry, so every index has moved; any number computed before this point is now stale.
11Reconcile the source of truth. If the bad change came from a repository or an automation pipeline, revert it there too, or the next scheduled run re-applies it with no operator behind it.
12Record the timeline and the diff: what the bad commit changed, what the revert restored, and which commits made after it were reverted as collateral.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The symptom that triggered the revert is gone, checked with the same command that first showed it rather than with a general health check
✓compare against the target entry returns nothing, proving the running configuration matches the state that was aimed for
✓Adjacencies have re-formed rather than merely being configured: show ip ospf neighbor at Full, show ip bgp summary Established with a prefix count in the expected band
✓show ip route holds the expected routes including the default, and traffic passes end to end through the router
✓No unrelated change was reverted as collateral, verified by walking the compare output line by line rather than by counting lines
✓show configuration commit shows a new entry for the revert commit, attributed to the operator who ran it, with no unexplained entry alongside it
✓The revert was confirmed and persisted: show log commit | last 5 shows the commit and the confirm, and diff /config/config.boot <(show configuration commands) returns nothing
✓The repository or pipeline that produced the bad change has been reverted too, so the next scheduled run does not re-apply it

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Before committing, discard throws the loaded candidate away and leaves the running configuration untouched. Right up to the commit, this procedure costs nothing to abandon
↶While the commit-confirm timer runs, doing nothing reverts the revert. That is the intended behaviour when the revert turns out to be wrong - and the reason the timer must be sized for the verification you actually intend to run
↶A revert that went to the wrong entry is corrected by selecting a different entry from the freshly re-read listing and repeating the procedure. Do not adjust the number by arithmetic; the revert commit already shifted the indices
↶A rollback that fails at validator time leaves the candidate holding the rejected content and the running configuration completely unchanged. Fix the offending node by hand, or discard and choose a different entry - do not re-run the same commit
↶POINT OF NO RETURN: an advertisement that was withdrawn while the bad configuration was live. Restoring the configuration re-announces the prefix; it does not undo the reconvergence, the traffic that moved, or any dampening a neighbour applied
↶POINT OF NO RETURN: sessions and flows dropped during the bad window. The revert restores the rule set and the peering configuration, not the connections that were torn down while it was wrong
↶POINT OF NO RETURN: anything the change did outside the configuration tree - an image installed, a credential agreed with a peer, a certificate issued. The configuration node that requested it reverts; the action does not
↶POINT OF NO RETURN: a pruned archive entry. Retention is finite and pruning is irreversible; past that horizon this is a restore from a file, not a rollback

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the owner of every commit made after the bad one before reverting past it. Their change is about to be undone and they are the only person who knows what depends on it
· Escalate to network engineering before reverting on one half of a redundant pair. The revert drops and re-forms adjacencies, so the pair runs on one member for the duration
· Escalate to security before reverting a change that was a security control. Reverting reopens whatever it closed, and that is a security decision rather than an operational one
· Escalate if the commit log shows the bad commit failed partway through. A half-applied change means the running state does not correspond to any archive entry, and reverting is not the only option on the table
· Escalate if show configuration commit holds an entry nobody can attribute. Reverting overwrites it, and an unexplained commit is a security question before it is a change-management one
· Escalate to the automation owner if the change originated from a pipeline or repository. A revert on the box that is not matched in the repository is undone by the next scheduled run, with nobody watching
· Escalate to the platform team if the target entry has been pruned and no backup file covers it. That is a rebuild conversation, not a rollback

Two failures account for most bad rollbacks on VyOS, and neither of them is subtle once you have seen it.

The first is that rollback N does not commit. It loads an archive entry into the candidate and stops. The running configuration is untouched, the router is still doing the wrong thing, and the operator - who has typed the recovery command and seen it succeed - reports the incident resolved. The lesson on rollback names this as the command’s most common misuse for a reason.

The second is that the bad commit is not always the last commit. Under pressure rollback 1 is what the fingers type, and if three commits have landed since the bad one, the state one step back still contains it. Worse, if the bad commit was the last one but two colleagues committed unrelated work before it, a revert that reaches too far takes their work with it and nothing in the output says so.

Everything in this runbook is arranged around those two: commit deliberately, and choose the target by reading rather than by reflex.

When to use this runbook

A recent commit caused the problem and the good state is still in the archive.
A change was applied correctly and turns out to be the wrong change.
A commit-confirm was confirmed and only then found to be wrong, so the timer is no longer available.
A validated change passed every check and broke something downstream.

When not to use it

The good state is older than the archive holds, or the box was reinstalled. That is vyos-rb-restore-vyos-config, which starts by choosing a file and accepting its age.
The problem is a credential, not a configuration. Reverting restores the compromised secret: run vyos-rb-respond-compromised-credentials.
The change locked you out and the box is unreachable. Recover access first over out-of-band; vyos-bf-remote-commit-lockout walks that path.
Nothing has been committed yet. The candidate is dirty and the fix is discard, which needs no archive at all.
You are planning a change rather than undoing one - vyos-rb-safe-remote-change.

Blast radius

The revert commit is a full commit: validators, generators, renderers, and a push to FRR and the kernel. Everything the bad change touched moves again, and everything committed between the target entry and now moves with it.

If the reverted nodes include routing protocol configuration, adjacencies drop and re-form and the routers around you see the reconvergence. If they include the interface, VRF, SSH or firewall configuration, the revert can remove the session it is being typed into - which is the same risk the original change carried, in the opposite direction.

Step 0: which commit is the bad one?

Read-only / Saferead the archive; do not compute the offset

show configuration commit
show system config-management

Each archive entry carries a commit number, a timestamp, the user who committed it and the source address they committed from. That is enough to attribute every entry, and attribution is the actual work of this step: you are looking for the boundary between “working” and “not working”, not for the newest line.

Step 1: how did the bad commit end?

Read-only / Safethe commit log tells you which phase failed, if any

show log commit | last 10
show log frr | last 20

A commit is transactional with respect to the running configuration: a validator rejection changes nothing and leaves the candidate for you to fix. If the log shows that shape of failure, there is nothing to roll back - the running configuration was never modified.

The awkward case is a failure later in the sequence. The commit engine renders each affected subsystem and pushes it, so a failure partway through can leave one backend on the new configuration and another still on the old. That is the origin of “the router is in a state that matches nothing”, and it matters here because a rollback assumes the running state corresponds to a known configuration. When it does not, reverting is one option among several and the choice belongs to someone more senior than the timer allows.

Step 2: inspect the target before you load it

Read-only / Safecompare against an entry without touching the candidate

compare 2
compare 3

compare N shows the difference between the candidate and archive entry N. Nothing is loaded and nothing is committed; it is a read of what a revert to that entry would do. Use it to confirm the target actually contains the state you want and does not contain a surprise - and use it on the neighbouring entries too, because reading two candidates against each other is how you find the boundary you identified by timestamp in Step 0.

Read-only / Safecapture the broken state before changing it

# Substitute your own values before running:
ROUTER=edge-01.example.com
CASE=/srv/incident/CASE-1234

mkdir -p "$CASE"

ssh "vyos@$ROUTER" 'show configuration commands' > "$CASE/config-bad.txt"
ssh "vyos@$ROUTER" 'show configuration commit'   > "$CASE/commits.txt"
ssh "vyos@$ROUTER" 'show log commit | last 20'   > "$CASE/commitlog.txt"
ssh "vyos@$ROUTER" 'show ip route'               > "$CASE/routes-bad.txt"
ssh "vyos@$ROUTER" 'show ip bgp summary'         > "$CASE/bgp-bad.txt"

The revert is about to destroy the only complete copy of the configuration that caused the problem. Ten seconds of capture now is the difference between a post-incident review that can name the offending node and one that reconstructs it from memory.

Step 3: load it - and notice that nothing has happened yet

Configuration changerollback loads the candidate; the router is still broken

configure
rollback 2
compare

At this point the candidate holds the target entry and the running configuration holds the bad change. The router is behaving exactly as it was ten seconds ago. Nothing has been reverted.

Step 4: read the diff, because the entry is a snapshot of another moment

The archive entry was correct for the network as it existed when it was taken. If anything else has changed since - an interface added, an address moved, a peer onboarded - the revert reverts that too, and the compare output is where that becomes visible.

The rollback lesson gives the canonical example: an operator rolls back to an entry from before an unrelated interface change, the interface reverts along with the bad change, and routing breaks in a new way. The revert succeeded, the original problem is gone, and the router is still not working.

What the diff shows	What it means
Only the nodes the bad change touched	The clean case. Proceed
Nodes belonging to a later, unrelated commit	You are reverting somebody else work as well. Stop and ask them
An interface, address or VRF you did not expect	The network moved since the snapshot. Consider a targeted fix instead
Nothing at all	The candidate already matches that entry. The target is wrong, or the revert already happened

When the diff is wider than the problem, a targeted delete of the offending node is often the better change: it is smaller, it is reviewable, and it does not carry collateral.

Step 5: commit, and know which way the timer falls

Service impact possiblethe revert commit

compare
commit-confirm 10
exit

For any revert that touches the path you are connected over, commit-confirm is the standard. It is the pattern the commit-confirm lesson prescribes for remote changes and a revert is a remote change.

Step 6: verify, then keep it

Read-only / Safedependency order, outside configure mode

show interfaces
show ip route
show ip ospf neighbor
show ip bgp summary
show log commit | last 5

Verify with the command that showed the symptom first, not with a general health sweep. A revert that clears the original symptom but leaves a new one is a different outcome from a revert that worked, and only the specific check separates them.

Configuration changeconfirm, save, and re-read the listing

configure
confirm
save
exit

show configuration commit

The last command is not ceremony. The revert was a successful commit, so it wrote its own archive entry, and every index in the listing has moved. Any number worked out earlier in the incident is now stale, and reusing it is how a second revert lands somewhere nobody chose.

What a rollback does not undo

Restoring the configuration does not reverse what the configuration did

The archive holds configuration. These are not configuration:

An advertisement that propagated. Re-announcing a prefix does not undo the reconvergence, the traffic that moved onto another path, or any dampening a neighbour applied to your prefixes while it flapped.
Sessions dropped while the change was live. The revert restores the rule set and the peering configuration; the connections that were torn down during the bad window are gone and their clients have already noticed.
Anything performed outside the tree. An image installed, a certificate issued, a credential agreed with a peer. The node that asked for it reverts. The thing itself does not.
A two-party secret. A routing-protocol password or pre-shared secret was changed on both ends. Reverting your half alone drops the adjacency instead of restoring it.
Time. The archive keeps a bounded number of entries and pruning is irreversible. Past that horizon the tool is a restore from a file, and the file has an age of its own.

Classify the bad change against this list before starting. If any of it applies, the revert is one part of the recovery rather than the whole of it, and saying so up front is cheaper than discovering it halfway through.

Common patterns

Symptom	Likely cause	Resolution
Revert reported done, symptom unchanged	`rollback N` loaded the candidate and was never committed	`compare`, then `commit`, then `save`
Symptom unchanged after a committed revert	The target entry still contained the bad change	Re-read the listing and pick the entry by timestamp
Original problem gone, a new one appeared	The revert reached past an unrelated later commit	Read the diff line by line; consider a targeted fix instead
Revert reverted itself after ten minutes	`commit-confirm` expired, returning to the bad configuration	Re-apply and `confirm` inside the window
Bad change returned after a reboot	`confirm` ran, `save` did not	Re-apply and `save`; verify with an empty `diff`
Commit rejected during the revert	The entry is invalid against the current schema or state	Fix the node by hand, or `discard` and choose another entry
Bad change reappeared with no operator behind it	The repository or pipeline still holds it	Revert at the source as well as on the box
Second revert landed on the wrong entry	Indices shifted when the first revert committed	Re-read `show configuration commit` after every commit
Target entry no longer listed	Retention pruned it	This is now a restore from a file
Running state matches no archive entry	The bad commit failed partway through	Escalate; a revert assumes a known starting state

Escalation

Escalate when:

Reverting would undo a commit that belongs to somebody else.
The router is one half of a redundant pair.
The change being reverted is a security control.
The commit log shows the bad commit failed partway through.
An archive entry cannot be attributed to a known operator.
The change originated from an automation pipeline.
The target entry has been pruned.

Why this matters

A rollback is the one procedure people believe they can run without reading, because the command is short and its name is a promise. That is exactly why it fails in the two boring ways rather than in interesting ones.

rollback N loads. It does not apply. Every part of the experience suggests otherwise - the command succeeds, an archive filename is echoed, the prompt returns - and the only contradicting evidence is that the router is still broken, which during an incident reads as patience being required. The habit that fixes it is mechanical: rollback, compare, commit, save, and never stop at the first one.

Choosing the target is the other half, and it is a reading problem rather than a command problem. The archive is a list of moments, each correct for its own, and the operator picks which moment to return to. Under pressure the number gets guessed instead of read, and a guessed number either leaves the bad change in place or takes a colleague’s work down with it. Both outcomes look like the revert failed; neither is a failure of the revert.

And underneath both: a rollback is a change. It commits, it re-converges, it writes its own archive entry and shifts every index behind it. Treating it as an undo - as something outside the change process, exempt from diffs and timers and verification - is what turns one incident into two.