high riskservice affecting~45 min

Runbook: Safe Remote Change

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The ticket names four things, not one: the intent, the blast radius, the rollback plan, and the verification commands together with the output that counts as a pass. A ticket that says "add the peer" and stops has not defined what success looks like, so nothing you do afterwards can be called verification
· The diff exists as a written list of set and delete lines and a NAMED second operator has read it against the ticket. Peer review happens before the session opens, not while a timer is running - a reviewer reading a diff under a five-minute clock is a witness, not a reviewer
· Out-of-band access has been USED, not assumed. Log in over the console, serial, IPMI or hypervisor console now and leave the window open. An OOB path that has never been tested is a line in a spreadsheet, and the moment you need it is the moment you find out the serial console is at the wrong baud rate
· You have decided whether the change touches your own access path - the interface it arrives on, the route that reaches it, the firewall input chain, the SSH service, the management VRF. If it does, commit-confirm is mandatory rather than advisable, and the timer must cover the whole verification
· The archive baseline is recorded before you start: show configuration commit names the newest entry, its user, its source address and its timestamp. If somebody else committed in the last hour, find out what they did before you commit on top of it
· Running and saved configuration already agree: diff /config/config.boot <(show configuration commands) is empty. If they differ, somebody left an unsaved change live, your rollback target is not what you think it is, and a reboot during your window would produce a state neither of you predicted
· A labelled snapshot exists outside the archive: save /config/backups/pre-TICKET.boot. The archive is pruned at system config-management commit-revisions (20 by default) and a busy router can churn through twenty entries in an afternoon; a file under /config/backups cannot be pruned out from under you
· The pre-change operational evidence has been captured OFF the router - the configuration dump plus the specific operational views your verification will compare against. Evidence stored only on the box is evidence you lose in exactly the failure mode you kept it for
· Anything the change is known NOT to be reversible by rollback is written down and accepted by the change owner: routing sessions that will reset, tunnels that will re-key, conntrack state that a firewall reload will flush

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the rollback target: show configuration commit. Write down the entry number, the timestamp and the user. This is the number you will type under pressure, and reading it now costs ten seconds while reading it later costs the outage.
2Confirm running and saved agree, then take the labelled snapshot: diff /config/config.boot <(show configuration commands) followed by save /config/backups/pre-TICKET.boot. A non-empty diff here stops the change until you know whose uncommitted work you are standing on.
3Capture the pre-change operational evidence to a file off the router - the configuration, plus the specific views the verification will compare against (show interfaces, show ip route, show bgp summary, show vrrp, whichever apply). You cannot prove afterwards that you did not break something you never looked at.
4Enter configure, apply the reviewed set and delete lines, and run compare. Read the output line by line against the ticket. compare is the only moment the router tells you what you are actually about to do, as opposed to what you meant to do.
5Size the timer to the VERIFICATION, not to the commit. A BGP session that must converge, an IPsec tunnel that must re-key or an OSPF adjacency that must reform needs minutes; add margin for a page or a colleague interrupting you. Five is the reflex and is frequently wrong.
6Apply with commit-confirm N for anything on the access path or anything service-affecting. Plain commit is only correct when you already have working console access and are prepared to use it.
7Leave configure mode with exit before verifying anything. Verification runs from the operational shell, and staying in configure mode holds the configuration lock against every other operator and every automation run.
8Run the verification commands the ticket named and compare against the expected output it named. Then read show log commit | last 10: a commit can return cleanly while a generator or FRR rejected part of the change, and the exit status will not tell you.
9Open a SECOND session to the router while the timer runs. An established SSH connection survives changes that would prevent a new one from being made, so the session you are already in cannot answer the question "can I still get in".
10Verify from where the users are, not only from the router. A route present in show ip route is a claim about this box; traffic arriving at the far end is the thing the change was for.
11If any verification fails or is ambiguous, do nothing. Let the timer fire. Doing nothing is the fastest and most reliable recovery available to you, and it requires no decision from an operator who may no longer be able to reach the box.
12If every verification passes, re-enter configure and run confirm, then run save. These are two separate actions: confirm cancels the rollback timer, save writes /config/config.boot. A confirmed change that was never saved dies at the next reboot, weeks later, with nobody connecting the two events.
13Prove the persistence rather than assuming it: diff /config/config.boot <(show configuration commands) must be empty, and show commit-confirm must show no pending timer.
14Close the ticket with the diff, the timer used, the verification output next to the expected output, the archive entry number the change created, and the name of the operator who verified. The next person to touch this router reads that, not your memory of it.

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every verification command named in the ticket has been run from the operational shell and its actual output is recorded in the ticket beside the expected output. "It looked fine" is not a verification and cannot be reviewed later
✓show log commit | last 10 shows the commit with no validator or generator failure, and the FRR log carries no error at the commit timestamp. A VyOS commit can succeed while bgpd rejects the configuration it was handed
✓A NEW session to the router succeeds - not the one already open. This is the only check that proves the access path survived, and it is the one people skip because the session they are typing in still works
✓The service the change was for works from where its users are, measured at the far end rather than at the router
✓Everything the change could plausibly have disturbed and was not aimed at is in the state the pre-change capture recorded: routing adjacencies, VRRP roles, tunnel status, interface counters. Compare against the capture, not against memory
✓show commit-confirm reports no pending timer. An operator who believes they confirmed and did not has a live change with minutes left on it
✓diff /config/config.boot <(show configuration commands) is empty, so the change survives a reboot
✓show configuration commit shows a new archive entry carrying your user and source address at the expected timestamp

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶While a commit-confirm timer is running, the rollback is to do nothing. The box reverts itself without needing a decision from you, which matters most in the case where you can no longer reach it
↶To revert a change already confirmed: configure, then compare N to read the diff, then rollback N, then compare again, then commit-confirm N, verify, confirm, save. The revert is itself a remote change and deserves the same net
↶Run compare N BEFORE rollback N. The diff covers everything committed since that archive entry, which on a shared router is regularly more than your change - reverting a colleague along with yourself is a common way to turn one incident into two
↶rollback N only loads the archive entry into the candidate. Without commit nothing has been reverted, and an operator who walks away after rollback has left the bad change live while believing it is gone
↶If the archive entry has been pruned, use the labelled snapshot instead: load /config/backups/pre-TICKET.boot, then compare, then commit-confirm N. This is why the snapshot was taken
↶If the router is unreachable from both the data and management paths, stop issuing remote commands and go to the console. A second remote attempt against a box that has just proved it can disappear spends the recovery window on a guess
↶POINT OF NO RETURN: a rollback restores configuration, not state. A BGP or OSPF session that reset re-converges on its own timers; an IPsec tunnel re-keys; conntrack entries flushed by a firewall reload mean established flows were dropped and their applications have to reconnect. None of that is undone by putting the old lines back
↶POINT OF NO RETURN: what other systems already did with the change stays done. Prefixes advertised to a peer are in their table and in their neighbours tables; a DHCP address handed out is in a client; a log line shipped is shipped. Withdrawing the configuration does not withdraw the consequences

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform or site team, and stop remote attempts, the moment the router is unreachable from both the data and the management path. That is now a physical-access problem, and every further remote retry is time the outage runs
· Escalate to network engineering when the router is one half of a redundant pair. A change on one member changes which member is preferred, so the blast radius includes the failover behaviour of the pair and not only the behaviour of this box
· Escalate to the change owner when the verification is ambiguous rather than failing. Ambiguous is a fail: confirm nothing you cannot show working, and never confirm to avoid the embarrassment of a rollback
· Escalate to the peer, provider or application NOC when the router-side verification passes and the far-end verification does not. The change may be correct and the fault theirs, and the evidence for that is the pair of results
· Escalate to security whenever the change touches credentials, key material, the SSH service, the HTTP API surface or the management VRF, regardless of how routine the change looks
· Escalate rather than re-attempt after the same change has failed twice. Two failures almost always means the pre-change model of the router is wrong, and a third attempt built on the same model makes the evidence worse rather than better

A remote change is not a local change done from further away. It is a different operation, because on a router the thing being changed is frequently the thing carrying the session you are typing into. Nothing else in operations has quite this property: a database administrator who breaks a query does not lose the ability to type the next one.

That single fact produces the whole discipline. You cannot rely on being able to fix a mistake, so the recovery has to be arranged before the mistake is possible — and it has to be a recovery that needs no action from you, because the failure mode you are guarding against is precisely the one where you cannot act.

VyOS gives you that for free in commit-confirm, and most operators know it. What this runbook is actually for is the three places where a change made with commit-confirm still goes wrong, all of which look like success at the time.

When to use this runbook

Any configuration change to a production VyOS router where the operator does not have hands on the box, or has hands but no console. It is the wrapper around the specific change runbooks: the NAT rule, the BGP neighbour, the firewall policy, the VRF. Those tell you what to type. This one tells you how to survive typing it.

When not to use it

You are already in an incident and the router is misbehaving. Change control is for planned work; use the runbook for the symptom.
The change is a revert of a bad commit and the archive holds the good state. That is vyos-rb-rollback-bad-config, which is faster and has a smaller decision surface.
The change is a system image upgrade. That is vyos-rb-perform-production-router-upgrade; an upgrade crosses a reboot, and everything below about commit-confirm stops being true across a reboot.

Blast radius

The honest answer is: whatever the change touches, plus your ability to make the next change. Those are different sets and the second one is the one that turns a small mistake into a site visit.

Before you commit, decide explicitly which of these the change is:

Off the access path. A firewall rule for a segment you do not manage the router from, a static route to a network you are not connected through. A bad commit here is recoverable at your leisure.

On the access path. Anything touching the interface your session arrives on, the route back to your workstation, the firewall input chain, the SSH service, or the management VRF. A bad commit here removes your ability to fix it. commit-confirm is not a nice-to-have for these; it is the entire recovery plan.

Neither, but shared. A router that is one member of a redundant pair. The change may be safe on this box and change which member wins, which is a failover decision wearing the clothes of a configuration change.

Step 1: record the state you will be asked to return to

Read-only / Safethe number you will type under pressure

show configuration commit
diff /config/config.boot <(show configuration commands)

show configuration commit prints one row per archive entry: the entry number, the timestamp, the user, the source address the commit came from, and the filename under /config/archive/. Two things come out of it that people skip and later wish they had not.

The first is the rollback target. You will need an entry number while the router is misbehaving and your judgement is worst; reading it now is free — and reading it now also means you read it against a calm compare, rather than counting rows under pressure.

The second is who else has been here. An entry from another operator twelve hours ago is a change you are about to commit on top of, and you do not know whether it was finished. Ask before you add to it.

The diff is the quieter of the two checks and the more important. If the running configuration and /config/config.boot already disagree, somebody left an uncommitted-to-disk change live. Your rollback target is not what you think, and a reboot during your window produces a configuration nobody predicted.

Configuration changea snapshot the archive cannot prune

configure
save /config/backups/pre-NET-1842.boot
exit

The archive keeps system config-management commit-revisions entries, twenty by default. A router under active work — an automation run, a colleague’s change, your own iteration — can push your baseline out of that window in an afternoon. A labelled file under /config/backups/ is not subject to that.

Step 2: read the diff, not your intent

Read-only / Safewhat the router thinks you asked for

configure
set protocols bgp neighbor 203.0.113.9 remote-as 65100
set protocols bgp neighbor 203.0.113.9 description 'ISP-B transit, NET-1842'
compare

compare is the only moment the router will tell you what you are actually about to do rather than what you meant to do. Read it against the ticket line by line, out loud if there is somebody to read it to.

The failure this catches is not usually a typo in a value — validators catch most of those. It is scope: a delete one level higher in the tree than intended, which removes a subtree the operator never looked at, and which compare prints in full while the operator reads only the line they expected to see.

Peer review belongs here and, better, before here. A reviewer reading a diff while a five-minute timer runs is a witness rather than a reviewer.

Step 3: size the timer to the verification

Service impact possiblethe net, sized on purpose

commit-confirm 15
exit

The timer must cover everything between the commit and the confirm: the protocol convergence, the far-end check, the phone call to whoever is testing, and the interruption that will arrive halfway through. Five minutes is the reflex and it is frequently too short — a BGP session that has to establish and exchange a table, an IPsec tunnel that has to re-key, an OSPF adjacency that has to reform through its own timers, all comfortably outrun it.

The cost of overrunning the timer is not just the wasted attempt. The change is reverted while you are in the middle of validating it, which means the evidence you were collecting now describes two different configurations, and the natural next move — re-apply and confirm quickly this time — is exactly the move that skips the verification.

The cost of an over-long timer is smaller and more honest: you sit under a bad change for longer before the automatic revert. Prefer that error.

Step 4: verify from outside, in a new session

Leave configure mode first. The lock you are holding blocks every other operator and every automation run against this router, and verification does not need it.

Read-only / Safethe commit exit status is not the verification

show log commit | last 10
show bgp summary
show ip route 198.51.100.0/24

show log commit is here because a commit can succeed at the VyOS layer while the backend rejects what it was handed. The candidate passes the validators, the generators render the FRR configuration, FRR is told to reload — and bgpd refuses part of it. VyOS reports a successful commit; the routing daemon disagrees; nothing in your shell prompt tells you.

Then the check that most operators skip:

Read-only / Safethe session you are in cannot answer this

# Run from your workstation, not from the router:
ROUTER=192.0.2.1

ssh "operator@$ROUTER" 'show configuration commit | head -3'

An established SSH connection is an existing conntrack entry and an existing TCP session. It survives firewall changes, service restarts and policy changes that would prevent a new connection from ever being made. The session you are typing in is therefore the one piece of evidence that cannot tell you whether you can still get in.

Open a new one. If it fails, do nothing further — the timer is your recovery and you have just proved you need it.

Finally, verify from where the users are. A prefix in show ip route is a claim this router is making about itself. Traffic arriving at the far end is the thing the change was for, and the two disagree often enough to be worth the extra minute.

Step 5: confirm and save are two different actions

Configuration changeboth, in this order, every time

configure
confirm
save
exit

confirm cancels the rollback timer. save writes the running configuration to /config/config.boot. Neither does the other’s job.

A change that was confirmed and never saved is live, correct, verified — and gone at the next reboot. That reboot is typically weeks away, performed by somebody else for an unrelated reason, and the resulting regression is attributed to whatever they were doing rather than to the change that was never persisted. It is one of the hardest defects in this course to trace backwards, and it is caused by two keystrokes.

Prove it rather than assuming it:

Read-only / Safepersistence, demonstrated

show commit-confirm
diff /config/config.boot <(show configuration commands)

An empty diff means running and saved agree. No pending timer means you really did confirm — an operator who believes they confirmed and did not is holding a correct change with minutes left on its life.

What a rollback does not give back

This is the part of the plan people write as one line and discover as several.

rollback restores configuration. It does not restore state, and a router is mostly state.

The change did this	The rollback restores	What stays broken
Reset a BGP session	The neighbour stanza	The session re-establishes on its own timers, and the peer’s view of you was withdrawn and re-advertised in between
Reloaded the firewall	The rule set	Conntrack entries flushed with it; every established flow was dropped and its application has to reconnect
Changed an IPsec proposal	The proposal	The tunnel re-keyed; traffic was down for the re-key and is not replayed
Bounced an interface	The interface configuration	Anything that timed out during the bounce — calls, backups, replication windows — is gone, not queued
Advertised a prefix	Your outbound policy	The prefix reached your peers and their neighbours; withdrawal propagates at Internet speed, not at yours
Handed out a DHCP address	The pool	The client holds the lease for its full duration

None of that is an argument against rolling back. It is an argument for writing the irreversible parts into the ticket before the change, so that the person deciding whether to proceed is deciding with them in view rather than discovering them during the recovery.

Common patterns

Symptom	Likely cause	Where to look
Change worked, vanished weeks later	`confirm` without `save`	`diff /config/config.boot <(show configuration commands)` at the time
Change vanished after five minutes	Timer sized to the commit, not the verification	`show log commit`, the auto-rollback entry
Commit succeeded, protocol did not	Backend rejected part of the render	The last ten commit-log entries, then the FRR log
Your session works, nobody else can connect	Access-path change plus an existing conntrack entry	A new SSH session from the workstation
Rollback reverted more than your change	`rollback N` without `compare N` first	`compare N` against the entry, `show configuration commit`
Rollback target missing	Archive pruned past your baseline	`system config-management commit-revisions`, `/config/backups/`
Router never came back, timer expired	Box died rather than the change being wrong	Console. The timer did not survive the crash
Two operators, overlapping windows	Shared configuration lock and shared archive	`show configuration commit` for the other user’s entries

Escalation

Escalate when:

The router is unreachable from both data and management paths — stop remote attempts and get to the console.
The router is one half of a redundant pair and the change moved the preference between them.
The verification is ambiguous. Ambiguous is a fail.
The router-side verification passes and the far-end one does not.
The change touches credentials, key material, SSH, the HTTP API or the management VRF.
The same change has failed twice.

Why this matters

Every part of this runbook exists because a specific, well-documented safety net has a hole in it, and the hole is always in the same place: the moment the operator believes the work is finished.

commit-confirm is the best mechanism of its kind on any router platform, and it has three. The timer lives in memory, so a crash keeps the change it was meant to revert. The timer is sized by the operator, so it is routinely sized to the commit rather than to the verification. And confirm reads like “finished” while save is the thing that actually makes it permanent, so the most careful operator in the team can do everything right and still lose the change to a reboot two months later.

None of these are exotic. They are all in the lesson material, they are all in the vendor documentation, and they all keep happening — because each one fails at the moment attention is lowest, which is immediately after a change that appeared to work.

The discipline that fixes it is not sophistication. It is refusing to treat “the command returned” as evidence. Read the commit log rather than the exit status. Open a new session rather than trusting the one you are in. Diff the saved file rather than remembering that you saved it. Each of those is under a minute, each one catches a failure that is otherwise invisible for weeks, and none of them require knowing anything at all about the change you just made.