Lab: commit-confirm and Rollback

Objective

You are going to disable the network interface carrying your own SSH session, on purpose, and get it back without touching the console. Then you are going to answer two questions about your router that most operators never ask until the answer costs them an outage:

When the commit-confirm timer fires, does this image revert the configuration in place, or does it reboot the router?
When you type rollback 1, does this image load a candidate for you to review, or does it apply a revision and reboot?

Those are not rhetorical. They have different answers on different VyOS lineages, and the difference between “my change reverted” and “my router rebooted” is the difference between a quiet recovery and a second incident. The lab is built so that you establish both on a router you can afford to lose, which is the only responsible place to find out.

Architecture

One VyOS router and two ways to reach it. The whole lab is about the relationship between them.

        your workstation
              |
              |  SSH  <-- the "remote" path. This lab breaks it on purpose.
              |
        +-----------+
        |    r1     |  eth0   (management network, DHCP)
        +-----------+
              |
        hypervisor console  <-- out-of-band. Used to WATCH, not to rescue.

There is no second router, no peer and no traffic. The subject of this lab is the router’s relationship with its own operator.

The change used to cause the lockout is set interfaces ethernet eth0 disable. It is one line, its effect is total, and it is undone by removing one line — which makes it the cleanest possible instrument for studying a recovery mechanism. In production the equivalent is a firewall rule, an address change, a routing-policy edit or a management-VRF move; the recovery mechanism is identical and the reason to rehearse it is that you will be much less calm when it happens for real.

Requirements

One VyOS 1.5.x (circinus) virtual machine you are permitted to reboot and, if the lab goes badly, rebuild. Do not use a router anyone depends on.
An SSH session to the router over eth0. This is the session the lab destroys.
Working console access through your hypervisor, verified before you start. Not “probably available” — open it, log in, and leave it open. It is your evidence channel throughout and your rescue path if a step goes wrong.
Roughly 25 minutes of the estimate is spent waiting for timers. That is not padding; the waiting is the lab. A commit-confirm you cancel early teaches you nothing about what happens when you cannot.

Scenario

You have a change to make on a router in a facility four hours away. The change touches the path your session arrives on. Nobody is on site. The remote-hands ticket for a console cable has a four-hour SLA and the change window is two hours long.

You have two options, and only one of them is a plan:

Make the change and hope. If it is wrong, the router is unreachable until somebody drives to it.
Make the change with a deadline attached, so that a router which stops hearing from you assumes the worst and undoes the change by itself.

The second is commit-confirm, and it works because it inverts the default. Normally a change persists unless you undo it. Under commit-confirm a change is undone unless you actively keep it — which means that losing your session, losing your VPN, losing consciousness or simply being wrong all produce the same safe outcome.

This lab makes you use it once in anger, where the cost of it not working is that you click on a console window.

Tasks

Task 1: Establish both paths and the baseline

Console session. Open it. Log in. Run:

uptime -s
uptime

Write down the boot time. It is the discriminator for the two questions in the Objective: if the boot time changes, the router rebooted.

SSH session. In a separate window, SSH to the router over eth0 and confirm you are on the same box:

uptime -s
ip -br addr show eth0

Then capture the baseline, in the SSH session:

mkdir -p ~/confirm-lab

show configuration commands > ~/confirm-lab/pre-active.txt
sudo ls -1 /config/archive/ | wc -l | tee ~/confirm-lab/pre-archive-count.txt
uptime -s | tee ~/confirm-lab/pre-boot-time.txt

Finally — and this is the pre-check that makes everything after it meaningful — confirm the running and saved configurations agree:

configure
load /config/config.boot
compare
discard
exit

Empty output. If it is not empty, stop and resolve that first.

Why running-equals-saved is a commit-confirm pre-check

commit-confirm’s revert target is the configuration that was running before your commit. If somebody has already committed a change and not saved it, then the state you will be reverted to is that unsaved state — which is not what the saved configuration says, and not what the last change ticket described.

Worse, if the auto-revert on your image is implemented as a reboot, you do not get reverted to the previous running configuration at all; you get whatever is in /config/config.boot, and every unsaved change on the box disappears along with yours.

Two different revert targets, depending on a mechanism you have not yet established. That is precisely why Task 3 exists, and why “running equals saved” belongs in the pre-checks of every remote change.

Task 2: Rehearse the mechanism on something harmless

Never let the first time you use a safety device be the time you need it. Use a dummy interface, which cannot affect anything:

configure
set interfaces dummy dum9 address '198.51.100.99/32'
set interfaces dummy dum9 description 'COMMIT-CONFIRM-REHEARSAL'
compare
commit-confirm 2
exit

Service impact possibler1 (configure mode)— Applies the candidate with a deadline. Not confirming within 2 minutes triggers the auto-revert.

$ commit-confirm 2

Commit confirm will be automatically reverted in 2 minutes unless you enter 'confirm'.

Illustrative output

Confirm the change is live right now:

ip -br addr show dum9

Then do nothing. Watch both sessions. Note the wall-clock time. In a minute or two the change will be taken away from you.

Doing nothing is a decision, and it needs an owner and an end time

“Let the timer fire” is not the absence of a plan; it is the plan, and it is frequently the correct one. If your verification is ambiguous — the session came back but the metrics look odd, or you cannot reach the person who can tell you whether the new behaviour is right — the safe action is to withhold confirm and let the router return itself to a known state.

What makes it a decision rather than a drift is that it is stated. In a change window that means saying, in the channel, before the window opens: “commit-confirm 10, I confirm only if all three checks pass, otherwise it reverts at 22:14 and I own the follow-up.” Now everyone knows what happens at 22:14 and who owns it. An unstated hold is just an operator who wandered off.

Task 3: Find out what the auto-revert actually does to your router

When the timer fires, gather evidence in both sessions.

In the SSH session (if it is still alive — it should be, because nothing about this change touched your path):

ip -br addr show | grep dum9
show configuration commands | match dum9
uptime -s

In the console session:

uptime -s
uptime

Now answer the question, from evidence:

The dum9 interface should be gone. The auto-revert happened. Good — that is the mechanism working.
Compare uptime -s against ~/confirm-lab/pre-boot-time.txt.
- Unchanged — the router reverted the configuration in place. Your sessions, your conntrack table, your routing adjacencies and your counters all survived.
- Changed — the auto-revert was implemented by rebooting the router. Everything on the box restarted: every BGP session, every IPsec tunnel, every established flow.

Record the answer in ~/confirm-lab/ with the evidence. It is the single most important operational fact about this image.

Why this question does not have one answer, and why it matters so much

The VyOS configuration-management verbs have a long history and their implementations have not been stable across the 1.2 / 1.3 / 1.4 / 1.5 lineage. The course’s own material reflects that: several lessons and runbooks describe the auto-revert and rollback as in-place operations on the candidate and running configurations, while another runbook warns that the documented behaviour of rollback is to apply a revision and reboot the system.

Both descriptions are in this course because both have been true of some VyOS release. That is uncomfortable to read and it is more honest than picking one and hoping your image agrees.

The operational consequence is large. An in-place revert is a targeted undo. A reboot is a whole-box outage that happens to also undo your change — and on a router carrying eBGP sessions, an IPsec estate or a VRRP partnership, the reboot is a far bigger event than the misconfiguration it is repairing. If you cannot state which one your fleet does, you cannot size the blast radius of your own safety net, and a safety net you cannot size is not one you can honestly recommend to a change board.

Establishing it takes one dummy interface and two minutes on a disposable router. Do it once per image version, write it in the runbook, and never guess again.

Task 4: Confirm, and prove that confirm is not save

Now the other branch. Same harmless change, but this time keep it:

configure
set interfaces dummy dum9 address '198.51.100.99/32'
set interfaces dummy dum9 description 'COMMIT-CONFIRM-REHEARSAL'
compare
commit-confirm 5
exit

Verify from outside configure mode, the way you would verify a real change — by checking the effect rather than the configuration:

ip -br addr show dum9

Then cancel the deadline:

configure
confirm

Wait past the original five minutes and check that dum9 is still there. It is: the timer is cancelled and the change is permanent.

Except it is not, and this is the trap. Ask whether the saved configuration knows about it:

load /config/config.boot
compare
discard
exit

compare is not empty. It shows dum9 being removed — because the candidate now holds the saved configuration, which has never heard of your change. You confirmed it, and a reboot will still destroy it.

Complete the sequence properly:

configure
save
exit

Task 5: Lock yourself out and be rescued

This is the one. Read the whole task before you run any of it.

Check your console session is alive and logged in. Note the wall-clock time. Then, in the SSH session:

configure
set interfaces ethernet eth0 disable
compare

Read the compare output. It should show exactly one addition, disable under interfaces ethernet eth0, and nothing else. This is the last moment at which reviewing the diff is free.

Service impact possibler1 (configure mode, over SSH)— Disables the interface carrying this session. The session dies at the moment this returns.

$ commit-confirm 3

Commit confirm will be automatically reverted in 3 minutes unless you enter 'confirm'.

Illustrative output

Your SSH session freezes and then dies. That is correct. Do not try to reconnect, do not touch the console, and do not do anything clever. You are now living inside the three minutes, which is the entire experience this lab exists to give you.

Watch from the console. The console still works because it is not the network. Confirm the router is doing what you told it to:

ip -br link show eth0

The interface is down and the address is gone. Keep watching.

When the timer fires, the change is taken back. Check from the console:

ip -br addr show eth0
uptime -s

Then reconnect over SSH from your workstation and prove it:

show configuration commands | match 'ethernet eth0'
uptime -s

Write the timeline into your journal: the time you committed, the time your session died, the time the interface came back, and the boot time before and after. Four timestamps. That is your evidence that the router rescued itself.

What commit-confirm does not protect you against

It is a safety net with specific holes, and knowing them is the difference between using it and trusting it:

A router that crashes or loses power during the window. The timer lives in the running system. If the system stops, the timer stops with it — and if the auto-revert is implemented in place, the committed change is still in the running configuration when the box comes back only if it was also saved. Either way, a crash during the window leaves you somewhere you did not plan for.
A change that breaks the revert itself. The revert has to be executed by the router. A change that destabilises the configuration daemon, fills the disk, or wedges the box does not get politely undone three minutes later.
Two operators with overlapping timers. The commit-confirm state is per-router, not per-person. A second operator’s commit-confirm and confirm interact with yours in ways neither of you modelled. One named operator per change window; this is why that rule exists.
A timeout shorter than your verification. A BGP session that takes four minutes to converge, verified under commit-confirm 3, reverts while you are still watching it come up — and now you are debugging a revert you caused, in the middle of a change window. Size the timeout to the slowest check you intend to run, then add margin.
Anything you did with ip rather than set. Runtime state you changed outside the configuration tree is invisible to the revert. Lab 01 made this concrete with a neighbour entry that survived everything.

Task 6: Read the archive and pick a revert target

Every successful commit — including the ones the auto-revert performed — has written an entry. Look at what your session produced:

sudo ls -1t /config/archive/ | head -n 6
sudo ls -1 /config/archive/ | wc -l

Compare against ~/confirm-lab/pre-archive-count.txt. The count has grown by more than the number of changes you intended to make, because the reverts are commits too. That is worth internalising: the archive records what the router did, not what you meant.

VyOS also exposes the archive through the CLI. The course uses show configuration commit for this; if your image does not accept it, tab-complete from show and record the verb your image uses. The files in /config/archive/ are the authority either way, and they carry the same information in a form no version can rename out from under you:

NEWEST=$(sudo ls -1t /config/archive/ | head -n 1)

sudo head -n 3 "/config/archive/$NEWEST"

Now identify the entry you want as a revert target. In this lab it is the state before Task 4 added dum9:

sudo grep -L dum9 /config/archive/* | tail -n 3

grep -L lists the files that do not contain the pattern — the archive entries from before dum9 existed. Take the newest of them; that is your target.

Task 7: Revert with commands whose behaviour you have verified

You now know exactly what your image’s auto-revert does, because Task 3 told you. You do not yet know what rollback 1 does, and the two behaviours differ by a whole-router outage.

Two options, and they are not equivalent:

Option A — establish rollback’s behaviour, on this disposable router, right now. Note the boot time, run it, and see:

uptime -s

configure
rollback 1

Then immediately check whether you still have a session and what uptime -s says. Record the result next to your Task 3 answer. If the router rebooted, you have learned something important and paid nothing for it. If it loaded a candidate, run compare and read it before doing anything else — a loaded candidate that you commit without reading reverts everything that changed since that entry, not only the thing you regret.

Option B — use the primitives that behave the same on every image. load takes an explicit path and does exactly one thing: it replaces the candidate. Nothing is applied until you commit.

configure
load /config/archive/config.20260819-104501.boot
compare

Substitute the filename you selected in Task 6. Read the compare output in full. It is the complete list of what committing will change, and on a router that has moved on since that entry it will contain things you did not expect.

Then apply the revert the same way you applied the change — with a deadline:

commit-confirm 5
exit

Verify, then confirm and save:

configure
confirm
save
exit

Reverting is a change, and it deserves the same discipline

The most dangerous moment in an incident is the revert, because everyone has stopped treating it as a change. It is a change. It has a diff, it has a blast radius, it can be wrong, and it can be wrong in a way that is harder to diagnose than the original fault because now two things have moved.

So: read the compare, apply it with commit-confirm, verify it, then confirm and save. The revert of a revert is the worst place in the whole tree to be improvising.

A specific trap: rolling back to an entry from before an unrelated change takes that change with it. On a busy router, “roll back to before the problem started” can silently undo a colleague’s work from earlier the same day. The compare output is where that shows up, and it is the only place it shows up.

Validation

Your journal records, with evidence, whether the commit-confirm timer on this image reverts in place or reboots. The evidence is uptime -s before and after, from the console session.
The Task 5 timeline has four timestamps: commit, session death, access restored, and the boot time on both sides of the event.
You did not touch the console to recover in Task 5. If you did, the lab did not demonstrate what it was supposed to — reset and run it again with a longer timeout so you are less tempted.
After Task 4, load /config/config.boot followed by compare showed the confirmed change as missing from the saved configuration. You can state in one sentence why confirm did not persist it.
sudo ls -1 /config/archive/ | wc -l is larger than ~/confirm-lab/pre-archive-count.txt by more than the number of changes you intended, and you can explain the extra entries.
You can name at least three things commit-confirm does not protect against, without looking them up.

Expected Outcome

eth0 is enabled, addressed as it was at the start, and carrying your SSH session.
dum9 is gone, and the running and saved configurations agree: load /config/config.boot then compare prints nothing.
~/confirm-lab/ contains the pre-lab captures plus your recorded answers about the auto-revert and rollback behaviour on this image.
/config/archive/ has several new entries, including the ones written by the reverts. That is correct and permanent.
The console session is still open, and you now regard that as normal rather than as paranoia.

Troubleshooting

The SSH session died in Task 5 and never came back. Use the console. configure, delete interfaces ethernet eth0 disable, commit, exit. Then work out which of two things happened: either the timer did not fire, or it fired and reverted to a configuration in which eth0 was already broken. Check ~/confirm-lab/pre-active.txt for what eth0 looked like at the start. This is exactly why the running-equals-saved pre-check in Task 1 is not optional.

commit-confirm 2 is rejected. Check you are in configure mode and that a previous commit-confirm is not still pending. If one is, confirm it or wait for it to expire; overlapping timers are the failure mode the callout in Task 5 warned about, and here it is.

The timer fired but dum9 is still present. You confirmed it without meaning to — confirm may have been in your shell history from Task 4 and recalled with an arrow key. Delete dum9 explicitly and re-run Task 2 with a clean history.

uptime -s is not available. Use uptime and read the uptime figure, or show version, which reports the same thing. Any measurement that distinguishes “the router has been up for two hours” from “the router has been up for forty seconds” answers the question.

The console shows a login prompt when you go back to it. The session timed out while you waited. Log back in — the console works regardless of network state, which is the entire reason it is your rescue path. Verify it is alive before Task 5, not during.

load reports that the file does not exist. The archive filename must be exact, and archive files are readable only with elevated privileges. Re-run the ls -1t from Task 6 and copy the filename rather than typing it.

Cleanup

Restore the router to its pre-lab state and leave it consistent:

configure
delete interfaces dummy dum9
delete interfaces ethernet eth0 disable
compare
commit
save
exit

Both delete lines are no-ops if the state has already been reverted, and running them anyway is the right instinct — you are asserting the end state rather than reasoning about which path you took to get here. Read the compare output before committing; it should show removals only.

Confirm you are back:

show configuration commands > ~/confirm-lab/post-active.txt

diff -u ~/confirm-lab/pre-active.txt ~/confirm-lab/post-active.txt \
  && echo 'ACTIVE CONFIGURATION RESTORED'

And confirm running and saved agree, which is how every change window ends:

configure
load /config/config.boot
compare
discard
exit

Empty output. Keep ~/confirm-lab/ — the recorded answers about your image’s auto-revert and rollback behaviour are the deliverable, and they belong in your team’s runbook rather than in a lab directory.

The archive entries this lab created stay. They are a log, and a log you can tidy is not evidence.

Production notes

Every change to a router you reach over the network gets a deadline. Not the risky ones — every one. The changes that lock people out are overwhelmingly the ones nobody classified as risky, because the risky ones get scheduled, reviewed and rehearsed. commit-confirm costs one word.

Size the timeout to your slowest verification, then add margin. BGP convergence, IPsec re-keying and IGP adjacency formation all take longer than operators expect at 02:00. A timer that fires while you are still verifying converts a successful change into a self-inflicted revert.

Say the deadline out loud before you commit. “commit-confirm 10, it reverts at 22:14 unless I confirm, I own the follow-up either way.” Now the hold is a decision with an owner and an end time rather than a colleague who stopped typing.

Know what your fleet’s auto-revert does, per image version, and write it in the runbook. If it reboots, then commit-confirm on a router carrying production eBGP sessions is a much larger commitment than it looks, and the change window has to be sized for a reboot rather than for a config edit. If it reverts in place, say so and cite the test. This is a one-off piece of homework that pays for itself the first time somebody asks.

Console access is not optional, it is the precondition. A remote change without a verified out-of-band path is not a change with a safety net; it is a change with a hope. Verify the console before the window, not during it.

What You Learned

commit-confirm inverts the default, and that is the whole idea. A change under a timer is undone unless you keep it, so losing your session and being wrong produce the same safe outcome. You proved it by destroying your own access and getting it back without the console.
You established what the auto-revert does to your router. Not what a document says, not what a lesson says — what uptime -s said before and after. In-place revert and reboot are different-sized events and you can now name which one you have.
confirm and save are two decisions. You watched a confirmed change be absent from the saved configuration, which is a change already scheduled for deletion at an unknown future reboot.
Doing nothing is a decision that needs an owner and an end time. Letting the timer fire is frequently correct and always something to say out loud beforehand.
The archive records what the router did, not what you meant. Your reverts wrote entries too, and the count grew faster than your change list.
Prefer verbs whose behaviour you have verified. load plus compare plus commit-confirm does exactly one thing on every image, which is why it is the pattern to reach for when the router is four hours away and you cannot afford a surprise.

Objectives

Prerequisites