Remote Commit Locks Administrator Out · Break/Fix · VyOS

Reported symptoms

On Tuesday an engineer closed an audit finding across the whole border fleet. The finding was that management services accepted connections from any source; the change added an explicit source list to the SSH and HTTPS API accept rules in each router’s input filter. Fourteen routers, one window, commit-confirm on every one, all fourteen confirmed and saved.

On Thursday morning the on-call engineer needs to look at edge3 during an unrelated partner circuit problem and cannot get in.

The picture that greets the bridge call is contradictory in every direction:

SSH to edge3, edge4 and edge5 from the EU-South jump host hangs and eventually times out. It is not refused — there is no immediate error, just silence.
All three routers answer ICMP from the same jump host. Their BGP sessions are Established. Traffic through the region is completely unaffected, and the partner circuit problem is real and now unfixable.
Monitoring is green for all three, including the management-plane checks.
The other eleven routers, which got the identical change in the same window, are reachable normally.

Three routers, one region, everything else healthy. The first hour goes to the transit provider, because that is what a regional problem looks like.

Two more facts arrive and make it worse rather than better. The engineer who made the change is certain they verified it, and their session log backs them up: eleven unbroken minutes of connectivity after the commit, then confirm, then save. And the out-of-band runbook names a console server for the site whose hostname no longer resolves.

Somebody requests a remote power cycle. It is performed. Nothing changes.

Evidence provided

Read-only / Safetimed out, not refused — nothing sent a reset

ops@jump-eu-south:~$ ssh -v edge3.eu-south.example.com

OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.11
debug1: Connecting to edge3.eu-south.example.com [203.0.113.9] port 22.
ssh: connect to host edge3.eu-south.example.com port 22: Connection timed out

Illustrative output

Read-only / Safethe router is up and the path to it works

ops@jump-eu-south:~$ ping -c 3 edge3.eu-south.example.com

PING edge3.eu-south.example.com (203.0.113.9) 56(84) bytes of data.
64 bytes from 203.0.113.9: icmp_seq=1 ttl=61 time=8.42 ms
64 bytes from 203.0.113.9: icmp_seq=2 ttl=61 time=8.39 ms
64 bytes from 203.0.113.9: icmp_seq=3 ttl=61 time=8.44 ms

--- edge3.eu-south.example.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

Illustrative output

Read-only / Safethe same router, the same moment, from a different source address

ops@collector01:~$ ssh vyos@edge3.eu-south.example.com 'show version | head -2'

Version:          VyOS 1.5-rolling
Release train:    circinus

Illustrative output

Read-only / Saferule 20 is counting, and so is the default drop

vyos@edge3:~$ show firewall ipv4 input filter

Rule     Action   Protocol  Source              Packets    Bytes
10       accept   all       0.0.0.0/0           4812390    1204M     state established
11       accept   all       0.0.0.0/0             21744     3021K     state related
15       drop     all       0.0.0.0/0                 12      768     state invalid
20       accept   tcp       198.51.100.0/24        1904     221K     dport 22
25       accept   tcp       198.51.100.0/24         840      96K     dport 443
30       accept   icmp      0.0.0.0/0             112044      9M
default  drop                                      36219     2174K

Illustrative output

Read-only / Safethe change, exactly as designed and reviewed

vyos@edge3:~$ show configuration commands | match 'input filter rule 2'

set firewall ipv4 input filter rule 20 action 'accept'
set firewall ipv4 input filter rule 20 protocol 'tcp'
set firewall ipv4 input filter rule 20 destination port '22'
set firewall ipv4 input filter rule 20 source address '198.51.100.0/24'
set firewall ipv4 input filter rule 25 action 'accept'
set firewall ipv4 input filter rule 25 protocol 'tcp'
set firewall ipv4 input filter rule 25 destination port '443'
set firewall ipv4 input filter rule 25 source address '198.51.100.0/24'

Illustrative output

Read-only / Safethe client the region is managed from; the register says 198.51.100.14

ops@collector01:~$ getent hosts jump-eu-south.example.com

203.0.113.71    jump-eu-south.example.com

Illustrative output

Read-only / Safeone change, committed and confirmed; the saved configuration matches it

vyos@edge3:~$ show system commit | head -4

0   2026-08-18 14:07:52 by netops via cli   SEC-1180 restrict mgmt sources
1   2026-06-02 10:44:19 by netops via cli   bgp timers per SEC-0994
2   2026-05-19 08:16:03 by netops via cli   syslog target change
3   2026-04-28 16:30:41 by netops via cli   interface description tidy

Illustrative output

Work the evidence before reading on

The router is refusing you and accepting somebody else at the same instant.

SSH from the jump host times out; SSH from the collector succeeds; ping works from both. What is different between those two clients, and which field in rule 20 is it?
Rule 20’s packet counter is not zero. Something is matching it. Whose traffic is that, and does a non-zero counter mean the rule is doing what it was written to do?
Rule 10 accepts established connections and sits above rule 20. The engineer held a session through the commit and for eleven minutes afterwards. Which rule was carrying that session, and what would the change have had to do to interrupt it?
commit-confirm, confirm and save were all used, in that order, by the book. Trace what each of them could have detected on Tuesday afternoon, given the answer to question 3.
The power cycle did not help. What does save do, and what would a reboot have done if save had not been typed?

Then the question that resolves it: what test could the engineer have run on Tuesday that would have failed?

Root cause

1. The source list came from a register that has been wrong since March

The change is well built. An explicit source list on the management accept rules is exactly what the audit asked for, and it was rolled to the whole fleet in one window with a confirm timer on every router.

The source list came from the network register, which is the estate’s system of record for management ranges, and which says the EU-South jump host is in 198.51.100.0/24. It moved to a new colo in March and was renumbered into 203.0.113.64/26. DNS was updated in the same change. The register was not.

Eleven routers are managed from jump hosts that never moved, so eleven routers were unaffected. The three that were affected are all in one region because the stale entry is regional, which is precisely why the fault impersonated a regional network problem for the first hour.

The rule is doing exactly what it was told. Rule 20’s counter is non-zero because 198.51.100.0/24 is still a real range with real hosts in it — the central monitoring collector is one of them — so the rule looks alive, and its liveness is unrelated to whether it lets the right people in.

2. The change is invisible to every connection that already exists

This is the half that turns a routine mistake into a two-day outage.

The input filter accepts established connections in rule 10, above the management rules. That is the standard build and it is correct: a stateful firewall classifies each packet against the connection table first, and a packet belonging to a flow that already has state is accepted there and never reaches the rules further down.

The engineer’s SSH session was established before the commit. After the commit it kept matching rule 10, exactly as it had before, and nothing about it changed. It would have survived any source-list value at all, including one with no overlap with reality whatsoever.

So the change applies only to connections that do not exist yet, and the only connection available to test it with was one that could not be affected by it. The eleven minutes of unbroken connectivity in the session log are real, and they are evidence of nothing.

3. Three safety practices ran correctly and none of them could see it

commit-confirm schedules a rollback if the operator does not confirm. It protects against a change whose damage is visible inside the window. Here the window elapsed with a healthy session as the only signal.

confirm was the correct action given that signal. save persisted a configuration that had been tested as far as anybody knew how to test it.

And save is why the power cycle failed. Without it, a reboot would have brought the router up on the last saved configuration — the one from June — and restored access, which is why “reboot it” is a genuine recovery path on VyOS and why it is a genuine recovery path exactly once, before somebody saves.

4. The recovery path that was supposed to catch this had been removed

The out-of-band runbook names a console server for the site. It was decommissioned in May in a cost-saving exercise, and the runbook was not updated, so the estate believed it had console access to these routers and did not.

Nobody discovered that on a quiet afternoon. They discovered it on a bridge call, forty minutes in, which is the worst possible moment to learn that a control does not exist.

Under the hood

The input filter is the chain for traffic destined to the router itself, as distinct from the forward filter, which handles traffic passing through it. That distinction is why the region kept forwarding perfectly while the routers were unmanageable: not one packet of transit traffic is evaluated by the chain that locked everybody out.

Within that chain, connection state is evaluated before addresses. A packet arrives, the kernel looks it up in the connection tracking table, and a packet belonging to a known flow is classified established and matched by rule 10. A packet that starts a new flow is classified new, falls past rule 10, and is matched against the rules that carry source addresses. That ordering is deliberate and universal — putting the established rule first is what stops every packet of every session being re-evaluated against the whole rule set — and its consequence is the one that matters here: an in-flight session is not subject to a rule change.

There is a second, sharper consequence in the shape of the failure. Rule 20 does not deny anything; a packet from the wrong source simply fails to match it, falls through every remaining rule, and is discarded by default-action drop. Drop sends nothing back, so the client waits for a TCP timeout instead of receiving a reset. That is why the symptom is a hang rather than an error, and why it looks like a path problem rather than a policy decision.

VyOS also offers source restriction at the SSH daemon with set service ssh source-restrict, which behaves differently: the daemon accepts the TCP connection and then closes it, so the client is refused immediately and learns that something on the far end made a decision about it. Same intent, different diagnostic signature, and worth choosing deliberately rather than by habit.

Testing an access change on the session that is already open

The commit-confirm pattern exists to catch a change that removes the operator’s own access, and this is the shape of change it cannot catch. The safety net watches a connection that the change was structurally incapable of affecting, reports healthy for the full window, and hands you a confident confirm.

The rule that fixes it is short enough to put in a procedure: after committing a change to the management path, open a new connection, from the client that is supposed to be able to connect, before you confirm. Ideally by somebody who is not you, from a machine you are not currently logged in from.

The same shape recurs everywhere state outlives configuration. A revoked credential that leaves existing tokens valid. A tightened database grant that does not disturb open connections. A rotated certificate that existing TLS sessions never re-verify. In each case the system tells you the truth about the past and nothing about the next attempt.

If a test cannot fail, it is not a test. Ask what result would have made you roll back — and if there is no such result, you have not verified anything.

Resolution

Get in from a source the rule still permits, before reaching for anything heavier. The routers are running and accepting management connections from 198.51.100.0/24, which is where the central monitoring collector lives. That host is the way in, and it is available immediately.
Stop the power-cycle request. A reboot cannot help: the configuration was saved, so the router boots straight back into the rule that is refusing you, and the reboot costs a forwarding outage the region does not currently have.
Confirm the diagnosis from the router before changing it. show firewall ipv4 input filter and the jump host's actual address take under a minute together, and they distinguish this from an SSH daemon problem, a routing problem or a transit problem — all of which are still live theories on the bridge.
Widen the rule with the security owner, not around them. This control exists to close an audit finding, so adding a range to it is a change to a security control. That conversation is short, and skipping it is how a lockout incident becomes a compliance incident.
Apply the correction under commit-confirm from the permitted host: add 203.0.113.64/26 to rules 20 and 25. One router first, not all three, so that a mistake costs one router rather than the region.
Verify from a NEW connection from the jump host before confirming. Not the session you are holding, not a reconnect from the collector — a fresh connection from the client that could not get in. This is the step whose absence caused the incident, so it is the step that closes it.
confirm, then save, in that order and not before the new connection succeeded. Then repeat on the remaining two routers and verify each one individually; they were changed as a batch and can be wrong as a batch.
Correct the network register in the same ticket. Adding the range without fixing the source of truth leaves the identical trap armed for the next jump host that moves, and the change that moved this one updated DNS correctly, so the register is the part of the process that failed.
Raise the console server separately and treat it as a finding in its own right. The estate believed it had out-of-band access to these routers for three months and did not, which is a gap in the recovery plan for every device behind it, not a detail of this incident.
Check the eleven working routers for the same latent dependency rather than filing them as unaffected. They are working because their jump hosts have not moved yet.

Verification

A new SSH connection from the EU-South jump host, opened after the change, succeeds. Not the session you are holding and not a reconnect from the host you fixed it from — the client that could not get in.
The HTTPS API answers from the same client if the API is part of the allowlist. Two services were restricted and only one of them is the one everybody tested.
The counters agree with the intent. show firewall ipv4 input filter shows the management accept rules incrementing as you connect, and the default-action drop counter no longer climbing in step with management attempts.
The control is still a control. Connect from an address that is not on the list and confirm it is refused. A rule widened to 0.0.0.0/0 by accident would also make every check above pass.
All three routers pass individually. They were changed as a batch, so infer nothing about the second and third from the first.
The running and saved configurations agree, so a reboot restores access rather than removing it. This is the property that made the power cycle useless, and it is worth confirming deliberately rather than assuming.
The nightly configuration backup succeeds for all three routers. That job failed on Tuesday night and again on Wednesday night, thirty-six hours before anybody noticed, and it is the off-router signal that this class of fault produces first.

Prevention

Never verify a management-path change on the connection you are already holding. The established-state rule accepts your session regardless of what you just changed, so the test is a new connection from the intended client, ideally opened by somebody who is not you.
Make the second session an explicit step in the remote-change procedure, and size the commit-confirm window around how long that takes rather than around how long the commit takes. A step that is implicit is a step that is skipped under pressure.
Keep save out of the confirm window. The sequence is commit-confirm, test from a new session, confirm, save. A save typed early turns a self-reverting change into a permanent one and removes the reboot as a recovery path, which is two safety nets lost to one keystroke of muscle memory.
Treat out-of-band access as a control to exercise, not a hostname in a runbook. Connect to it on a schedule. A console server decommissioned in May is a change to the recovery plan for every router behind it.
Build source allowlists from an inventory validated against reality, not from a register updated by convention. The renumbering here updated DNS and not the register, and the review that approved the change checked it against the register, so every step of the process was internally consistent and wrong.
Read the signals you already collect. The configuration backup failed the same night, for exactly the right reason, into a digest nobody opens.

What if the rule had used reject instead of drop?

The incident would have been an hour long instead of two days, and it would almost certainly have been found on Tuesday.

default-action drop discards the packet and sends nothing back, so the client sits through a TCP timeout and reports “connection timed out” — a phrase that names no actor and reads identically to a routing problem, a transit problem, a dead router or a firewall in the middle. That ambiguity is what sent the first hour to the provider.

A reject returns an ICMP unreachable or a TCP reset. The client fails immediately, and “connection refused” is unambiguous: something at the far end made a decision about this connection. Nobody calls a transit provider about a connection refused.

That is not an argument for rejecting everything. Silence is the right answer on an internet-facing chain, because a reset confirms to a scanner that the address is live and something is listening, and denying an attacker that confirmation is worth a great deal. The judgement is per-chain rather than global: drop on the outside, and consider reject on the management path, where every client that will ever hit the rule is one of yours and the diagnostic value is worth more than the concealment.

Whichever you choose, choose it deliberately and write down why, because the choice decides what your own on-call engineer sees at three in the morning.

Why this matters

Locking yourself out of a router is the oldest mistake in network operations and the profession has a good answer to it: make the change reversible on a timer, test it, and confirm only if it worked. That answer was applied here, correctly, on all fourteen routers, and it did not help — because the test was run over a connection the change could not touch, and a safety net is only as good as the signal you feed it.

The transferable idea is that state outlives configuration. Connection tracking, issued tokens, open database handles, negotiated TLS sessions: all of them carry a decision made under the old rules into a world running the new ones. Any test performed through such a thing is a test of the past. The new-connection rule is narrow and worth memorising, but the general habit is broader — ask what your verification would look like if the change had gone badly, and if the answer is “the same”, find a different verification.

And the boring half deserves equal weight. A register that DNS overtook five months ago, a runbook naming a decommissioned console server, a failing backup job in an unread digest. None of those is an interesting engineering problem and all three were load-bearing. The change was reviewed against the register, and the review was thorough, and being thorough against the wrong document produces a confident, well-documented outage.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The source list came from a register that has been wrong since March

2. The change is invisible to every connection that already exists

3. Three safety practices ran correctly and none of them could see it

4. The recovery path that was supposed to catch this had been removed

Resolution

Verification

Prevention