Router Credentials Compromised · Break/Fix · VyOS

Reported symptoms

edge1 is a VyOS border router with two eBGP sessions, a management VRF, and an HTTP API that a nightly Ansible compliance run uses to read its configuration. The estate is in a declared change freeze for the quarter close.

At 09:20 on Tuesday the configuration-drift job raised a ticket. Three commits landed on edge1 overnight — 03:04, 03:09 and 03:15 — against no change record, with nobody from the on-call rota working. All three are attributed to user vyos, which is the shared account the automation also authenticates as, so the opening theory on the bridge was a pipeline that had run on a schedule somebody had forgotten about.

Three more tickets were already open, each filed by a different team as its own problem:

The central syslog collector has a gap for edge1 beginning at 03:15 and still open. The logging team have checked their ingest, found it healthy for every other source, and pushed the ticket back with “router-side or WAN”.
The management chain has logged no default-deny hits since about 03:04. The NOC raised it as “firewall logging broken on edge1” and it is sitting in a queue.
A read-only monitoring account’s SSH check has been flapping since 03:20. The monitoring team have already attributed that to their own collector restart that evening.

The on-call engineer’s first move was the obvious one: look for unexpected logins. show log auth.log across the whole window is clean. No SSH authentication from anything outside the operator allow-list, successful or failed. Combined with the fact that the router is forwarding perfectly — both sessions up, no customer reports, dashboards green — the ticket was downgraded to P4 and left for the morning.

Evidence provided

Read-only / Safethe user column is useless here; the source-address column is not

vyos@edge1:~$ show configuration commit

18  2026-08-18T03:04:11+01:00   vyos   192.0.2.77    config.20260818-030411.boot
19  2026-08-18T03:09:47+01:00   vyos   192.0.2.77    config.20260818-030947.boot
20  2026-08-18T03:15:02+01:00   vyos   192.0.2.77    config.20260818-031502.boot

Illustrative output

Read-only / Safenothing between 22:31 and 06:02 — the result that downgraded the ticket

vyos@edge1:~$ show log auth.log

Aug 17 22:31:08 edge1 sshd[4412]: Accepted publickey for vyos from 203.0.113.50 port 51122 ssh2
Aug 18 06:02:19 edge1 sshd[7781]: Accepted publickey for monitoring from 10.0.0.60 port 44980 ssh2

Illustrative output

Read-only / Safea different door, and a different log file

vyos@edge1:~$ show log | match vyos-http-api

Aug 18 03:04:11 edge1 vyos-http-api: INFO [api] POST /configure from 192.0.2.77 (key ci-runner-legacy, user vyos) - 200 OK
Aug 18 03:09:47 edge1 vyos-http-api: INFO [api] POST /configure from 192.0.2.77 (key ci-runner-legacy, user vyos) - 200 OK
Aug 18 03:15:02 edge1 vyos-http-api: INFO [api] POST /configure from 192.0.2.77 (key ci-runner-legacy, user vyos) - 200 OK

Illustrative output

Read-only / Safetwo keys, one consumer, and a listener that was never moved

vyos@edge1:~$ show configuration commands | match 'service https'

set service https api keys id ansible-prod key '****'
set service https api keys id ansible-prod user 'vyos'
set service https api keys id ci-runner-legacy key '****'
set service https api keys id ci-runner-legacy user 'vyos'
set service https listen-address '0.0.0.0'
set service https port '8443'

Illustrative output

The three diffs, taken with compare 3, compare 2 and compare 1, are each about two lines long:

Read-only / Saferule 5 — below the logging drop rule, which is rule 10

vyos@edge1# compare 3

+ firewall ipv4 name MGMT-LOCAL rule 5 action accept
+ firewall ipv4 name MGMT-LOCAL rule 5 source address 192.0.2.77

Illustrative output

Read-only / Safethe identifier is attacker-chosen text and is styled to be unremarkable

vyos@edge1# compare 2

+ system login user vyos authentication public-keys ops-backup type ssh-ed25519
+ system login user vyos authentication public-keys ops-backup key AAAAC3NzaC1lZDI1NTE5AAAA...

Illustrative output

Read-only / Safe03:15:02 — the same second the collector-side gap begins

vyos@edge1# compare 1

- system syslog host 203.0.113.100 facility all level info
- system syslog host 203.0.113.100 protocol tls

Illustrative output

Work the evidence before reading on

There are four open tickets and one strong negative result. Before reading on, work out how many of them describe the same event.

auth.log is clean for the whole window, and three commits landed inside it. Both statements are true at once. What has to be the case for that to be possible, and what does it tell you about which log you should have read?
The syslog gap begins at 03:15:02 and the third commit is timestamped 03:15:02. Is the gap a symptom of a broken collector, or is it the change itself? What would you check to tell those apart, and can you check it from this router?
Rule 5 accepts. Rule 10 drops and logs. Which one fires, and what does that do to the ticket the NOC raised about broken firewall logging?
Every archive entry carries a user, and the user on all three is vyos. Why does that column answer nothing here, and what would have to change about the estate for it to answer something?
The three diffs total six lines and none of them touches forwarding. Why would an intruder with full configuration access make changes this small?

Then the question that decides how the rest of the incident is run: if you revert all three commits, what is the state of the access that produced them?

Root cause

1. Two survivable mistakes that are not survivable together

The first is a credential that outlived its consumer. ci-runner-legacy was created fourteen months ago for a CI runner that was decommissioned three months later. The runner is gone; the key is still in the configuration and still valid. Nobody rotated it, because rotation is a task that belongs to an owner and this key’s owner stopped existing. Its string had lived in the runner’s environment variables and in a repository that was archived rather than purged, so who holds it now is not a question anyone can answer.

The second is a listener that was never moved. service https listen-address is still on its default, so the API answers on every address the router holds, including the public ones.

On their own, each is a finding in an audit. Together they are an unauthenticated person’s route to the configuration tree, gated by a string that is no longer controlled.

2. The API and the CLI arrive at the same commit engine

This is why the evidence looks the way it does. An operator in configure and an HTTPS request to /config reach the same configuration tree and the same commit engine; the API is a different door into the same room. Everything downstream of the commit follows from that: the validators ran, the archive entry was written, the diff is inspectable with compare, and the router carried on forwarding because nothing about forwarding was touched.

It is also why the archive is the one place worth looking first on a router you suspect. Access paths write to different logs and some of those logs can be turned off, but every path that changes the configuration converges on the commit engine, and the commit engine writes an archive entry. “Who logged in” is a question with several answers in several places. “What changed, when, and from where” has one.

3. The three commits are one sequence

Read together rather than as three tickets, the changes are ordered the way an intruder orders them.

rule 5 accept from a single source, inserted below the rule that drops and logs. Firewall rules evaluate in sequence and the first match wins, so the logging rule is no longer consulted for that source. This is not a permission the intruder needed — they already had the API — it is the removal of a record. The NOC’s “firewall logging broken” ticket is this line, seen from the outside.

An extra public key on the shared vyos account. Access via SSH now exists independently of the API key, so revoking the key alone leaves the door open. The identifier ops-backup is text the intruder chose; it is designed to survive a skim of the configuration.

The syslog host deleted. From 03:15:02 the box stops reporting off-site. Every remaining record is local, which means every remaining record is under the intruder’s control. The collector-side gap the logging team pushed back on is not a collector problem and never was — it is the third commit, observed from the one place the intruder could not reach.

4. The clean auth.log

auth.log records SSH authentication. No SSH authentication happened. The file is accurate and the conclusion drawn from it was wrong, because the question being asked was “did anyone get in” and the file only answers “did anyone get in this way”.

The monitoring check flap is the one report here that is genuinely unrelated. The monitoring team’s explanation was correct the first time. It matters only because it is what made somebody open the router at all — the intrusion was found by coincidence, which is a finding about the estate rather than about the intruder.

Under the hood

Every access path on a VyOS router writes its record somewhere, and the somewheres are not the same file.

sshd writes authentication outcomes to auth.log through syslog: accepted and failed publickey attempts, the source address, the user. The HTTP API writes its own request log — method, path, source, key id, response code — through the same syslog daemon but under its own program name, which is why show log | match vyos-http-api finds the three requests and show log auth.log finds nothing. Console logins land somewhere else again. Commits, from any of those paths, land in the archive.

The practical consequence is that a negative result from one log is evidence about one path and nothing more. An investigation that begins with “check auth.log” and stops when it comes back clean has not established that nothing happened; it has established that nothing happened over SSH. On this router the correct opening move was the archive, because the archive is path-independent: whatever the door, changing the configuration means passing through the commit engine.

There is a limit worth stating plainly. All of these records are local files on a box that someone else has administrative access to. That the API request lines were still there is luck, not a guarantee — the intruder removed the export and did not clear the local logs. Treat local evidence as useful when present and worthless when absent, and treat off-box evidence as the only kind that survives an adversary with commit rights.

The credential nobody owned

ci-runner-legacy was created properly. It was scoped to one consumer, it was named after that consumer, and it was placed alongside ansible-prod in exactly the way the one-key-per-tool guidance asks for. Everything about its creation was right.

What was missing was the other end. When the CI runner was decommissioned, the runner was removed from the asset inventory and the key was not removed from the router, because deleting the key was nobody’s task — the person decommissioning the runner was not thinking about the routers, and the person who maintains the routers had no reason to know the runner had gone.

That is the shape of this failure and it is worth recognising away from VyOS entirely. Credentials do not become dangerous when they are weak; they become dangerous when they become ownerless. An ownerless credential is never rotated, never audited, never noticed in a review, and never revoked, because every one of those actions is something an owner does. Rotation policies do not fix it — a ninety-day rotation cadence applies to keys somebody is tracking, and this one had fallen off the list.

The control that does fix it is an expiry attached to the credential itself, so that neglect produces a failure rather than an opening, plus a periodic reconciliation that asks of every key not “is it strong” but “does the thing that uses it still exist”.

Resolution

Decide containment first and write the decision down. Pulling edge1 out of service is a customer-visible outage; leaving it in service accepts that an unknown party may still have a path onto it. Holding is defensible while the intruder has only touched the management plane, but it is a decision with a cost, so it gets a named owner and an end time rather than becoming the default because nobody said otherwise.
Do not treat rollback as the remediation. rollback 3; compare; commit reverses the three changes, but the credential that admitted them was created long before the oldest surviving archive entry — every configuration you can roll back to still contains it. Rollback here removes the intruder's changes and leaves the intruder's access.
Revoke the credential explicitly and by id: delete service https api keys id ci-runner-legacy. Then remove the added public key with delete system login user vyos authentication public-keys ops-backup. Commit both together, and use commit-confirm if you are working over the same path you are changing.
Restore the evidence channel before anything else that is optional: put the syslog host back, and confirm at the collector — not on the router — that messages are arriving again. Until that is true, everything you do next is being recorded only where the intruder can reach it.
Put the management chain back in order so that the logging drop rule is consulted again for every source, and verify the ordering by observing a drop rather than by reading the configuration.
Only now do the hardening: move service https listen-address onto the management VRF address, add set service ssh source-restrict ip-address for the operator networks, and rotate ansible-prod as well, because it has been sitting on the same reachable listener as the key that was used.
Widen the scope beyond the three commits you know about. Diff every archive entry you still have, not only the three that produced symptoms, and check the other routers that shared the same CI runner era for the same key id.
Bound the exposure window from off-box evidence — collector history, netflow, the partner-side records — and if it cannot be bounded, stop cleaning up in place. A router whose post-03:15 history cannot be reconstructed is rebuilt from a known-good configuration, not repaired.

Verification

The revoked key is dead, demonstrated rather than assumed. Present ci-runner-legacy to the API and confirm a 401. Its absence from show configuration commands says the configuration no longer contains it; only a rejected request says the router no longer accepts it.
The account carries only keys with living owners. show system login user vyos authentication public-keys lists them, and each is confirmed by the person who holds the private half — not by recognising the identifier, which is text the intruder was free to choose.
The collector is receiving again and the gap has a definite end. This check is performed at the collector, because it is the one piece of evidence in this incident that a party with commit rights on edge1 could not have staged.
The management chain records again. Probe it deliberately from a disallowed source and watch the default-deny counter advance. Reading the rule order back from the configuration confirms your edit, not the behaviour.
Every archive entry has been diffed, not only the three. The three you found are the three that produced visible symptoms, which is not a reason to believe they are all there were.
The exposure window is stated, with its source. "Nothing else happened" is a claim that needs off-box evidence covering the period; if you do not have that evidence, the finding is that the window is unbounded.
The decision record exists outside the incident channel: what was held, who owns it, when it is reviewed, and what was rebuilt rather than repaired.

Prevention

Give every credential an owner and an expiry. One key per consumer was already the pattern here and it did not help, because the control that was missing sits at the other end of the lifecycle: decommissioning the consumer must be what deletes the key. An expiry makes neglect fail closed instead of failing open.
Take the listener off the default address. Bind the API to the management VRF or to localhost and add a source restriction, so that holding a leaked key is necessary but not sufficient. This is the control that would have made the fourteen-month-old key harmless.
Alert on commits, not on people. A commit outside a change window is rare, cheap to detect and very hard for an intruder to avoid, because every path to the configuration passes through the commit engine. The archive listing already carries the timestamp, the user and the source address the alert needs.
Export syslog to two collectors, and alert on silence as well as on content. A single export host is a single line an intruder can delete; a router that stops reporting is telling you something, and here nobody was listening for that.
Retire the shared vyos account for interactive use. Named accounts turn the archive’s user column from a dead end into an answer, and they let a single person’s access be revoked without breaking the automation.
Raise system config-management commit-revisions above the default of twenty on any router that matters. The archive is your incident timeline and the default retention is measured in changes rather than in days, so a busy router can age its own evidence out before anyone looks.
Reconcile the credential inventory against the asset inventory on a schedule, and make the question “does the thing that uses this still exist” rather than “is this key strong enough”.

What if the key had been bound to a read-only role instead of the shared admin account?

The blast radius collapses, and the incident becomes an alert rather than an intrusion.

Both API keys on this router authenticate as vyos, which is an admin account. The key inherits everything that account can do, so possession of the string is possession of full configuration rights. A key bound to a user with a restricted role could read what the compliance job needed to read and nothing more — the set operations behind all three commits would have been refused, and the refusals would have been logged from an unexpected source address at three in the morning, which is precisely the signal nothing in the estate was generating.

The CI runner that owned this key needed to read configuration. It was given an account that could rewrite it, because that was the account that already existed and binding a new key to it was one line. That trade — a minute of work against a permanent expansion of what a leaked string is worth — is the one to notice, and it recurs far beyond routers. Every bearer token is worth exactly what the identity behind it can do.

It is worth being precise about what this would not have fixed. A read-only key on a listener reachable from the internet still leaks the configuration, and the configuration contains other secrets. Least privilege reduces the damage; it does not replace getting the listener off the public address.

Why this matters

Four tickets were open and three of them were the same event, filed by three teams who each had a partial view and a plausible local explanation. The syslog team had a gap and no cause, so it was the router’s fault. The NOC had a log that stopped, so logging was broken. The drift job had commits with no change record, so it was probably automation. Each explanation was reasonable in isolation, and each one absorbed a piece of the intrusion into a category where it stopped being alarming.

What broke the deadlock was not a cleverer tool. It was noticing that a negative result had been over-read: auth.log was clean, and clean meant nothing happened over SSH, not that nothing happened. Every path leaves a record somewhere, and the somewheres differ; an investigation shaped by which log came to hand first will find whatever that log covers and confidently stop.

The transferable model is about where evidence converges and who controls it. Configuration changes converge on the commit engine, so the archive answers “what changed and from where” regardless of the door used. And an adversary with commit rights controls every local record on the box, which is why the collector — the one witness outside their reach — was the piece of evidence that actually settled this, and why the first restorative action was to turn the export back on before doing anything else.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Two survivable mistakes that are not survivable together

2. The API and the CLI arrive at the same commit engine

3. The three commits are one sequence

4. The clean auth.log

Resolution

Verification

Prevention