Runbook: Respond to Compromised Router Credentials

“A router credential has been compromised” describes at least four incidents that share a sentence and very little else: a password or key that authenticates an interactive login, an API key that authenticates automation, a community string or service account with no human behind it, and a break-glass account whose whole purpose is to work when nothing else does. The containment differs for each, and so does what you are allowed to break while containing it.

What they have in common is the thing that makes this runbook different from its equivalent on a server. A VyOS router’s configuration is a secret store. BGP passwords, OSPF key material, IPsec pre-shared secrets, WireGuard private keys, SNMP communities and the API keys themselves all live in the configuration tree, and any credential that can run show configuration has read all of them. That is why the first question is not “what did they do” but “what could they see” - and why a credential you would describe as read-only can be the more expensive incident, because it is the one whose blast radius includes other people’s routers.

The second thing that makes this runbook different is that the ordinary recovery tools are contaminated. rollback 1 restores the previous archive entry, and the previous archive entry contains the compromised credential. So does every backup file, every hourly rsync copy and every commit in the configuration repository. In every other VyOS incident, “go back to the last-known-good” is the fast answer. Here it is how the compromise comes back.

When to use this runbook

A router login credential - password, SSH key, or the vyos break-glass account - is known or suspected to be in the wrong hands.
An API key configured under service https api keys has leaked: pasted into a ticket, committed to a repository, or held by a host that has itself been compromised.
A workstation, laptop or automation host that held router credentials has been lost or breached.
An operator has left, or changed role, under circumstances that make an orderly deprovisioning inappropriate.
show log auth or the central syslog stream shows a successful authentication that no operator claims.

When not to use it

The credential is being rotated on schedule with nothing suspected. That is a planned change: vyos-rb-safe-remote-change, at change-window pace, with a peer review.
Nobody can log in and no compromise is suspected. A lockout after a management-plane change is vyos-bf-remote-commit-lockout territory - the fix is to undo the change, and rollback is safe there because the credential is not the problem.
The configuration is known-bad but not disclosed. Use vyos-rb-rollback-bad-config, which can use the archive freely.
The router itself is suspected of having been altered, not merely read. That is a rebuild decision, not a rotation: it runs through vyos-rb-recover-failed-router with the configuration treated as untrusted input.

Blast radius

Two circles, and operators routinely see only the inner one.

The inner circle is this router: who can log in, what the automation can still do, and whether the management plane stays reachable while you work. Every containment action in this runbook narrows management access, which is exactly what it is for, and which is why Step 0 exists.

The outer circle is every party that shares a secret with this router. A BGP password is agreed with a peer. An OSPF key is agreed across an area. An IPsec pre-shared secret is agreed with the far end. A WireGuard private key corresponds to a public key held by peers. Rotating any of these is a coordinated change with somebody else’s maintenance window attached, and none of them is optional once the configuration has been read.

Step 0: out-of-band first, and prove it works

Read-only / Safetest the path you will need, before you need it

# Substitute your own values before running:
CONSOLE=console-server.example.com
CONSOLE_PORT=4001
BMC=192.0.2.50

ssh "$CONSOLE" -p "$CONSOLE_PORT"

ipmitool -I lanplus -H "$BMC" -U admin power status

Not “we have a console server”. Log in over it now, from the workstation you will be using, and leave the session open. Containment on the management plane is indistinguishable, from the router’s point of view, from an attack on the management plane: a source-restrict list that omits your current source address, or a role change applied to the wrong account, locks out the operator just as effectively as the attacker.

Step 1: what could this credential do?

Read-only / Safethe two inventories

show system login user
show system login role

show configuration commands | match "system login"
show configuration commands | match "service https api keys"
show configuration commands | match "service snmp"

Read the role; do not assume it. The RBAC lesson’s blast-radius model is the one to apply here: a monitor or custom read-only role means the attacker can read system state, an operator role adds restarting services, and admin means they could edit configuration, commit, save and lock everyone else out.

What the role permits	What the incident actually is
A custom role listing exact `show` commands	Disclosure bounded by that list
Any role permitting bare `show`	Disclosure of the entire configuration, including every shared secret
`operator`	The above, plus service restarts - availability as well as disclosure
`admin`	The above, plus commits. Treat the configuration as potentially altered, not merely read
An API key bound to a user	Whatever that user’s role permits, over HTTPS, without a shell

Step 2: capture what containment destroys

Read-only / Safethe central copy first, then the router

# Substitute your own values before running:
ROUTER=edge-01.example.com
SYSLOG_HOST=syslog.example.com
CASE=/srv/incident/CASE-1234

mkdir -p "$CASE"

# 1. The exported authentication record. This one is outside the blast radius.
ssh "$SYSLOG_HOST" "grep -h '$ROUTER' /var/log/remote/*.log" > "$CASE/auth-central.log"

# 2. The router's own view, for comparison. A divergence is itself evidence.
ssh "vyos@$ROUTER" 'show log auth' > "$CASE/auth-router.log"

# 3. The commit record: who committed what, when, from where.
ssh "vyos@$ROUTER" 'show configuration commit' > "$CASE/commits.txt"

# 4. The configuration as it stands, before you change it.
ssh "vyos@$ROUTER" 'show configuration commands' > "$CASE/config-pre.txt"

Order matters. The central syslog copy comes first because it is the only one the compromised credential could not have touched - the local logging lesson’s whole argument for exporting is this moment. Compare the two afterwards: if the router’s local record is shorter than the central one, that difference is a finding, not a glitch.

show configuration commit is the one operators forget, and it is the record this runbook turns on. It carries the commit number, timestamp, user and source address for every archived commit. Read it line by line and attribute every entry. One unattributable line changes the incident: you are no longer asking what was disclosed, you are asking what was changed, and the configuration in front of you can no longer be treated as yours.

Read-only / Safedid the credential get used, and did it change anything

show log auth | match "Accepted"
show log auth | match "Failed"
show log auth | match "vyos-http-api"
show configuration commit

Three findings, in ascending order of cost. Failed attempts only: the credential is being tried, not held - contain and rotate at your own pace. A successful authentication: it is held; disclosure has happened. A commit in the archive attributed to that user: it is held and it was used to change the router, and everything after this point runs alongside a change investigation.

Step 3: contain the path before the credential

Service impact possiblepath containment - note the timer

configure
set service ssh source-restrict ip-address '203.0.113.50'
set service ssh source-restrict ip-address '203.0.113.51'
compare
commit-confirm 10
exit

The peer runbooks treat the commit-confirm timer as the mechanism that saves you from your own mistake. Here it saves you from a lockout and simultaneously threatens the containment, because what expires is the security control. Set the timer to cover the verification you actually intend to run, stay in the session, and go straight to confirm when the check passes.

Service impact possibleaccount containment - disable, do not delete

configure
set system login user compromised-operator role 'disable'
delete system login user compromised-operator authentication public-keys laptop
delete system login user compromised-operator authentication plaintext-password
compare
commit-confirm 10
exit

Both halves are needed and they do different things. The disable role blocks authentication as policy; deleting the public-keys entry removes the credential itself, which matters because the same key may be installed on other routers and because a later role edit could re-enable an account whose key was left in place.

Service impact possiblerevoke one API key, not the API

configure
delete service https api keys id ansible-prod
compare
commit-confirm 10
exit

Key ids exist so that revocation can be granular. Disabling the HTTPS service outright takes down every consumer, including the pipeline you may need to push the remaining fixes. Revoke the id that leaked; leave the others answering.

Configuration changekeep the containment

configure
confirm
save
exit

confirm before the timer, save after it. A containment that is not confirmed reverts on a timer; one that is not saved reverts at the next reboot. Both failures restore the compromised credential without a human deciding to, and both are invisible until someone uses it.

Step 4: the secret inventory nobody wants to do

If Step 1 established that the credential could run show configuration, then disabling the account is the beginning of the response and not the end of it. Everything in the tree that is a secret has been read.

Read-only / Safeenumerate what was exposed

show configuration commands | match "password"
show configuration commands | match "authentication"
show configuration commands | match "pre-shared-secret"
show configuration commands | match "wireguard"
show configuration commands | match "service snmp"
show configuration commands | match "service https api keys"

Secret	Where it lives	What rotating it costs
BGP session password	`protocols bgp neighbor ... password`	A coordinated change with the peer AS; the session drops if only one side changes
OSPF key material	`protocols ospf interface ... authentication`	Adjacency-wide; every router in the area must change together
IPsec pre-shared secret	`vpn ipsec site-to-site peer ... authentication pre-shared-secret`	Tunnel down until both ends agree the new value
WireGuard private key	`interfaces wireguard ... private-key`	The public key changes with it, so every peer must be updated
SNMP community	`service snmp community`	Monitoring blind until the poller is updated
Remaining API keys	`service https api keys id ... key`	Each consumer must be updated; do them one at a time
TLS key material	the `pki` tree	Re-issue rather than re-use; the CA question belongs to the PKI owner

Rotate one at a time, each with its own verification, and verify on the protocol rather than in the configuration. A BGP password that is present in show configuration commands and a BGP session that is Established are different claims, and only the second one means the rotation worked.

Step 5: the recovery path is contaminated

Rollback and restore both reintroduce the compromised credential

The commit archive holds the running configuration at every successful commit. /config/backups/ holds whatever snapshots were taken. The hourly rsync copy and the configuration repository hold the same content off-box.

Every one of those, for every point in time before the rotation, contains the compromised password, the compromised key and the compromised API key. They are correct backups of a configuration that is now unsafe.

So rollback 1 is not the undo for this runbook, and a restore is not a recovery from it. If a restore becomes necessary for an unrelated reason - hardware failure mid-incident is not hypothetical - the restore must be followed immediately by re-applying every rotation, and that dependency has to be written down before anyone reaches for the archive.

Read-only / Safemark the contaminated restore sources

show configuration commit
show system config-management

Record which archive entries, which backup files and which repository commits predate the rotation, and attach that list to the incident. The person who restores this router in six months will not be the person who ran this runbook, and the archive gives them no signal at all that entry 7 carries a credential somebody else knows.

Step 6: verify from outside

Read-only / Safeprove the credential is dead

# Substitute your own values before running:
ROUTER=edge-01.example.com
OLD_KEY=~/.ssh/compromised_ed25519
OLD_API_KEY=the-leaked-key-string

# Must fail.
ssh -i "$OLD_KEY" -o IdentitiesOnly=yes "compromised-operator@$ROUTER" 'show version'

# Must return 401, not a result.
curl -k -s -o /dev/null -w '%{http_code}' -H "X-API-Key: $OLD_API_KEY" "https://$ROUTER:8443/show/version"

Test the negative. A configuration that says the account is disabled and an authentication attempt that is refused are different claims, and only the second one is evidence. Then confirm the containment persists:

Read-only / Safeand prove it survives a reboot

show system login user
show log auth | match "Accepted"
show configuration commit
diff /config/config.boot <(show configuration commands)

Common patterns

Symptom	Likely cause	Resolution
Containment reverted itself after ten minutes	`commit-confirm` timer expired unconfirmed	Re-apply, then `confirm` and `save` in the same session
Containment gone after a reboot	`confirm` ran, `save` did not	Re-apply and `save`; verify the running and saved configurations agree
Compromised credential works again weeks later	A restore or `rollback` reintroduced it	Re-apply the rotations; the archive and the repository are contaminated by design
Account disabled, key still accepted elsewhere	The `public-keys` entry was left in place, and the key is installed on sibling routers	Delete the key entry here and audit the fleet for the same key
Audit entries now reference an unknown user	The account was deleted rather than disabled	Not recoverable. Use `role disable` next time
BGP session dropped during rotation	Only one side of a two-party secret changed	Coordinate the change with the peer; verify with `show ip bgp summary`
Monitoring went blind after containment	The SNMP community or the monitoring role was rotated without updating the poller	Rotate service credentials with their consumer, one at a time
Operator locked out along with the attacker	`source-restrict` omitted the current source address	Correct the list over OOB; do not remove the restriction wholesale
Nothing in `show log auth`, but the API was used	HTTP API activity is logged under `vyos-http-api`, not as an SSH login	Search the API lines and the central syslog copy separately
An archive entry nobody can attribute	The credential had write access	Stop. This is a change incident; the configuration is untrusted input

Escalation

Escalate when:

Before the first containment commit - to the security owner, because containment destroys evidence.
To every peer holding a shared secret that appears in this configuration.
Before restricting the management plane on one half of a redundant pair.
The moment show configuration commit shows an entry nobody can account for.
Out-of-band access cannot be established.
The credential in question is the break-glass vyos account or the API key the automation pipeline depends on.
There is any sign of reuse on another router, the backup host, or the configuration repository.

Why this matters

The reason this runbook spends so long before it changes anything is that the two instincts an operator brings to it are both slightly wrong.

The first instinct is to make the account disappear. It feels like the most complete possible action, and it is the one that destroys the audit trail, cuts the link between the log entries and a configured identity, and buys nothing that role disable did not already buy. Containment and erasure look similar under pressure and are not the same thing.

The second instinct is to reach for the archive. Every other VyOS incident rewards that: the commit archive is the operator’s safety net, and “go back to the last-known-good” is usually the fastest correct answer on the box. In a credential incident the last-known-good is the last-known-compromised. The archive, the backups and the repository are all faithful copies of a configuration that contains the secret somebody else now has, and the moment anyone restores one of them the incident quietly reopens.

What is left, once both instincts are set aside, is the part that actually determines the cost: the configuration is a secret store, so the reach of the credential is the reach of show configuration, and that extends to every peer that agreed a password with this router. The account was the easy half. The shared secrets are the incident.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation