API Automation Removes Critical Route

Reported symptoms

At 06:10 the storage team escalates. Every replication job from the primary site to the DR target failed overnight. The failures are connection timeouts — not authentication errors, not “target full” — and they start at 21:07 with a clean edge: all jobs, no partial successes, nothing recovered on retry.

Three things make this look like anything but a routing change.

• The DR network’s synthetic probe is green, and has been all night. The monitoring collector sits in a different segment and reaches that range over the transit path, so it has never used the route that disappeared.
• Both edge routers look exactly like last week. show ip bgp summary on edge-01 reports the same two sessions, the same uptimes and the same received-prefix counts it reported on Monday. Nothing flapped, nothing reconverged, no session bounced.
• The only change record for the evening is CR-2291, “add two NTP servers”, approved in ninety seconds and applied by the configuration pipeline at 21:05. It contains no routing lines of any kind.

And edge-02 is fine. The same pipeline ran against it in the same minute, with the same template change, and nothing on it broke.

The storage team’s working theory is a firewall change on the DR side. The network team’s working theory is that this is not a network problem.

Evidence provided

vyos@edge-01:~$ show ip route static

S>* 192.0.2.128/25 [1/0] via 192.0.2.2, eth0, 3w4d
S>* 198.51.100.64/26 [1/0] via 198.51.100.2, eth1, 3w4d

vyos@edge-01:~$ show configuration protocols static

route 192.0.2.128/25 {
  next-hop 192.0.2.2
  description "CR-1902 management range via core"
}
route 198.51.100.64/26 {
  next-hop 198.51.100.2
  description "CR-2044 DR management via interconnect"
}

vyos@edge-01:~$ grep -n 203.0.113 /config/config.boot; show configuration commands | grep 203.0.113

47:    route 203.0.113.0/24 {

vyos@edge-01# show configuration commit

1   2026-08-17T21:05:12+01:00   apiuser   192.0.2.40   config.20260817-210512.boot
2   2026-07-27T02:41:07+01:00   oncall    192.0.2.51   config.20260727-024107.boot
3   2026-07-26T18:12:44+01:00   apiuser   192.0.2.40   config.20260726-181244.boot

The pipeline’s job log for that run, kept by CI, records exactly two calls against the router:

POST /config  {"op": "load", "file": "rendered/edge-01/config.boot"}
  -> {"success": true, "data": null, "error": null}
POST /config  {"op": "commit"}
  -> {"success": true, "data": null, "error": null}

And the rendered document that was loaded — also kept as a CI artefact — contains no line matching 203.0.113.

Work the evidence before reading on

Every command above returned correct output. Nothing is broken, no daemon crashed, and no operator typed a delete. Work these four questions before reading the root cause.

1. The saved configuration has the route and the running configuration does not. Which VyOS operation writes one and not the other, and what does that tell you about how the route left?
2. There is exactly one commit between “the route was there” and “the route is gone”, and its content is two NTP servers. What does a full-document apply do to configuration that is not in the document?
3. The change was reviewed. The diff that was reviewed was a diff between two rendered documents. Where in that diff would a route that has only ever existed on the router appear?
4. edge-02 was untouched by the same run. What is different about edge-01 — the pipeline, or its history?

Before continuing: if you restore the route right now and change nothing else, how long do you have before it disappears again?

Root cause

1. load replaces the candidate; it does not merge into it

The VyOS HTTP API’s /config endpoint supports four operations: set, delete, load and commit. set and delete are surgical — they touch one node and leave everything else alone. load is not. It replaces the candidate configuration with the complete document supplied, and the commit that follows makes the running configuration equal to that document.

That is the whole mechanism. Making the router match the document is what the operation is for, so there is no warning, no confirmation and no per-line diff at apply time. Every line in the document is applied and every line on the router that is absent from the document is removed, with identical confidence.

2. Git had never heard of the route

At 02:41 on 27 July, during INC-4471, an engineer added a path to the DR replication target by hand:

configure
set protocols static route 203.0.113.0/24 next-hop 198.51.100.2
set protocols static route 203.0.113.0/24 description 'INC-4471 temporary path to DR replication target'
commit
save
exit

Read that carefully, because almost everything about it is right. The route carries a description with an incident number. It was committed. It was saved. It appeared in the commit log with the engineer’s username and source address, exactly as the audit trail is supposed to record it.

The step that was not taken was the fifth one: the route was never added to host_vars/edge-01.yml. From 02:41 that morning, the router and Git disagreed by one route, and the pipeline’s semantics had already decided what would happen the next time they were reconciled.

3. The review could not have caught it

The pipeline renders a document from the templates and host variables in Git, and the change process diffs the newly rendered document against the previously rendered one. Both sides of that diff come from Git.

A configuration line that exists only on the router is on neither side. It cannot appear in the diff, however carefully the diff is read, because the diff is not a statement about the router at all. The reviewer read a correct and complete diff of a correct and complete change, and approved two NTP servers in ninety seconds, which is a reasonable amount of time to spend on two NTP servers.

The deletion was not in the change. It was in the pipeline’s semantics, and it had been armed for three weeks waiting for any run at all.

4. Why edge-02 survived, and why a reboot would have “fixed” it

edge-02 has no drift, because nobody hand-edited it during INC-4471. The pipeline did exactly the same thing to both routers; only one of them had something to lose.

And on edge-01 the pipeline issues commit but never save. commit makes the candidate configuration live; save writes it to /config/config.boot. So the deletion is live and not persistent, the saved configuration still contains the route, and rebooting edge-01 would restore replication within minutes — while destroying the clearest piece of evidence in the incident, leaving the cause untouched, and re-arming it for the next pipeline run.

Resolution

1. Disable the pipeline schedule before touching the router. The pipeline runs on a timer and it is the thing that removes the route; restoring first and disabling second is a race you can lose.
2. Decide explicitly whether to restore now or to hold, and say which out loud. If the next replication window is eight hours away and the change window is two, holding is a legitimate answer: it is cheaper than an unreviewed edit to a production edge router at 06:15. A hold needs a named owner and a stated end time, or it is not a decision, it is a delay.
3. If restoring now, restore narrowly. Enter configure, re-add the two set lines exactly as the INC-4471 commit wrote them - prefix, next-hop and description - and apply them with commit-confirm 10 so a mistake on the interconnect path reverts itself.
4. Do not reach for rollback 1 because it is shorter. Rollback loads a whole archive entry: it would restore the route and revert the NTP change and anything else that run applied. compare after a rollback is the step that tells you what else you are about to revert, and it is not optional.
5. Verify from the storage host, not from the router, then confirm the commit-confirm before the timer expires.
6. Run save. The saved and running configurations disagreed in one direction before the fix and disagree in the other direction after it; leaving that unresolved stores up a separate incident for whoever reboots this router next.
7. Back-port the route into host_vars/edge-01.yml in this same working session. The next scheduled pipeline run is a hard deadline, and everything before this step is temporary by construction.
8. Decide which semantics the pipeline has and write the decision into the repository. Either Git is authoritative - in which case a pre-apply drift check must fail the run rather than report it - or the apply becomes additive and the estate accepts that nothing is ever removed automatically. Both are defensible; the combination that caused this incident is authoritative semantics with additive expectations.
9. Sweep the rest of the estate for the same exposure before re-enabling the schedule. Every router that has been hand-edited since the pipeline adopted a full-document load is carrying the same armed deletion, and the sweep is a diff of each router against its rendered document.
10. Re-enable the schedule and watch one full run against edge-01 complete.

Verification

1. The route is in the FIB, not merely in the RIB. show ip route static must show S>* 203.0.113.0/24 [1/0] via 198.51.100.2, eth1. Read the codes: a static route that is in the RIB and not the FIB prints as S without the > and *, often with inactive, and forwards nothing.
2. The kernel agrees. ip route show 203.0.113.0/24 returns the route with proto static. The RIB and the kernel FIB can disagree, and the kernel is what forwards packets.
3. The configuration matches. show configuration protocols static shows the route with its description, so the next operator can see why it exists and which incident put it there.
4. The save happened. grep 203.0.113 /config/config.boot and show configuration commands | grep 203.0.113 now agree. Before the fix they disagreed, and that disagreement was the single most informative piece of evidence in the incident.
5. The service works. Run one of the replication jobs that failed overnight, from the host that ran it. A routing table is evidence that packets can leave; it is not evidence that the job succeeds.
6. The route is in the render. Render the pipeline document for edge-01 from current Git HEAD and grep it for the prefix. If it is absent, the route is still scheduled for deletion no matter how healthy the router looks right now - and this is the check that distinguishes a fixed incident from a postponed one.
7. The route survives an apply. Re-enable the schedule, watch one full run, and confirm the route is still there afterwards. A fix nobody has seen survive an apply is a hypothesis.
8. The drift gate can fail. Introduce a deliberate one-line drift on a lab router and confirm the pre-apply check refuses the run. A guard that has never rejected anything is a comment.

Prevention

• Diff against the router, not against the previous render. A pipeline that compares Git to Git can only tell you what you already knew. The question that matters before an authoritative apply is what the router has that the document does not, and that is a different command.
• Make drift detection a gate, not a nightly report. A job that mails a divergence at 03:00 and applies anyway has documented the incident rather than prevented it.
• Give every emergency change a back-port task that closes before the incident does. The three weeks between the INC-4471 edit and the outage is the window in which this is free to fix; afterwards it is an outage with a storage team in the bridge.
• Use commit-confirm for any change that reaches a router over the network it is changing. The vyos.vyos collection exposes it as confirm: automatic with a confirm_timeout, and it is off by default.
• Decide whether the pipeline saves, and be consistent about it. An estate where running and saved configurations silently disagree is an estate where rebooting a router changes its behaviour, and nobody will remember that at 03:00.
• Treat the additive-versus-authoritative choice as a blast-radius decision with its own review. It is not an implementation detail of the deploy script; it is the answer to “what can this pipeline delete”, and the answer changed silently here.

Cross-course references

The Ansible course’s LVI-Ansible-Breakfix carries the same class of failure at the inventory layer, where a change that adds no code doubles the number of hosts a play writes to. The Linux course’s XXII-Linux-NetTroubleshoot covers the kernel FIB commands used in verification. Within this course, vyos-liv-01-vyos-http-api covers the /config endpoint’s four operations, vyos-liv-03-config-as-code covers the render-and-apply pipeline, vyos-liv-06-automation-anti-patterns names the manual-override anti-pattern that armed this failure, and vyos-vi-04-rollback covers why rollback is a whole-configuration operation.