Configuration Syntax Error Rejected at Commit

The incident

A security advisory has landed and the response is to enable GTSM on every eBGP transit neighbour in the estate. Eight border routers, one neighbour each, three lines per router, one maintenance window starting at 02:00.

Seven routers take it without comment. border-07 does not. The commit is rejected, and the message names ebgp-multihop — a node nobody typed tonight.

By 03:10 the window is more than half gone and the team has done all of this:

• Discarded the candidate, exited configure mode, re-entered and retyped the three lines by hand. Identical failure.
• Run compare and read it out loud to a second engineer. Three lines, all under ttl-security, exactly matching the seven routers that worked.
• Checked the change against the lab router, where it passes.
• Proposed save followed by a reboot, on the theory that the configuration database is corrupt.
• Proposed commit-confirm, on the theory that it might get further.
• Started drafting an escalation that describes border-07 as “unable to accept configuration changes”.

Every one of those actions is reasonable given the assumption they share, and the assumption is wrong. Nobody has read the running configuration for that neighbour.

Evidence

vyos@border-07# commit

[ protocols bgp neighbor 192.0.2.2 ]
ebgp-multihop and ttl-security are mutually exclusive

Commit failed

vyos@border-07# compare

[edit protocols bgp neighbor 192.0.2.2]
+ttl-security {
+    hops 1
+}

vyos@border-07:~$ show configuration commands | match 'neighbor 192.0.2.2'

set protocols bgp neighbor 192.0.2.2 remote-as '65100'
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast
set protocols bgp neighbor 192.0.2.2 update-source '198.51.100.9'
set protocols bgp neighbor 192.0.2.2 ebgp-multihop '2'

vyos@border-07:~$ show log commit | last 5

configd: commit by netops from 203.0.113.40 (pts/1)
configd: validation failed
configd: rollback to previous candidate
configd: unlock

The session that this change was going to protect is, meanwhile, entirely healthy:

vyos@border-07:~$ show bgp summary

Neighbor        V   AS  MsgRcvd  MsgSent  Up/Down  State/PfxRcd
192.0.2.2       4 65100   418293   211044  14w2d      842104

And the reason the peer needs multihop in the first place:

vyos@border-07:~$ traceroute 192.0.2.2
 1  198.51.100.10
 2  192.0.2.2

Work the evidence before reading on

Five facts, and the useful one is the one that was never treated as evidence.

1. The change is byte-identical to seven changes that succeeded. So the change is not malformed.
2. Retyping it produces the same failure. So it is not a typing slip, a stray character or a paste artefact.
3. The error names a node that is not in the diff. Error messages are not usually random, and this one is naming something specific.
4. compare is clean. The team treated this as proof that nothing is wrong; it is only proof that nothing is wrong in the delta.
5. The router is otherwise completely healthy — the session it is about to be reconfigured for has been up for fourteen weeks.

Before continuing: compare shows what you are changing. What does the validator look at — and if those two are different, where would a conflict live that compare structurally cannot show you?

Root cause

1. This is a validation conflict, not a syntax error

Everyone in the incident called it a syntax error, and the word is what kept the search inside three lines for an hour. A syntax error is a statement about the form of a value: an address out of range, an AS number outside 32 bits, a malformed MAC.

This is a different kind of failure. Both lines are individually well-formed. The candidate as a whole is not valid, because it asserts two things about the same neighbour that cannot both be true.

ebgp-multihop and ttl-security are two ways of describing the same property — how far away the peer is allowed to be, in TTL terms. The VyOS tree documents them as mutually exclusive on a neighbour, and the validator enforces it. A commit failure here is not evidence that the operator mistyped; it is evidence that the operator chose a model that conflicts with the model already in place.

2. compare and the validators look at different things

This is the mechanical heart of the incident.

• compare answers “what am I changing?” It prints the delta between the candidate and the active configuration. Three lines in, three lines out.
• The validators answer “is the result valid?” They run over the entire candidate, which is the active configuration with the delta applied on top. Roughly nine hundred lines here, not three.

A conflict between a new line and a line that has been in the running configuration since the router was built therefore lives in a region that compare structurally cannot show. It is not hidden — a single show configuration commands | match 'neighbor 192.0.2.2' prints it — but it is invisible in the one view the team spent the hour reading.

3. Why this router and not the other seven

border-07’s transit hand-off is not directly connected. The provider’s BGP speaker is one device further away, which the traceroute shows in two hops. A directly connected eBGP session uses a TTL of 1 and would never have established across that intermediate device, so ebgp-multihop 2 was set at build time and has been carrying the session ever since.

The other seven routers hand off on a directly connected link. They have no multihop node, so there is nothing for GTSM to conflict with, and the change lands cleanly. The lab router is built from the same template as those seven — which is why the lab test passed and proved nothing about border-07.

There is a second consequence of the same mismatch, and it is worth noticing because it survives the fix. The change as written sets hops 1, which is correct for a directly connected peer and wrong for this one whatever happens to the multihop node. Even if the two nodes could coexist, this router would need a different value. The conflict is what made the change stop; the wrong hop count is what would have made it fail quietly.

The change was written for one topology and rolled at a site that has a different one. That is the finding, and it is a change-management finding rather than a router finding.

4. What the commit log tells you that the terminal does not

show log commit records the failure as a validation failure with a rollback to the previous candidate. That single word matters operationally, and the next section is about why.

Resolution

1. Read the active configuration for the node the error named: show configuration commands filtered to that neighbour. This is the step that ends the incident, and it should have been the first one.
2. Establish whether the existing node is load-bearing before considering removing it. Here, traceroute to the peer address shows two hops, so ebgp-multihop is carrying the session and is not vestigial configuration.
3. Decide which of two paths you are on, and say so out loud in the incident channel. Path A: this router is out of the change scope. Path B: GTSM is required here and the TTL contract must be rewritten. Do not drift between them.
4. Path A — discard, exit configure mode, and change nothing. Record an exception for border-07 against the advisory, with a named owner and an expiry date. An emergency change that skips one of eight routers is a decision only if it is written down; otherwise it is an omission that nobody will find until the next audit.
5. Path B — measure the hop count with traceroute rather than inheriting the number already configured, then in a single commit delete protocols bgp neighbor 192.0.2.2 ebgp-multihop and set protocols bgp neighbor 192.0.2.2 ttl-security hops 2. One commit, so the candidate never contains both and never contains neither.
6. Path B — treat it as service-affecting. Changing the TTL contract resets the session and withdraws every prefix from that peer until it re-establishes. Get a window, tell the peer NOC, and confirm the backup path can carry the site while the session is down.
7. Path B — prepare the rollback before committing: restoring ebgp-multihop 2 and deleting ttl-security, again in one commit. commit-confirm is worth using here, unlike during the validation failure, because this change can actually take the session away.
8. Either path — prove the router is committable. Make a trivial unrelated change, commit it, confirm success in show log commit, then revert it and commit again.

Verification

1. The router accepts a commit. A trivial interface-description change commits and reverts cleanly, and show log commit | last 5 shows two successful commits. This is thirty seconds of work and it retires the "border-07 cannot accept configuration changes" escalation.
2. Path A — compare is empty and show log commit records no successful configuration commit on this router during the window. Nothing was changed by accident while the team was experimenting.
3. Path A — the neighbour is still Established with the same prefix count as before the window, and the exception is recorded against the advisory with an owner and an expiry date. An exception with no expiry is a permanent gap wearing a temporary label.
4. Path B — the hop count configured matches a measured traceroute, not the previous configuration value. Inheriting the number is how a wrong number survives a change intended to fix it.
5. Path B — the session re-establishes and then survives longer than the negotiated hold time. Reaching Established proves the TTL contract is acceptable to both ends; surviving a hold interval proves keepalives are getting through.
6. Path B — the received prefix count matches the peering agreement, and a sample of received prefixes appears in show route with the expected next-hop rather than only in the BGP RIB.
7. Path B — traffic moves. Confirm the transit circuit is carrying its share before closing, because a session that is Established and carrying nothing is a different fault with the same green status line.
8. Either path — the write-up names the layer that failed. show log commit said validation, which means the running configuration never moved, which is the fact that makes the reboot proposal retrospectively unnecessary and worth recording.

Prevention

• Fix the mental model, because the tooling is behaving correctly. compare answers “what am I changing”; the validators answer “is the result valid”. Those are questions about different amounts of configuration, and a conflict with a year-old line lives only in the second one.
• When a commit error names a node that is not in your diff, read the active configuration for that node before re-reading your own change. The message has already told you where to look. An hour was spent proving three lines were correct, which they were.
• Put it in the pre-change checklist in those words: grep the running configuration for every node your change interacts with, not only for the node you are setting.
• Gate estate-wide changes on a pre-flight query that finds the routers where the preconditions do not hold. A change written for directly connected peers should have been paired with a search for ebgp-multihop across all eight routers before anyone opened a configure session at 02:00. That search takes a minute and would have turned an incident into a line in the plan.
• Never resolve a rejection by deleting whichever node the error names, and be most careful about it exactly when the deletion is the fastest way to make an emergency change proceed. Time pressure is what makes this failure mode dangerous rather than merely wrong.
• Read show log commit to identify the failing layer before choosing a recovery. A validator failure is safe to retry and leaves nothing behind; a generator or runtime failure can leave half-applied state and has to be inspected first.
• Record exceptions with an owner and an expiry. Seven of eight is a perfectly good outcome for one window and a bad outcome for a quarter, and the only thing that distinguishes them is whether somebody wrote down which router was skipped.

Related material

• vyos-vi-01-commit-semantics — the commit phases, and the distinction between validator, generator and runtime failures.
• vyos-iii-04-commit-engine — the verify, render, diff and apply pipeline, and why a successful commit is not a working configuration.
• vyos-v-04-show-and-compare — what compare shows and what it does not.
• vyos-xxiv-05-ebgp-multihop and vyos-xxiv-04-bgp-authentication — the multihop design and the GTSM alternative that conflicts with it.
• vyos-vi-02-commit-confirm — where commit-confirm genuinely helps, which is a change that can succeed and cut you off, not one that is rejected before it is applied.