Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-ospf~30 min

OSPF Area Mismatch

Reported symptoms

  • ●Users at branch site 14 cannot reach the HQ file server, the print queues or the intranet, and have not been able to since Friday morning
  • ●The ERP at the DC is reachable from the same site but noticeably slow, so the service desk is carrying two tickets against one site at two different priorities
  • ●Internet browsing from site 14 is completely unaffected, which is what convinced the first responder that the WAN circuit was healthy
  • ●Monitoring shows rtr-br-14 up and answering ICMP; its OSPF neighbour check has been red since Thursday 21:40 but sits inside a maintenance suppression that was opened for the rebuild and never closed
  • ●The engineer who rebuilt the site 14 router on Thursday evening wrote in the handover note that the neighbour table was empty at the end of the build and that OSPF was expected to converge overnight
  • ●The WAN carrier ran scheduled maintenance on the branch aggregation circuit on Thursday night, which is the change every responder has reached for first

Evidence

  • · `show ip ospf neighbor` on rtr-br-14 prints the column header and nothing under it - no Init entry, no ExStart entry, no entry at all
  • · `show ip ospf neighbor` on the aggregation router rtr-agg-3 lists eleven branch adjacencies in Full and has no line for router-id 192.0.2.114
  • · `show interfaces ethernet eth0` on rtr-br-14 shows the WAN tail up, no input or output errors, and packet counters advancing in both directions
  • · `ping 198.51.100.129` from rtr-br-14 to the aggregation side of the tail returns with zero loss, so the circuit is carrying unicast IP in both directions
  • · `tcpdump -ni eth0 proto ospf -vv` on rtr-agg-3 shows Hello packets arriving from 198.51.100.130 every ten seconds, so nothing is filtering or losing them
  • · The same capture prints Area 0.0.20.0 on the Hellos arriving from the branch and Area 0.0.0.20 on the Hellos the aggregation router sends
  • · `show ip ospf interface eth0` reports Area 0.0.20.0 on rtr-br-14; the matching command on rtr-agg-3 reports Area 0.0.0.20
  • · `show ip route ospf` on rtr-br-14 returns nothing at all, while `show ip route` shows a static default to the local breakout and two floating statics over the IPsec tunnel
  • · `show system commit` on rtr-br-14 lists exactly one commit, timestamped Thursday evening; rtr-agg-3 has not been committed to in three weeks
Diagnosis and resolutionclick to reveal

Root cause

The rebuilt branch router was configured with its OSPF area written as 0.0.20.0 where the branch area is 0.0.0.20. An OSPF Area ID is a 32-bit field, so both are syntactically valid and neither is a typing error the parser can catch: 0.0.0.20 is area 20 and 0.0.20.0 is area 5120. The configuration committed cleanly, FRR started an area with one interface and one router in it, and every command on the branch router reported a healthy OSPF process. RFC 2328 requires a receiving router to check the Area ID in the OSPF header against the area configured on the receiving interface and to discard the packet when they differ, so the Hellos crossed the circuit in both directions and were thrown away at both ends. That is why the neighbour table is empty rather than stuck: no neighbour data structure is ever created, because the packet is rejected before the neighbour state machine is reached. The rest of the incident is the estate absorbing the failure unevenly. The branch has a static default route to its local internet breakout at administrative distance 1, so internet access never depended on the IGP and never noticed. Floating statics over the IPsec tunnel cover the DC supernet, so the ERP kept working over a slower path. Nothing covers the HQ campus prefixes, which were only ever learned by OSPF, so those went dark. Three populations of traffic, three different outcomes, one missing adjacency.

Remediation

The correction is one area token on one router, and the discipline around it matters more than the edit. Move the branch interface and the branch LAN networks out of area 0.0.20.0 and into area 0.0.0.20, delete the orphaned area so the router is not left holding a stub area with no members, read the result with `compare` before committing, and use `commit-confirm` because you are changing the routing protocol on a router you can only reach over the backup path. Then decide deliberately whether to commit now or tonight, because this fix moves live traffic. The moment the adjacency comes up, the branch LAN prefixes enter the IGP, the core prefers the private WAN over the floating statics, and every established session that is currently traversing the IPsec tunnel changes path. Sessions that survive a path change will survive it; long-lived TCP connections pinned by the tunnel endpoint may not. Holding until the evening is a legitimate choice and should be recorded as one: the site is degraded rather than down, users at site 14 have a working internet breakout and a slow ERP, and the cost of waiting is measured in frustration rather than in outage minutes. A hold needs a named owner, an end time, a note on both service-desk tickets so they stop being worked as separate faults, and the monitoring suppression lifted immediately so the alert that has been red since Thursday is visible to whoever picks the ticket up. What is not defensible is changing anything on the aggregation router or raising a fault with the carrier: eleven other branches prove that both are working.

Verification

Check the adjacency from both ends, because a one-sided check is the specific omission that created this incident. `show ip ospf neighbor` on rtr-br-14 must show rtr-agg-3 in Full, and the same command on rtr-agg-3 must show twelve branch adjacencies rather than eleven. Run it twice, several minutes apart and longer than the dead interval, so an adjacency caught mid-cycle cannot be mistaken for a stable one. Confirm the databases agree: `show ip ospf database` on the branch must now hold the area LSAs and the Type-3 default the ABR injects into the stub area, and `show ip route ospf` must return the HQ campus prefixes it returned nothing for before. Verify from the core that traffic actually moved, not merely that a route exists - `show ip route` for the site 14 LAN supernet on a core router should show the OSPF path rather than the floating static, which is the real statement that the backup path has been released. Prove the user-visible symptom rather than the router state: open the HQ file server and a print queue from a workstation at the site. Finally, close the loop on monitoring: lift the maintenance suppression, confirm the neighbour check goes green on its own, and confirm the neighbour count on the aggregation router reads twelve, because that counter is the thing that should have caught this on Thursday night.

Prevention

A router build is not finished when the configuration commits; it is finished when the expected number of adjacencies is Full on both ends. The handover note that said OSPF was expected to converge overnight is the artefact of a build checklist with a step for configuring the IGP and no step for proving it, and it is worth fixing the checklist rather than the engineer: with hello and dead intervals of ten and forty seconds, an OSPF adjacency either forms in under a minute or is not going to. There is no state in which it is legitimately pending. Alert on OSPF neighbour count per device and treat a drop as a page rather than a dashboard entry; the course covers the FRR JSON output that makes this a few lines of scripting. Give every monitoring suppression a mandatory expiry, because a suppression that outlives its change window is functionally a deleted alert, and this one hid a correct alarm for four days. Write area identifiers one way in the design standard and prefer the plain decimal form: 20 cannot be transposed into another valid area, while 0.0.0.20 can be transposed into three of them and every one will commit. Build branch routers from a parameterised template with generated values rather than from a sibling configuration edited by hand, which is what the automation and config-as-code parts of the course are for, and make the post-build validation compare rendered operational output against the expected state rather than comparing configuration text against a document.

Reported symptoms

It is Monday morning and branch site 14 has two tickets open, at two different priorities, filed by two different people.

The first says the HQ file server is unreachable. So are the print queues, the intranet and the internal wiki. Everything at the HQ campus, from every machine at the site, since Friday morning at the latest — one user thinks it may have been broken on Thursday evening but was not sure enough to call.

The second says the ERP is slow. The ERP is hosted at the DC, not at HQ. It works. Page loads take four or five seconds instead of well under one. It has been filed as a performance complaint and routed to the application team.

Internet browsing from the site is fine. Video calls are fine. That single fact is why the first responder closed out the possibility of a WAN problem within about ninety seconds and started looking at HQ.

Two more things are true and nobody has connected them yet.

The site 14 router was replaced on Thursday evening. The original died of a power supply failure on Wednesday; a spare was couriered out, built on site, and handed over at 21:40. The handover note is one line: OSPF neighbour table empty at handover, expect convergence overnight.

And the WAN carrier ran scheduled maintenance on the branch aggregation circuit that same night. That is the change everybody has reached for, and a fault has already been raised against it.

The estate

rtr-agg-3 is the branch aggregation router at the regional hub. eth0 faces the core in area 0. eth1 is a trunk toward the branch WAN with one sub-interface per site, and every one of those sub-interfaces is in area 0.0.0.20, which is configured as a stub area. Twelve branches hang off it.

rtr-br-14 is the replacement router at site 14, a VyOS 1.5 LTS install built on Thursday. eth0 is the WAN tail, 198.51.100.130/30, facing 198.51.100.129 on rtr-agg-3. eth1 carries the site LANs, summarised as 10.14.0.0/16. Its router-id is 192.0.2.114.

Three things about the branch design matter here, and all three predate the incident:

  • Every branch has its own internet breakout and a static default route to it at administrative distance 1. That static beats the OSPF default the ABR injects into the stub area, so branch internet access has never depended on the IGP.
  • Every branch has an IPsec tunnel to the DC carrying floating static routes for the DC supernet, at a distance high enough that OSPF wins when OSPF has something to offer.
  • The HQ campus prefixes are learned only through OSPF. Nobody ever wrote a backup static for them, because the private WAN was considered the primary and the tunnel was built for DC replication.

Evidence provided

Read-only / Safea header and nothing under it - not Init, not ExStart, nothing
vyos@rtr-br-14:~$ show ip ospf neighbor
Neighbor ID     Pri State           Dead Time Address         Interface

Illustrative output

Read-only / Safeeleven branches Full, and the design says twelve
vyos@rtr-agg-3:~$ show ip ospf neighbor
Neighbor ID     Pri State           Dead Time Address         Interface
192.0.2.106       1 Full/DR          36.117s 198.51.100.114   eth1.6
192.0.2.111       1 Full/DR          38.902s 198.51.100.126  eth1.11
192.0.2.115       1 Full/DR          33.480s 198.51.100.134  eth1.15
... eight more, all Full, none of them site 14

Illustrative output

Read-only / Safethe tail carries unicast IP in both directions
vyos@rtr-br-14:~$ ping -c 5 198.51.100.129
PING 198.51.100.129 (198.51.100.129) 56(84) bytes of data.
64 bytes from 198.51.100.129: icmp_seq=1 ttl=64 time=3.42 ms

--- 198.51.100.129 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms

Illustrative output

Read-only / Safeboth sides are talking; read the Area field on each line
vyos@rtr-agg-3:~$ tcpdump -ni eth1.14 proto ospf -vv
10:22:31.417 IP 198.51.100.130 > 224.0.0.5: OSPFv2, Hello, length 48
Router-ID 192.0.2.114, Area 0.0.20.0, Authentication Type: none (0)
10:22:34.902 IP 198.51.100.129 > 224.0.0.5: OSPFv2, Hello, length 48
Router-ID 192.0.2.103, Area 0.0.0.20, Authentication Type: none (0)

Illustrative output

Read-only / Safethe branch router is perfectly happy, in an area of one
vyos@rtr-br-14:~$ show ip ospf interface eth0
eth0 is up
Internet Address 198.51.100.130/30, Area 0.0.20.0
MTU mismatch detection: enabled
Router ID 192.0.2.114, Network Type BROADCAST, Cost: 10
Timer intervals configured, Hello 10s, Dead 40s
Neighbor Count is 0, Adjacent neighbor count is 0

Illustrative output

show ip route ospf on the branch returns nothing. show system commit shows a single commit, Thursday at 21:31.

Work the evidence before reading on

Everything on this router is up, and the router has no complaint to make about anything.

  1. The neighbour table is empty, not stuck. Compare that with the sibling scenario where an adjacency sits in ExStart. What does each of those two shapes tell you about how far the protocol got before it gave up?
  2. The Hellos arrive. The tcpdump proves it, and it costs one command. What half of the search space does that single result eliminate, and what is left in the other half?
  3. OSPF has a short list of parameters that must match before a neighbour structure is created at all. Name them, then work out which of them you can read directly off the capture you already have.
  4. Internet works, the ERP is slow, the HQ prefixes are dead. Three populations of traffic, three outcomes. What does each one depend on, and what single missing thing produces all three?
  5. Eleven other branches on the same aggregation router, over the same carrier maintenance window, are fine. What does that do to the carrier hypothesis, and why has nobody used it yet?

Before continuing: an empty neighbour table is not an absence of information. It is the single most specific state OSPF has.

Root cause

1. Two valid areas, one character apart

The branch area is 0.0.0.20. The rebuilt router was configured with 0.0.20.0.

The Area ID in the OSPF header is a 32-bit field. VyOS accepts it written as a decimal or as a dotted quad, and FRR renders it back as a dotted quad. 0.0.0.20 is area 20. 0.0.20.0 is area 5120. Both are legal. Neither is a syntax error, so the commit succeeded and the validation the engineer ran — show ip ospf interface, which printed an area and an interface that was up — returned exactly what he expected to see.

The router then did the only sensible thing available to it: it started area 5120, put one interface in it, and waited for somebody else to show up.

2. Why the table is empty rather than stuck

RFC 2328 has the receiving router validate the OSPF header before anything else happens, and the Area ID is part of that validation. A packet whose Area ID does not match the area configured on the receiving interface is discarded there and then.

That is the whole mechanism, and its consequence is the most useful diagnostic signal in this incident. The packet never reaches the neighbour state machine, so no neighbour structure is created, so there is nothing to print. Every state after the header check — Init, 2-Way, ExStart, Exchange, Loading — exists only because a Hello was accepted. An adjacency that reaches any of them has already proved that the area matches.

So the states form a sieve, and where the adjacency stops tells you which check rejected it:

  • Nothing in the table at all. Either no Hello is arriving (the interface is not in OSPF, or is passive, or something is filtering multicast to 224.0.0.5), or Hellos are arriving and failing header or parameter validation: Area ID, hello and dead intervals, authentication, or the E-bit and N-bit that encode area type.
  • Init. Hellos are arriving and being accepted, but the neighbour is not listing our router-id back. The path is one-way.
  • ExStart or Exchange. Every parameter agreed. The Database Description exchange is failing — MTU, a duplicate router-id, or a path that cannot carry the packet.

One tcpdump splits the first bullet in half, which is why it is the cheapest check available and should have been the second command anybody ran. The Hellos are arriving. That kills the carrier fault, the filtering hypothesis and the passive-interface hypothesis in one capture, and leaves parameter validation — a short, enumerable list, most of which is printed in the same capture.

3. The estate hid it for four days

The three symptoms are three different dependencies on the missing IGP.

Internet traffic follows a static default at distance 1. It has never depended on OSPF and did not notice. The DC prefixes are covered by floating statics over the IPsec tunnel, which is exactly what a floating static is for: when OSPF has nothing to offer, they install, and traffic keeps flowing over a longer, slower path. That is the ERP ticket, and the application team will never find anything wrong with the application. The HQ campus prefixes have no backup at all, so they simply stopped existing at site 14.

Resolution

  1. Read the area off both routers before touching anything. show ip ospf interface eth0 on the branch and on the matching sub-interface at the hub. The correction is derived from the hub value, not from the design document, because the eleven working branches are the authority on what the area actually is.
  2. Decide now whether you are fixing this minute or holding until the evening, and write the decision down. Bringing the adjacency up moves live traffic off the IPsec tunnel and onto the private WAN, which will disturb established sessions. The site is degraded, not down. A hold is defensible; an undeclared hold is not.
  3. If you hold, lift the monitoring suppression immediately, name an owner and an end time, and annotate both service-desk tickets so the ERP performance ticket stops being worked as an application fault.
  4. Withdraw the carrier fault. Eleven branches crossed the same maintenance window on the same aggregation router without incident, which is stronger evidence than anything the carrier will produce, and leaving a false fault open costs somebody else a day.
  5. Move the branch interface and the branch LAN networks into the correct area, and delete the orphaned one so the router is not left holding a stub area with no members. Enter configuration mode and stage the change without committing.
  6. Read the change with compare before you commit it. This is a routing-protocol change on a router reachable only over the backup path; the diff is the last cheap opportunity to notice a second transposed digit.
  7. Commit with commit-confirm and a short window. If the area value is wrong a second time the router reverts on its own, which matters more than usual here because a branch router that loses both paths is a site visit.
  8. Watch the adjacency come up before you confirm the commit. It should reach Full within a dead interval. If it does not, you have not finished diagnosing.
  9. Save the configuration once the adjacency is stable, and only then close the confirm window.
# On rtr-br-14, in configuration mode. Confirm the area value against
# the hub before pasting anything.
delete protocols ospf area 0.0.20.0
set protocols ospf area 0.0.0.20 network 198.51.100.128/30
set protocols ospf area 0.0.0.20 network 10.14.0.0/16
set protocols ospf area 0.0.0.20 area-type stub
compare
commit-confirm 5

Verification

  1. Check from both ends. show ip ospf neighbor on rtr-br-14 shows rtr-agg-3 in Full, and the same command on rtr-agg-3 shows twelve adjacencies rather than eleven. Checking one end is the exact omission that created this incident.
  2. Check twice, several minutes apart and longer than the dead interval. An adjacency caught between two resets looks healthy for a moment.
  3. The databases agree. show ip ospf database on the branch now holds the area LSAs and the Type-3 default the ABR injects into the stub area. A Full adjacency with an empty database is not finished.
  4. The routes are installed, not merely learned. show ip route ospf on the branch returns the HQ campus prefixes it returned nothing for before.
  5. Traffic actually moved. On a core router, show ip route 10.14.0.0/16 shows the OSPF path rather than the floating static. That is the statement that the backup path has been released, and it is the one that a check run on the branch alone cannot make.
  6. The user-visible symptom is gone, proven from a workstation at the site rather than from a router: open the HQ file server and print a page.
  7. Monitoring is telling the truth again. Lift the suppression, confirm the neighbour check clears on its own rather than because you closed it, and confirm the neighbour count on the aggregation router reads twelve.

Prevention

  • Make the adjacency count the acceptance criterion for a router build. Not “OSPF configured” — the expected number of neighbours Full, confirmed on both ends, pasted into the build record. Every other step in this incident is downstream of a checklist that let a router be handed over unfinished.
  • Alert on OSPF neighbour count per device. A drop is a page, not a dashboard entry. The course covers reading FRR JSON output, which makes this a few lines of scripting, and it would have fired at 21:41 on Thursday while the engineer was still in the car park.
  • Give every monitoring suppression a mandatory expiry. The alert here was correct and it fired on time. A suppression opened for a change window and never closed is functionally a deleted alert, and it cost four days.
  • Write area identifiers one way, and prefer the decimal form. 20 cannot be transposed into another valid area. 0.0.0.20 can be transposed into three of them, and every one of those will commit without complaint.
  • Build branch routers from a parameterised template, not from a sibling configuration edited by hand. The automation and config-as-code parts of the course exist for exactly this, and the validation step should compare rendered operational output against expected state rather than configuration text against a document.
  • Write backup paths down as coverage, not as a list of tunnels. Nobody at this site knew that the DC prefixes had a fallback and the HQ prefixes did not. That asymmetry decided which tickets got raised, which team got them, and how long the incident lasted.