OSPF Neighbours Stuck in EXSTART/EXCHANGE

Reported symptoms

Three teams are working three tickets, and none of them has mentioned the others.

The server team has a ticket from an engineer who cannot get a useful SSH session to any host on the DR management network. The session connects. The banner prints. The prompt appears. Then the first command that produces more than a screen of output hangs and never returns. Smaller commands are fine. The engineer has tried three hosts and two jump boxes and concluded the DR site has a problem.

The backup team has a ticket that the nightly configuration archive to the DR site has failed on each of the last three nights. The job connects, transfers something, and stalls. It was fine on Wednesday.

The network team has a ticket that the DC-hosted ERP is slow. Not down. Slow. Users can work; page loads take several seconds instead of under one. Someone has already established that the traffic is crossing the internet IPsec tunnel rather than the site-to-site circuit, which explains the latency, and has filed it as a routing question.

The routing question turns out to be the interesting one. The area 0 adjacency between rtr-core-1 and rtr-dc-2 is not Full. It is also not consistently anything else:

• 09:12 — show ip ospf neighbor reports EXSTART.
• 09:13 — the same command reports EXCHANGE.
• 09:15 — EXSTART again.

The second adjacency on the same physical circuit, to rtr-dr-3, has been Full for eighteen months and has not moved during any of this.

Neither router has been committed to in five weeks. The only change anywhere near this is the carrier work over the weekend, which replaced the dark fibre between the two sites with a managed Ethernet service. The window closed at 03:40 on Saturday. The hand-back test was an interface check and a ping, both of which passed, and the change was signed off as a like-for-like replacement.

The carrier portal shows the circuit up with no errors, and their first line will not escalate a circuit that is up.

The estate

rtr-core-1 is a VyOS 1.5 LTS router at the HQ campus. eth1 faces the site-to-site circuit as a trunk carrying two VLANs:

• eth1.30, 198.51.100.5/30, to rtr-dc-2 at 198.51.100.6. This is area 0 — the backbone, several hundred LSAs, and the path the DC prefixes are learned over.
• eth1.31, 198.51.100.9/30, to rtr-dr-3 at 198.51.100.10. This is area 3, configured as a stub, holding the DR management network and a few dozen LSAs.

Router IDs are 192.0.2.1, 192.0.2.2 and 192.0.2.3. Every interface in the estate is MTU 1500 and has been since it was built.

There is a floating static route to the DC prefixes over an internet IPsec tunnel, with an administrative distance high enough that it only takes effect when OSPF has nothing to offer. That tunnel has MSS clamping configured on it. Nobody currently on the team knows who added the clamping or why; it was there when they arrived.

Evidence provided

vyos@rtr-core-1:~$ show ip ospf neighbor

Neighbor ID     Pri State           Dead Time Address         Interface
192.0.2.2         1 ExStart/DROther   38.402s 198.51.100.6    eth1.30
192.0.2.3         1 Full/DROther      37.911s 198.51.100.10   eth1.31

vyos@rtr-core-1:~$ show ip ospf neighbor detail

Neighbor 192.0.2.2, interface address 198.51.100.6
In the area 0.0.0.0 via interface eth1.30
Neighbor priority is 1, State is ExStart, 47 state changes
Interface MTU 1500, Neighbor Interface MTU 1500
Database Summary List 0
Link State Request List 0

vyos@rtr-core-1:~$ ping -M do -s 1472 -c 3 198.51.100.6

PING 198.51.100.6 (198.51.100.6) 1472(1500) bytes of data.

--- 198.51.100.6 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

vyos@rtr-core-1:~$ ping -M do -s 1414 -c 3 198.51.100.6

PING 198.51.100.6 (198.51.100.6) 1414(1442) bytes of data.
1422 bytes from 198.51.100.6: icmp_seq=1 ttl=64 time=1.11 ms

--- 198.51.100.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms

vyos@rtr-core-1:~$ tcpdump -ni eth1.30 proto ospf

09:31:02.114 IP 198.51.100.5 > 224.0.0.5: OSPFv2, Hello, length 48
09:31:02.219 IP 198.51.100.6 > 224.0.0.5: OSPFv2, Hello, length 48
09:31:04.882 IP 198.51.100.5 > 198.51.100.6: OSPFv2, Database Description, length 32
09:31:04.901 IP 198.51.100.6 > 198.51.100.5: OSPFv2, Database Description, length 32
09:31:05.117 IP 198.51.100.5 > 198.51.100.6: OSPFv2, Database Description, length 1452
09:31:10.118 IP 198.51.100.5 > 198.51.100.6: OSPFv2, Database Description, length 1452

The same capture running on rtr-dc-2 at the same moment shows the Hellos and the 32-byte Database Description packets arriving, and no sign of the 1452-byte ones.

Work the evidence before reading on

Nothing on either router is misconfigured, and both routers agree about everything they can see.

1. show ip ospf neighbor detail reports the same Interface MTU on both sides. That single line eliminates the failure everyone reaches for first. What class of MTU problem is left when both ends agree?
2. The adjacency is EXSTART sometimes and EXCHANGE at others. Ask what is different about the packets sent in each of those states, rather than what is different about the states.
3. The second adjacency on the same wire is Full. What property of an area — not of a link — decides how large a Database Description packet is?
4. ping -M do gets silence rather than an ICMP Fragmentation Needed message, and ping -M dont fails too. The standard PMTUD black hole is a router that drops the packet and has its ICMP filtered. What kind of device drops an oversized frame and could not send ICMP even if nobody filtered it?
5. Three tickets: an adjacency that will not form, SSH sessions that hang once output grows, and a file-copy job that stalls. What single quantity, if it were smaller than everyone believes, produces all three?

Before continuing: both routers say 1500, and one of them is describing a link that no longer exists.

Root cause

1. The circuit is not what it replaced

The managed Ethernet service carries the customer frame inside the carrier network, and that encapsulation costs bytes. The circuit delivers 1442 usable bytes. The dark fibre it replaced delivered 1500. Nothing in the hand-back test measured the difference, because the only packets it sent were small.

The routers were not told and cannot infer it. Both still claim MTU 1500 in their configuration, in show ip ospf interface, and in the Interface MTU field of every Database Description packet they send. Their agreement is genuine and completely irrelevant: they agree about a link neither of them can measure.

2. EXSTART and EXCHANGE are one failure caught twice

The OSPF adjacency states are not arbitrary; each one sends a different size of packet.

• Hellos are tens of bytes. They cross. The neighbours find each other and reach 2-Way.
• The Database Description packets that negotiate master and slave are effectively empty, around 32 bytes. They cross, so the adjacency reaches EXSTART and gets past it.
• The Database Description packets that follow carry the LSA headers, twenty bytes each, and are filled up to the interface MTU. On the backbone, with several hundred LSAs, that means packets close to 1500 bytes.

Those are the packets the circuit discards. The exchange cannot complete, the sequence number never advances, and the adjacency resets and starts again. Poll it during the negotiation and it reads EXSTART; poll it during the doomed exchange and it reads EXCHANGE. The oscillation is not a second symptom to explain. It is the retry loop.

3. The stub area is the control experiment

rtr-dr-3 sits on the same physical circuit and is Full, which is the fact that makes the circuit look innocent.

It is in a stub area. A stub area does not carry the external LSAs that dominate a backbone LSDB, so its database is a few dozen entries rather than several hundred. Its Database Description packets are a few hundred bytes, and a few hundred bytes fit inside 1442.

That adjacency is not evidence that the circuit is healthy. It is evidence that the failure threshold is somewhere between the size of a stub area database and the size of the backbone one — which is a statement about the circuit, and one that would have been read that way by anyone who knew both areas were crossing the same wire.

4. The data-plane tickets are the same number

TCP endpoints on both sides derived their maximum segment size from their own 1500-byte interfaces. An SSH server sending a page of output fills a segment, the resulting packet exceeds 1442 bytes, and the circuit discards it. The session does not reset; it hangs, because TCP retransmits a segment that will never fit. The archive job stalls for the same reason on its first large write.

The ERP is slow rather than broken because OSPF failing on this path let the floating static route take over, and the IPsec tunnel it points at has MSS clamping configured. Somebody solved this problem once, on one interface, and the fix stayed where they put it.

Resolution

1. Measure before you change anything. Binary-search the payload with ping -M do -s between the two router interfaces until you have the largest size that crosses. Path MTU is that payload plus 28. Everything below is derived from this number, and a fix applied without it is a guess.
2. Put the number in the ticket and in the circuit record. A carrier fault that says the link is broken gets closed as no fault found; one that says the circuit delivers 1442 bytes where 1500 was ordered does not.
3. Raise the fault with the carrier in parallel with anything you do locally. Raising the circuit MTU so a 1500-byte customer frame survives transport is the fix that restores the estate to the MTU it was designed around, and it is theirs to make.
4. Decide explicitly whether to hold. The IGP failure is already being absorbed by the backup path, so the visible cost is latency rather than an outage, and waiting a few hours for the carrier is defensible. A hold needs an owner, an end time, and a message to the server and backup teams so they stop investigating their tickets as separate faults.
5. If you are fixing it locally, lower the interface MTU on both sub-interfaces to the measured value, in one window. set interfaces ethernet eth1 vif 30 mtu 1442 on each end. One side alone recreates the classic mismatch and sticks the adjacency in EXSTART for a different reason.
6. Understand what you have bought. Everything crossing that link is now limited to 1442 bytes. In exchange, the routers become IP hops with a smaller MTU, which means they generate ICMP Fragmentation Needed and turn a silent layer-2 black hole into a bottleneck that Path MTU Discovery can actually see.
7. Add MSS clamping on the interface with the ip adjust-mss node so TCP does not depend on ICMP that some firewall between here and the far end may be dropping. Set it to the path MTU minus 40.
8. Remove the triage mtu-ignore. It changed nothing here, and leaving it behind disables the check you will want the next time two ends genuinely disagree.
9. Use compare to read the diff and commit-confirm to commit it. This is an MTU change on the interface carrying the IGP; if the number is wrong you want the router to undo it without you.

Verification

1. The adjacency is Full and stays Full. Check show ip ospf neighbor twice, several minutes apart and longer than the dead interval. An adjacency caught between two resets looks healthy for a moment.
2. The databases agree. show ip ospf database on both routers shows the same LSAs. A Full adjacency with a divergent LSDB means the exchange completed and flooding is still losing packets, which is the same cause not yet fully fixed.
3. The failure mode of the ping has changed, not just its result. An oversized probe should now return an ICMP Fragmentation Needed message naming the new MTU instead of vanishing. Silence becoming a named error is the evidence that the black hole is gone.
4. The data plane is proven from a client, not from the router. Open an SSH session to a DR host and run something with a long output. Run the archive job against a real file rather than a test byte.
5. The routing has returned to the IGP. show ip route shows the DC prefixes learned by OSPF rather than via the floating static. The ERP being fast again is the symptom the business actually reported.
6. Interface counters on both routers show no errors introduced by the change, read after the link has carried real traffic for a while rather than immediately after commit.
7. If the carrier raised the circuit MTU instead, their claim is verified with your own ping ladder and not with their ticket update - and only then is any local MTU reduction reverted.

Prevention

• Make path MTU an acceptance test for every circuit hand-off. The test that was run here — interface up, ping succeeds, no errors — measures almost nothing, because a 56-byte packet crosses a badly degraded path happily. The ping -M do -s ladder takes a minute and produces a number. That number belongs in the circuit record beside the bandwidth, so the next engineer inherits it instead of rediscovering it at 09:00 on a Monday.
• Treat “like-for-like” as a supplier claim, not a measurement. It was true about the endpoints, the media and the bandwidth. It was false about the only property that mattered, and nobody involved was lying.
• Alert on OSPF neighbour state. Any adjacency not Full for more than a couple of polling intervals deserves a notification. The course covers reading FRR JSON output, which makes this a few lines of scripting; it would have fired on Saturday night while the window was open and the carrier engineer was still on the bridge, instead of surfacing on Monday as three tickets in three queues.
• Clamp MSS on any link whose MTU is not provably 1500 end to end. The estate already contained this fix on one tunnel. Fixes that live on one interface because one person once had a bad afternoon are not controls.
• Do not reach for mtu-ignore first. It is the right tool when two ends must run different MTUs and you have independently established that nothing larger than the smaller one needs the link. Used as a reflex it skips the measurement that tells you which of the two MTU failures you have.