Skip to main content
RunBook Academy

← All runbooks in VyOS

high riskservice affecting~45 min

Runbook: Troubleshoot OSPF Adjacency

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The adjacency is named precisely before any command is run: which router, which interface, which neighbour router-id, and which area it is supposed to be in. "OSPF is down" is a ticket; "R1 eth0 to R2 in area 0 is stuck in ExStart" is a symptom with a first command attached to it.
  • · Access to BOTH ends is arranged, or an owner for the far end is on the call. Nearly every field this runbook checks is a matching requirement between two routers, and a matching requirement cannot be diagnosed from one side. Starting without the far end is how an afternoon disappears.
  • · Whether this adjacency has ever been Full is established, and if so when it stopped. An adjacency that never formed is a configuration mismatch; one that formed and then failed is a change, a link event, or a timer expiry — and those have different first commands.
  • · The scope is established: one adjacency, every adjacency on one interface, or every adjacency on this router. Every adjacency at once points at the router — its router-id, its OSPF process, its control-plane policy — not at any one peer.
  • · The change record for the last 72 hours is checked on both routers, covering interface MTU, area configuration, authentication, timers, network type, router-id and any firewall or control-plane policy change.
  • · The pre-diagnosis state is captured off-box from both ends and attached to the ticket: show ip ospf neighbor, show ip ospf neighbor detail, show ip ospf interface, show ip ospf, and show configuration commands | match ospf.
  • · Out-of-band console access is confirmed if there is any chance the fix will touch the path the operator is connected over. An adjacency fix that turns into an interface MTU change is exactly that case, and an MTU change bounces the interface.
  • · It is agreed up front that clear ip ospf process is not a diagnostic step. It tears down every adjacency on the router, not the one being investigated, and refloods the whole database. It is a deliberate service event with an owner, used when a router-id change requires it and not before.
  • · It is agreed up front that mtu-ignore is a decision, not a fix. It suppresses the symptom of a real MTU disagreement that the data plane still has to live with, so it needs a reason recorded in the ticket and a follow-up owner.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Read the neighbour state from both ends, not one. show ip ospf neighbor on each router. The two ends frequently report different states, and the pair is more informative than either alone — a router that sees the peer in Init while the peer sees nothing at all is a one-way link, which no single-ended view reveals.
  2. 2Classify the state, because the state names the layer. No neighbour at all means the Hellos are not being exchanged or are being discarded. Init means Hellos arrive in one direction only. ExStart means the Hellos matched and the database exchange did not start. Exchange and Loading mean the exchange started and is not completing. Full but flapping is a different fault entirely.
  3. 3Before treating 2-Way as a fault, check the network type and the DR election. On a broadcast segment, two routers that are both DROther hold a Full adjacency with the DR and the BDR and stay in 2-Way with each other by design. show ip ospf interface reports the state and the role; Neighbor Count against Adjacent neighbor count in the detail output is the gap the design intends.
  4. 4For no neighbour at all, work the Hello matching list rather than guessing: area-id, area-type, hello and dead intervals, network type, authentication, subnet mask, and whether either side is passive. Every one of these must agree, every disagreement discards the Hello silently, and none of them produces an error the operator will find by accident.
  5. 5Confirm the Hellos are physically arriving before blaming configuration. A capture filtered to the OSPF protocol on the interface, run on both ends, separates "the peer is not sending" from "the peer is sending and we are discarding". Those two have completely different owners.
  6. 6For a neighbour stuck in ExStart, compare the interface MTU on both sides first. The database description packet carries the interface MTU, and a router discards a packet whose claimed MTU exceeds its own interface MTU, which stalls the master and slave negotiation. This is the dominant ExStart cause by a wide margin.
  7. 7If the MTU agrees and the adjacency is still in ExStart, check the router-ids on both sides. Two routers holding the same router-id cannot complete the negotiation, and the same duplication also produces adjacency flapping elsewhere in the area.
  8. 8For a neighbour stuck in Exchange or Loading, stop looking at configuration and look at the link. The database exchange has started, so the matching requirements are satisfied; what is failing is delivery. Read the interface error and drop counters, and check whether the path is lossy or unidirectional.
  9. 9For an adjacency that reaches Full and does not stay there, treat it as one of three faults and separate them with evidence: a duplicate router-id, dead-interval expiry caused by loss or CPU starvation, or a physical link that is flapping. The FRR log carries duplicate-LSA warnings for the first; the interface counters carry the third.
  10. 10Enable OSPF debug only after the cheaper evidence is exhausted, only on the affected router, and with an explicit end. The debug prints every packet with its fields parsed, which is exactly what is needed for an authentication or database-description fault and is also enough log volume to disturb a busy router.
  11. 11Apply the fix inside configure, with compare read before the commit and commit-confirm 10 for anything that can remove the operator path or bounce an interface. Confirm only after the adjacency has been observed at Full from both ends.
  12. 12If the fix is a router-id change, understand that the configuration change alone does not take effect: the router-id is established when the OSPF process starts, so the change needs a process restart, and the restart is a service event affecting every adjacency on the router. Plan it as such rather than discovering it.
  13. 13Verify from both ends, then verify that routes actually arrived. An adjacency at Full is a means, not an end; the ticket was about reachability, and a Full adjacency with an empty database is the next runbook, not a closed incident.

4 · Verification

Confirm the procedure actually fixed the problem.

  • show ip ospf neighbor on both routers reports the peer in state Full, and the two ends agree on which router holds DR and which holds BDR where the segment elects them.
  • On a broadcast segment, Neighbor Count and Adjacent neighbor count in show ip ospf neighbor detail match the DR design — a gap between them is expected on a segment with more than two routers and is not a fault.
  • show ip ospf interface on both ends shows the same area, the same network type, the same hello and dead intervals, and the same MTU, or an explicitly recorded reason why the MTU differs.
  • show ip ospf on both routers reports different router-ids, and each matches the value the estate register allocates to that router.
  • The link-state database is synchronised: show ip ospf database on both ends holds the same set of LSAs for the area, and neither end is missing LSAs the other has.
  • The routes the adjacency was expected to deliver are present in show ip route ospf and installed in the kernel table, because a Full adjacency that delivers no routes has not fixed the reported problem.
  • The adjacency has stayed at Full across a deliberate observation window rather than being declared healthy on the first sample. A flapping adjacency is Full most of the time.
  • Any OSPF debug enabled during the investigation has been turned off, and the router log volume has returned to its baseline.
  • The diagnosis in the ticket names the layer that failed — Hello matching, database exchange, delivery, or identity — and carries the command output from both ends that established it.

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • Every diagnostic step in this runbook is read-only. Nothing needs rolling back until a fix is applied, and the fixes are individually small and individually reversible.
  • A fix applied with commit-confirm 10 and found to be wrong needs no action: do nothing and the router reverts. Use that rather than typing a correction into a router whose adjacency state you no longer trust.
  • A committed and confirmed fix is reverted by deleting the specific nodes that were added, with compare read before the commit. Prefer that to a whole-configuration revert, which also discards anything else committed in the same window.
  • An interface MTU change bounces the interface. Reverting it bounces it again, so the revert is a second service event and needs to be planned like the first rather than typed reflexively.
  • A mtu-ignore added to bring an adjacency up is removed as part of fixing the underlying MTU disagreement, not left behind. Left in place it hides the mismatch from the next operator while the data plane keeps paying for it.
  • A router-id change is not reverted by the configuration alone; the running process keeps the value it started with. Reverting means another process restart, and therefore another full reconvergence, so the revert costs what the change cost.
  • Any OSPF debug is disabled explicitly as part of the rollback. A debug left running is a slow-burning fault that surfaces later as unexplained CPU and log growth.

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to the owner of the far-end router as soon as the evidence shows the mismatch is on their side. A capture proving their Hellos carry a different area-id, a different hello interval or a different MTU is complete evidence, and sending the evidence rather than the conclusion is what makes the handover fast.
  • · Escalate to the transport or facilities owner when a capture shows Hellos leaving one end and never arriving at the other. A unidirectional path is a physical or provider fault; it is not fixable from either router and no OSPF configuration change should be attempted to work around it.
  • · Escalate to network engineering before changing anything about an area — area-id, area type, or summarisation. Those changes are area-wide events rather than link-local ones, they will affect adjacencies that are currently healthy, and they belong in a planned window with the whole area covered in one change.
  • · Escalate to security when authentication is failing and nobody can account for the change. An adjacency that stopped forming because a key changed is either a change nobody recorded or a peer that is not the peer you think it is, and the second possibility has to be excluded rather than assumed away.
  • · Escalate to the estate owner on a duplicate router-id. Fixing it means changing an identity that other routers hold in their databases and restarting an OSPF process, so it needs the register updated and a window agreed, not a quick edit on whichever router is in front of you.
  • · Hold rather than improvise when the only available fix suppresses evidence instead of resolving a fault — mtu-ignore to make ExStart go away, a debug left running because it seems to help, an adjacency cleared repeatedly to "reset" a flap. Record the hold with an owner and an end time, and state plainly what stays broken while it holds.

An OSPF adjacency does not simply fail. It stalls at a specific state, and the state names the layer that failed. Reading it — from both ends — is most of this runbook, because the state determines which of five completely different investigations you are about to run.

StateWhat has already succeededWhat to look at
No neighbour listedNothingHello matching: area, timers, network type, authentication, passive, subnet
InitOur Hellos reach themOne-way path, or they discard ours: capture from both ends
2-Way, both routers DROtherEverything that shouldNothing. This is the DR design working
ExStartHellos matchedInterface MTU first, then duplicate router-id
Exchange or LoadingThe database exchange beganDelivery: loss, errors, a lossy or one-way link
Full, then down, repeatedlyAll of it, intermittentlyDuplicate router-id, dead-interval expiry, a flapping link

The single most common mistake in OSPF diagnosis is reading the state from one router. Nearly every field this runbook checks is a matching requirement between two routers, and a matching requirement is not observable from one side.

When this runbook applies

  • An adjacency that has never reached Full.
  • An adjacency that reached Full and dropped.
  • An adjacency that will not stay at Full.
  • A new router whose neighbours never appear.
  • An adjacency that fails after an MTU, area, authentication or addressing change.

When it does not

  • Two DROther routers sitting at 2-Way on a broadcast segment. That is correct. On a multi-access segment the DR and the BDR describe the topology; every other router holds a Full adjacency with those two and stays at 2-Way with its DROther peers. Treating it as a fault leads to changing priorities on a working segment.
  • A Full adjacency that is not delivering the route you want. The adjacency is doing its job. Whether an LSA was originated, whether it survived an area boundary, and whether the route was selected and installed are separate questions with separate evidence — that is the missing-route runbook.
  • A router that has no OSPF configuration on the interface at all. That is a build request, not an incident, and adding OSPF to an interface during a page has a blast radius nobody has assessed.

Blast radius

Every diagnostic step below is read-only, which makes the ordering free: there is no reason to skip a cheap check to save time. The risk in an OSPF incident is concentrated in three actions, and all three are readily available.

The second is mtu-ignore, discussed below. The third is any change to an area — its identifier, its type, its summarisation — which is an area-wide event that will disturb adjacencies that are currently healthy.

Step 0: name the adjacency, and get the far end

Which router, which interface, which neighbour router-id, which area. Then arrange access to the other end, or get its owner on the call.

That second half is not process for its own sake. Area identifier, area type, hello interval, dead interval, network type, authentication and MTU are all requirements that two routers must agree on. Every one of them is invisible from one side. Working a matching problem single-ended is how an operator spends an afternoon confirming that their own configuration is correct — which it may well be.

Step 1: read the state from both ends

Read-only / Saferun this on each router, and compare
show ip ospf neighbor
show ip ospf neighbor detail
show ip ospf interface
show ip ospf

The pair of answers is more informative than either one. A router that sees the peer in Init while the peer sees no neighbour at all is a one-way path — our Hellos are arriving there, theirs are not arriving here — and no single-ended view shows that.

show ip ospf interface is the one to read carefully. It carries the area, the network type, the timers, the MTU, the cost and the interface’s own state and role, which is the whole Hello matching list in one place. show ip ospf carries the router-id, which is the identity the rest of the area holds.

Step 2: no neighbour at all — the Hello matching list

Every field below must agree between the two ends. Every disagreement causes the Hello to be discarded, and every discard is silent: no error, no log line an operator would notice, nothing but an empty neighbour list.

FieldRead it withFailure signature
Area identifiershow ip ospf interfaceEmpty neighbour list on both ends
Area type (stub, NSSA, standard)show configuration commands | match ospfEmpty neighbour list; the type is carried in the Hello options
Hello and dead intervalsshow ip ospf interfaceEmpty neighbour list, both sides look correct in isolation
Network typeshow ip ospf interfaceEmpty neighbour list; broadcast and point-to-point do not interoperate
Authentication type, key identifier, keyshow configuration commands | match authenticationEmpty neighbour list; the packet is dropped after being received
Passiveshow ip ospf interfaceNo Hellos sent at all on that interface
Subnet and maskshow interfacesTwo routers on the same wire in different subnets never see each other

Two of these deserve naming because they are the ones that produce the most confident wrong answers.

Passive on both ends. A passive interface participates in OSPF — its subnet is advertised — but sends no Hellos. Marked passive at both ends, both routers are listening and neither is speaking. The evidence is an empty neighbour list on two routers whose configuration each look entirely reasonable.

Authentication. The type, the key identifier and the key itself must all agree. A mismatch is dropped after the packet has been received, so the interface counters show traffic and the neighbour list stays empty. VyOS configures this per interface, for example set protocols ospf interface eth0 authentication md5 key-id 1 md5-key SHAREDSECRET, with a per-area form available for an area-wide policy. Where a key chain is in use, the lifetime windows and the routers’ clocks matter as much as the key: an expired acceptance window and a wrong key look identical from the neighbour list.

Before concluding the far end is misconfigured, prove the Hellos are arriving.

Read-only / Safeare the packets even there — run on both ends
monitor traffic interface eth0 filter 'proto ospf'

That separates “the peer is not sending” from “the peer is sending and we are discarding”, which have entirely different owners.

Step 3: ExStart — check MTU before anything else

The database description packet carries the sending interface’s MTU. A router that receives a packet claiming an MTU larger than its own interface MTU discards it, so the master and slave negotiation never completes and the adjacency sits in ExStart.

Read-only / Safethe same command on both routers; compare the MTU line
show ip ospf interface eth0
show ip ospf neighbor detail

This is the dominant ExStart cause by a wide margin, which makes it the cheapest first check rather than merely a likely one. There are two fixes and they are not equivalent.

Aligning the MTU on both sides is the real fix, and it is the right answer whenever both ends are under one owner. It bounces the interface, so it is a short service event rather than a free one.

If the MTU agrees and the state is still ExStart, read the router-ids. Two routers holding the same one cannot complete the negotiation, because each believes the other is itself.

Step 4: Exchange or Loading — stop reading configuration

Both of these states mean the matching requirements are already satisfied: the Hellos agreed, the negotiation completed, and the two routers began exchanging database information. What is failing is delivery.

Read-only / Safethe link, not the protocol
show interfaces ethernet eth0
show ip ospf neighbor detail
show ip ospf database

Read the interface error and drop counters, and read them as a rate rather than a total — a counter that has been climbing since the last reboot says much less than one climbing now. A neighbour in Loading with a growing queue of requested LSAs is the signature of update packets being lost in transit, which is congestion or a lossy path, not a configuration fault.

Continuing to edit OSPF configuration at this point is the classic wasted hour. The protocol is working; the link is not.

Step 5: Full, and it will not stay there

A flapping adjacency has three common causes, and they are separable with cheap evidence.

A duplicate router-id. Two routers holding the same identity originate router LSAs that the database cannot tell apart. The database oscillates, the adjacency flaps, and FRR logs the duplicate. This is the highest-impact of the three, because it corrupts the area’s view of the topology rather than merely interrupting one link.

Read-only / Safeidentity, then the database, then the log
show ip ospf
show ip ospf database self-originate
Read-only / Safethe daemon's own complaint
sudo journalctl -u frr | grep -i duplicate

Dead-interval expiry. Hellos are being sent and are not arriving within the dead interval — packet loss on the link, or a router too busy to service the protocol. The interface counters and the router’s CPU state separate these from each other.

A flapping link. The physical path is going up and down and OSPF is faithfully reporting it. The interface counters and the system log carry the link events; the adjacency is a symptom, not the fault.

Step 6: debug, narrowly and with an end

The OSPF debug commands print every packet with its fields parsed, which is exactly what an authentication or database-description fault needs and is also a great deal of log volume. A single adjacency forming can produce hundreds of lines, and the daemon is writing them while it is also running the protocol.

Enable it on the affected router only, after the cheaper evidence is exhausted, with the disable command already decided. A debug left running is a fault that surfaces weeks later as unexplained CPU and a full log partition, and by then nobody connects it to this incident.

What each step costs if you get it wrong

ActionCost when it is the wrong move
Diagnosing a matching requirement from one endAn afternoon confirming your own configuration is correct
Treating 2-Way between DROthers as a faultPriority changes on a working segment, and a new fault
Reaching for clear ip ospf processEvery adjacency on the router drops, and the evidence is gone
Setting mtu-ignore instead of comparing MTUA quiet MTU mismatch the data plane still pays for
Changing an area to fix one linkAn area-wide event that disturbs adjacencies that were healthy
Committing a router-id without restarting the processConfiguration and running state disagree, and the fault persists
Leaving OSPF debug enabledCPU and log growth that surfaces long after anyone remembers why
Declaring victory at FullA synchronised adjacency delivering none of the routes the ticket was about

Escalation, and the option to hold

Escalate the moment the evidence crosses a boundary you do not own, and send the evidence rather than the conclusion. A capture showing the peer’s Hellos carrying a different area or a different interval is complete proof and makes the handover a two-minute conversation. A capture showing your Hellos leaving and nothing arriving is complete proof that the fault is in the path, and no configuration change on either router will address it.

Hold rather than improvise when the only available action suppresses evidence instead of resolving a fault. Clearing the process to “reset” a flap, enabling mtu-ignore to make ExStart go away, restarting a router because the ticket is old: each removes today’s symptom and moves the fault to a moment nobody is watching. Record the hold with an owner and an end time, and state what stays broken while it holds — including which prefixes are unreachable, because that is the number the person deciding actually needs.

References

  1. VyOS documentation — OSPF
  2. FRRouting documentation — OSPFv2
  3. RFC 2328 — OSPF Version 2
  4. RFC 5709 — OSPFv2 HMAC-SHA cryptographic authentication
  5. VyOS documentation — Ethernet interfaces and MTU