Runbook: Troubleshoot OSPF Adjacency
1 · Prerequisites
Confirm every item is in place before any state change.
- OSPF neighbours and adjacency — the hello protocol and the eight-state machine
- DR/BDR election — designated router, broadcast vs point-to-point, and why the DR exists
- OSPF basics — enabling OSPF, router-id, default route, and area assignment
- OSPF interface configuration — network type, timers, MTU, passive
- OSPF authentication — plaintext, MD5, SHA-256, key chain, virtual links
- Neighbour stuck in EXSTART / EXCHANGE / Loading — MTU, DD exchange, LSR retransmission
- MTU mismatch — DF bit, fragmentation, `ip ospf mtu-ignore`, vendor behaviour
- Area mismatch — hello area field, type 1 mismatch, summary vs external in stub
- OSPF authentication — simple-text, MD5, SHA, key-id mismatch, key chain timing
- Duplicate router-id — Type 1 LSA conflict, why uniqueness matters, recovery
- Evidence first — collect before changing, write down symptoms, no action without data
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · The adjacency is named precisely before any command is run: which router, which interface, which neighbour router-id, and which area it is supposed to be in. "OSPF is down" is a ticket; "R1 eth0 to R2 in area 0 is stuck in ExStart" is a symptom with a first command attached to it.
- · Access to BOTH ends is arranged, or an owner for the far end is on the call. Nearly every field this runbook checks is a matching requirement between two routers, and a matching requirement cannot be diagnosed from one side. Starting without the far end is how an afternoon disappears.
- · Whether this adjacency has ever been Full is established, and if so when it stopped. An adjacency that never formed is a configuration mismatch; one that formed and then failed is a change, a link event, or a timer expiry — and those have different first commands.
- · The scope is established: one adjacency, every adjacency on one interface, or every adjacency on this router. Every adjacency at once points at the router — its router-id, its OSPF process, its control-plane policy — not at any one peer.
- · The change record for the last 72 hours is checked on both routers, covering interface MTU, area configuration, authentication, timers, network type, router-id and any firewall or control-plane policy change.
- · The pre-diagnosis state is captured off-box from both ends and attached to the ticket:
show ip ospf neighbor,show ip ospf neighbor detail,show ip ospf interface,show ip ospf, andshow configuration commands | match ospf. - · Out-of-band console access is confirmed if there is any chance the fix will touch the path the operator is connected over. An adjacency fix that turns into an interface MTU change is exactly that case, and an MTU change bounces the interface.
- · It is agreed up front that
clear ip ospf processis not a diagnostic step. It tears down every adjacency on the router, not the one being investigated, and refloods the whole database. It is a deliberate service event with an owner, used when a router-id change requires it and not before. - · It is agreed up front that
mtu-ignoreis a decision, not a fix. It suppresses the symptom of a real MTU disagreement that the data plane still has to live with, so it needs a reason recorded in the ticket and a follow-up owner.
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Read the neighbour state from both ends, not one.
show ip ospf neighboron each router. The two ends frequently report different states, and the pair is more informative than either alone — a router that sees the peer in Init while the peer sees nothing at all is a one-way link, which no single-ended view reveals. - 2Classify the state, because the state names the layer. No neighbour at all means the Hellos are not being exchanged or are being discarded. Init means Hellos arrive in one direction only. ExStart means the Hellos matched and the database exchange did not start. Exchange and Loading mean the exchange started and is not completing. Full but flapping is a different fault entirely.
- 3Before treating 2-Way as a fault, check the network type and the DR election. On a broadcast segment, two routers that are both DROther hold a Full adjacency with the DR and the BDR and stay in 2-Way with each other by design.
show ip ospf interfacereports the state and the role;Neighbor CountagainstAdjacent neighbor countin the detail output is the gap the design intends. - 4For no neighbour at all, work the Hello matching list rather than guessing: area-id, area-type, hello and dead intervals, network type, authentication, subnet mask, and whether either side is passive. Every one of these must agree, every disagreement discards the Hello silently, and none of them produces an error the operator will find by accident.
- 5Confirm the Hellos are physically arriving before blaming configuration. A capture filtered to the OSPF protocol on the interface, run on both ends, separates "the peer is not sending" from "the peer is sending and we are discarding". Those two have completely different owners.
- 6For a neighbour stuck in ExStart, compare the interface MTU on both sides first. The database description packet carries the interface MTU, and a router discards a packet whose claimed MTU exceeds its own interface MTU, which stalls the master and slave negotiation. This is the dominant ExStart cause by a wide margin.
- 7If the MTU agrees and the adjacency is still in ExStart, check the router-ids on both sides. Two routers holding the same router-id cannot complete the negotiation, and the same duplication also produces adjacency flapping elsewhere in the area.
- 8For a neighbour stuck in Exchange or Loading, stop looking at configuration and look at the link. The database exchange has started, so the matching requirements are satisfied; what is failing is delivery. Read the interface error and drop counters, and check whether the path is lossy or unidirectional.
- 9For an adjacency that reaches Full and does not stay there, treat it as one of three faults and separate them with evidence: a duplicate router-id, dead-interval expiry caused by loss or CPU starvation, or a physical link that is flapping. The FRR log carries duplicate-LSA warnings for the first; the interface counters carry the third.
- 10Enable OSPF debug only after the cheaper evidence is exhausted, only on the affected router, and with an explicit end. The debug prints every packet with its fields parsed, which is exactly what is needed for an authentication or database-description fault and is also enough log volume to disturb a busy router.
- 11Apply the fix inside
configure, withcompareread before the commit andcommit-confirm 10for anything that can remove the operator path or bounce an interface. Confirm only after the adjacency has been observed at Full from both ends. - 12If the fix is a router-id change, understand that the configuration change alone does not take effect: the router-id is established when the OSPF process starts, so the change needs a process restart, and the restart is a service event affecting every adjacency on the router. Plan it as such rather than discovering it.
- 13Verify from both ends, then verify that routes actually arrived. An adjacency at Full is a means, not an end; the ticket was about reachability, and a Full adjacency with an empty database is the next runbook, not a closed incident.
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓
show ip ospf neighboron both routers reports the peer in state Full, and the two ends agree on which router holds DR and which holds BDR where the segment elects them. - ✓On a broadcast segment,
Neighbor CountandAdjacent neighbor countinshow ip ospf neighbor detailmatch the DR design — a gap between them is expected on a segment with more than two routers and is not a fault. - ✓
show ip ospf interfaceon both ends shows the same area, the same network type, the same hello and dead intervals, and the same MTU, or an explicitly recorded reason why the MTU differs. - ✓
show ip ospfon both routers reports different router-ids, and each matches the value the estate register allocates to that router. - ✓The link-state database is synchronised:
show ip ospf databaseon both ends holds the same set of LSAs for the area, and neither end is missing LSAs the other has. - ✓The routes the adjacency was expected to deliver are present in
show ip route ospfand installed in the kernel table, because a Full adjacency that delivers no routes has not fixed the reported problem. - ✓The adjacency has stayed at Full across a deliberate observation window rather than being declared healthy on the first sample. A flapping adjacency is Full most of the time.
- ✓Any OSPF debug enabled during the investigation has been turned off, and the router log volume has returned to its baseline.
- ✓The diagnosis in the ticket names the layer that failed — Hello matching, database exchange, delivery, or identity — and carries the command output from both ends that established it.
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶Every diagnostic step in this runbook is read-only. Nothing needs rolling back until a fix is applied, and the fixes are individually small and individually reversible.
- ↶A fix applied with
commit-confirm 10and found to be wrong needs no action: do nothing and the router reverts. Use that rather than typing a correction into a router whose adjacency state you no longer trust. - ↶A committed and confirmed fix is reverted by deleting the specific nodes that were added, with
compareread before thecommit. Prefer that to a whole-configuration revert, which also discards anything else committed in the same window. - ↶An interface MTU change bounces the interface. Reverting it bounces it again, so the revert is a second service event and needs to be planned like the first rather than typed reflexively.
- ↶A
mtu-ignoreadded to bring an adjacency up is removed as part of fixing the underlying MTU disagreement, not left behind. Left in place it hides the mismatch from the next operator while the data plane keeps paying for it. - ↶A router-id change is not reverted by the configuration alone; the running process keeps the value it started with. Reverting means another process restart, and therefore another full reconvergence, so the revert costs what the change cost.
- ↶Any OSPF debug is disabled explicitly as part of the rollback. A debug left running is a slow-burning fault that surfaces later as unexplained CPU and log growth.
6 · Escalation
When the runbook isn't enough, contact:
- · Escalate to the owner of the far-end router as soon as the evidence shows the mismatch is on their side. A capture proving their Hellos carry a different area-id, a different hello interval or a different MTU is complete evidence, and sending the evidence rather than the conclusion is what makes the handover fast.
- · Escalate to the transport or facilities owner when a capture shows Hellos leaving one end and never arriving at the other. A unidirectional path is a physical or provider fault; it is not fixable from either router and no OSPF configuration change should be attempted to work around it.
- · Escalate to network engineering before changing anything about an area — area-id, area type, or summarisation. Those changes are area-wide events rather than link-local ones, they will affect adjacencies that are currently healthy, and they belong in a planned window with the whole area covered in one change.
- · Escalate to security when authentication is failing and nobody can account for the change. An adjacency that stopped forming because a key changed is either a change nobody recorded or a peer that is not the peer you think it is, and the second possibility has to be excluded rather than assumed away.
- · Escalate to the estate owner on a duplicate router-id. Fixing it means changing an identity that other routers hold in their databases and restarting an OSPF process, so it needs the register updated and a window agreed, not a quick edit on whichever router is in front of you.
- · Hold rather than improvise when the only available fix suppresses evidence instead of resolving a fault —
mtu-ignoreto make ExStart go away, a debug left running because it seems to help, an adjacency cleared repeatedly to "reset" a flap. Record the hold with an owner and an end time, and state plainly what stays broken while it holds.
An OSPF adjacency does not simply fail. It stalls at a specific state, and the state names the layer that failed. Reading it — from both ends — is most of this runbook, because the state determines which of five completely different investigations you are about to run.
| State | What has already succeeded | What to look at |
|---|---|---|
| No neighbour listed | Nothing | Hello matching: area, timers, network type, authentication, passive, subnet |
| Init | Our Hellos reach them | One-way path, or they discard ours: capture from both ends |
| 2-Way, both routers DROther | Everything that should | Nothing. This is the DR design working |
| ExStart | Hellos matched | Interface MTU first, then duplicate router-id |
| Exchange or Loading | The database exchange began | Delivery: loss, errors, a lossy or one-way link |
| Full, then down, repeatedly | All of it, intermittently | Duplicate router-id, dead-interval expiry, a flapping link |
The single most common mistake in OSPF diagnosis is reading the state from one router. Nearly every field this runbook checks is a matching requirement between two routers, and a matching requirement is not observable from one side.
When this runbook applies
- An adjacency that has never reached Full.
- An adjacency that reached Full and dropped.
- An adjacency that will not stay at Full.
- A new router whose neighbours never appear.
- An adjacency that fails after an MTU, area, authentication or addressing change.
When it does not
- Two DROther routers sitting at 2-Way on a broadcast segment. That is correct. On a multi-access segment the DR and the BDR describe the topology; every other router holds a Full adjacency with those two and stays at 2-Way with its DROther peers. Treating it as a fault leads to changing priorities on a working segment.
- A Full adjacency that is not delivering the route you want. The adjacency is doing its job. Whether an LSA was originated, whether it survived an area boundary, and whether the route was selected and installed are separate questions with separate evidence — that is the missing-route runbook.
- A router that has no OSPF configuration on the interface at all. That is a build request, not an incident, and adding OSPF to an interface during a page has a blast radius nobody has assessed.
Blast radius
Every diagnostic step below is read-only, which makes the ordering free: there is no reason to skip a cheap check to save time. The risk in an OSPF incident is concentrated in three actions, and all three are readily available.
The second is mtu-ignore, discussed below. The third is any change to an
area — its identifier, its type, its summarisation — which is an area-wide
event that will disturb adjacencies that are currently healthy.
Step 0: name the adjacency, and get the far end
Which router, which interface, which neighbour router-id, which area. Then arrange access to the other end, or get its owner on the call.
That second half is not process for its own sake. Area identifier, area type, hello interval, dead interval, network type, authentication and MTU are all requirements that two routers must agree on. Every one of them is invisible from one side. Working a matching problem single-ended is how an operator spends an afternoon confirming that their own configuration is correct — which it may well be.
Step 1: read the state from both ends
show ip ospf neighbor
show ip ospf neighbor detail
show ip ospf interface
show ip ospfThe pair of answers is more informative than either one. A router that sees the peer in Init while the peer sees no neighbour at all is a one-way path — our Hellos are arriving there, theirs are not arriving here — and no single-ended view shows that.
show ip ospf interface is the one to read carefully. It carries the area,
the network type, the timers, the MTU, the cost and the interface’s own state
and role, which is the whole Hello matching list in one place. show ip ospf
carries the router-id, which is the identity the rest of the area holds.
Step 2: no neighbour at all — the Hello matching list
Every field below must agree between the two ends. Every disagreement causes the Hello to be discarded, and every discard is silent: no error, no log line an operator would notice, nothing but an empty neighbour list.
| Field | Read it with | Failure signature |
|---|---|---|
| Area identifier | show ip ospf interface | Empty neighbour list on both ends |
| Area type (stub, NSSA, standard) | show configuration commands | match ospf | Empty neighbour list; the type is carried in the Hello options |
| Hello and dead intervals | show ip ospf interface | Empty neighbour list, both sides look correct in isolation |
| Network type | show ip ospf interface | Empty neighbour list; broadcast and point-to-point do not interoperate |
| Authentication type, key identifier, key | show configuration commands | match authentication | Empty neighbour list; the packet is dropped after being received |
| Passive | show ip ospf interface | No Hellos sent at all on that interface |
| Subnet and mask | show interfaces | Two routers on the same wire in different subnets never see each other |
Two of these deserve naming because they are the ones that produce the most confident wrong answers.
Passive on both ends. A passive interface participates in OSPF — its subnet is advertised — but sends no Hellos. Marked passive at both ends, both routers are listening and neither is speaking. The evidence is an empty neighbour list on two routers whose configuration each look entirely reasonable.
Authentication. The type, the key identifier and the key itself must all
agree. A mismatch is dropped after the packet has been received, so the
interface counters show traffic and the neighbour list stays empty. VyOS
configures this per interface, for example set protocols ospf interface eth0 authentication md5 key-id 1 md5-key SHAREDSECRET, with a per-area form
available for an area-wide policy. Where a key chain is in use, the lifetime
windows and the routers’ clocks matter as much as the key: an expired
acceptance window and a wrong key look identical from the neighbour list.
Before concluding the far end is misconfigured, prove the Hellos are arriving.
monitor traffic interface eth0 filter 'proto ospf'That separates “the peer is not sending” from “the peer is sending and we are discarding”, which have entirely different owners.
Step 3: ExStart — check MTU before anything else
The database description packet carries the sending interface’s MTU. A router that receives a packet claiming an MTU larger than its own interface MTU discards it, so the master and slave negotiation never completes and the adjacency sits in ExStart.
show ip ospf interface eth0
show ip ospf neighbor detailThis is the dominant ExStart cause by a wide margin, which makes it the cheapest first check rather than merely a likely one. There are two fixes and they are not equivalent.
Aligning the MTU on both sides is the real fix, and it is the right answer whenever both ends are under one owner. It bounces the interface, so it is a short service event rather than a free one.
If the MTU agrees and the state is still ExStart, read the router-ids. Two routers holding the same one cannot complete the negotiation, because each believes the other is itself.
Step 4: Exchange or Loading — stop reading configuration
Both of these states mean the matching requirements are already satisfied: the Hellos agreed, the negotiation completed, and the two routers began exchanging database information. What is failing is delivery.
show interfaces ethernet eth0
show ip ospf neighbor detail
show ip ospf databaseRead the interface error and drop counters, and read them as a rate rather than a total — a counter that has been climbing since the last reboot says much less than one climbing now. A neighbour in Loading with a growing queue of requested LSAs is the signature of update packets being lost in transit, which is congestion or a lossy path, not a configuration fault.
Continuing to edit OSPF configuration at this point is the classic wasted hour. The protocol is working; the link is not.
Step 5: Full, and it will not stay there
A flapping adjacency has three common causes, and they are separable with cheap evidence.
A duplicate router-id. Two routers holding the same identity originate router LSAs that the database cannot tell apart. The database oscillates, the adjacency flaps, and FRR logs the duplicate. This is the highest-impact of the three, because it corrupts the area’s view of the topology rather than merely interrupting one link.
show ip ospf
show ip ospf database self-originatesudo journalctl -u frr | grep -i duplicateDead-interval expiry. Hellos are being sent and are not arriving within the dead interval — packet loss on the link, or a router too busy to service the protocol. The interface counters and the router’s CPU state separate these from each other.
A flapping link. The physical path is going up and down and OSPF is faithfully reporting it. The interface counters and the system log carry the link events; the adjacency is a symptom, not the fault.
Step 6: debug, narrowly and with an end
The OSPF debug commands print every packet with its fields parsed, which is exactly what an authentication or database-description fault needs and is also a great deal of log volume. A single adjacency forming can produce hundreds of lines, and the daemon is writing them while it is also running the protocol.
Enable it on the affected router only, after the cheaper evidence is exhausted, with the disable command already decided. A debug left running is a fault that surfaces weeks later as unexplained CPU and a full log partition, and by then nobody connects it to this incident.
What each step costs if you get it wrong
| Action | Cost when it is the wrong move |
|---|---|
| Diagnosing a matching requirement from one end | An afternoon confirming your own configuration is correct |
| Treating 2-Way between DROthers as a fault | Priority changes on a working segment, and a new fault |
Reaching for clear ip ospf process | Every adjacency on the router drops, and the evidence is gone |
Setting mtu-ignore instead of comparing MTU | A quiet MTU mismatch the data plane still pays for |
| Changing an area to fix one link | An area-wide event that disturbs adjacencies that were healthy |
| Committing a router-id without restarting the process | Configuration and running state disagree, and the fault persists |
| Leaving OSPF debug enabled | CPU and log growth that surfaces long after anyone remembers why |
| Declaring victory at Full | A synchronised adjacency delivering none of the routes the ticket was about |
Escalation, and the option to hold
Escalate the moment the evidence crosses a boundary you do not own, and send the evidence rather than the conclusion. A capture showing the peer’s Hellos carrying a different area or a different interval is complete proof and makes the handover a two-minute conversation. A capture showing your Hellos leaving and nothing arriving is complete proof that the fault is in the path, and no configuration change on either router will address it.
Hold rather than improvise when the only available action suppresses evidence
instead of resolving a fault. Clearing the process to “reset” a flap,
enabling mtu-ignore to make ExStart go away, restarting a router because the
ticket is old: each removes today’s symptom and moves the fault to a moment
nobody is watching. Record the hold with an owner and an end time, and state
what stays broken while it holds — including which prefixes are unreachable,
because that is the number the person deciding actually needs.