MTU mismatch — DF bit, fragmentation, `ip ospf mtu-ignore`, vendor behaviour

MTU mismatch is the single most common cause of OSPF adjacency failures stuck in EXSTART. The DBD packet carries the interface MTU; the receiving router compares the claimed MTU to its own interface MTU; if the claimed MTU exceeds the local interface MTU, the DBD is dropped.

This lesson covers the protocol-level behaviour, the Don’t Fragment bit and ICMP Fragmentation Needed interaction, the operational escape hatch (ip ospf mtu-ignore), vendor-specific behaviour, and the production playbook for tunnel and VLAN MTU reduction.

The DBD packet and the MTU field

The OSPF Database Description packet has a header that includes two important fields: the interface MTU (16 bits) and the Options field. The interface MTU field carries the MTU of the outgoing interface.

OSPF DBD packet header (RFC 2328 Section 10.3):
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Interface MTU         |    Options    |0|0|0|0|0|I|M|MS
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

When R1 sends a DBD, it fills the Interface MTU field with the MTU of the outgoing interface. R2 receives the DBD and compares the claimed MTU to its own interface MTU.

The comparison rule (RFC 2328):

If the Interface MTU field in the DD packet indicates an IP datagram size that exceeds the IP MTU of the receiving interface, the DD packet is discarded.

This is the rule that causes EXSTART failures. If R1 claims MTU 1500 but R2 has interface MTU 1400, R2 discards the DBD packet. The master/slave negotiation cannot complete; the adjacency is stuck in EXSTART.

sequenceDiagram
  participant R1 as R1 (eth0 MTU 1500)
  participant R2 as R2 (eth0 MTU 1400)
  R1->>R2: DBD Interface MTU = 1500<br/>Size: ~1500 bytes
  Note over R2: 1500 > 1400<br/>DD packet discarded
  R2->>R1: DBD Interface MTU = 1400<br/>Size: ~1400 bytes
  R1->>R2: DBD (slave ack)
  Note over R1,R2: R2's DBD never received<br/>because R1's DBD was dropped
  Note over R1,R2: Master/slave negotiation fails<br/>Stuck in EXSTART

The Don’t Fragment bit and Path MTU Discovery

OSPF packets are IP packets with the Don’t Fragment (DF) bit set. The DF bit tells intermediate routers that the packet must not be fragmented; if the packet is too large for an outgoing interface, the router sends an ICMP Fragmentation Needed message back to the sender.

The Path MTU Discovery (PMTUD) mechanism (RFC 1191) uses the DF bit and ICMP messages to discover the smallest MTU along a path. OSPF packets use PMTUD; the sender starts with the interface MTU and reduces it as ICMP messages indicate.

For OSPF, the local router’s DF bit is set on the DBD packet; if the packet is too large for the local outgoing interface, the kernel drops it (the kernel does not fragment). The ICMP Fragmentation Needed message is sent back to the sender; the sender’s PMTUD reduces the packet size.

sequenceDiagram
  participant R1 as R1 (eth0 MTU 1500)
  participant R2 as R2 (eth0 MTU 1400)
  R1->>R2: DBD Interface MTU = 1500<br/>DF=1 Size: 1500
  Note over R2: 1500 > 1400<br/>ICMP Fragmentation Needed<br/>DF=1
  R2-->>R1: ICMP Fragmentation Needed<br/>MTU = 1400
  Note over R1: PMTUD reduces to 1400<br/>Resends DBD at 1400

The PMTUD behaviour is for IP packets in general. For OSPF specifically, the DBD packet is dropped by the receiving router without sending an ICMP message back. The protocol rule is: if the MTU in the DBD exceeds the receiving interface’s MTU, the DBD is silently discarded.

The diagnostic for PMTUD is:

ping -M do -s 1472 10.0.0.2
# -M do: set DF bit
# -s 1472: payload size (1500 - 20 IP header - 8 ICMP header)
# If the path MTU is smaller, the ping fails with "Message too long"

tracepath 10.0.0.2
# Discovers the path MTU

The ip ospf mtu-ignore escape hatch

The ip ospf mtu-ignore configuration tells the OSPF process to skip the MTU check on the DBD packet. With mtu-ignore set, the receiving router accepts DBD packets with a claimed MTU exceeding the local interface MTU; the DBD exchange proceeds.

The configuration:

set protocols ospf interface eth0 mtu-ignore
commit
save

The operational use case: the local interface has a lower MTU than the neighbour (e.g. a tunnel interface with reduced MTU) and the operator wants the OSPF adjacency to form anyway. The mtu-ignore allows the exchange.

flowchart LR
  subgraph "Without mtu-ignore"
    A1["R1 MTU 1500"]
    A2["R2 MTU 1400"]
    A1 -- "DBD 1500 dropped" --> A2
    A2 -- "DBD 1400 OK" --> A1
    A1 -. "EXSTART stuck" .-> A1
  end
  subgraph "With mtu-ignore on R1"
    B1["R1 MTU 1500 mtu-ignore"]
    B2["R2 MTU 1400"]
    B1 -- "DBD 1500 OK (ignored)" --> B2
    B2 -- "DBD 1400 OK" --> B1
    B1 -- "DBD exchange proceeds" --> B2
    B2 -- "Full adjacency" --> B1
  end

The mtu-ignore configuration is per-interface and per-direction. Setting it on R1 does not affect R2; the operator must set it on the side that has the larger MTU and is sending oversized DBD packets.

Vendor-specific MTU handling

Different vendors handle OSPF MTU mismatch differently:

• Cisco IOS / IOS-XE. Default behaviour: silent drop, same as FRR. ip ospf mtu-ignore interface command enables the escape hatch. Also supports ip ospf database-filter all for additional control.
• Juniper Junos. Default behaviour: silent drop. set protocols ospf interface <if-name> mtu-ignore enables the escape hatch. Also supports set protocols ospf interface <if-name> retransmit-interval for timing control.
• FRR (VyOS). Default behaviour: silent drop. set protocols ospf interface eth0 mtu-ignore enables the escape hatch. Same as Cisco and Juniper.
• Huawei VRP. Default behaviour: silent drop. ospf mtu-ignore interface command enables the escape hatch.

The common pattern: all major vendors default to silent drop on MTU mismatch and offer an mtu-ignore escape hatch. The operational answer is the same across vendors.

Tunnel MTU and the production playbook

A common production scenario: an OSPF adjacency is configured over a tunnel (IPsec, GRE, WireGuard). The tunnel interface has a reduced MTU (e.g. 1400 for IPsec with encryption overhead). The physical interface has MTU 1500. The OSPF adjacency fails because the DBD packet (1500 bytes) is larger than the tunnel MTU (1400).

The operational playbook:

1. Reduce the OSPF DBD packet size. The DBD packet carries the LSA headers; the size is the size of the largest LSA header. Reducing the LSDB size (e.g. by summarisation) reduces the DBD size, but the headers alone can be larger than a tunnel MTU.
2. Reduce the interface MTU to the tunnel MTU. Set the interface MTU on the physical interface to match the tunnel MTU. The OSPF DBD then fits.
3. Use mtu-ignore. Tell OSPF to skip the MTU check.

The recommended approach depends on the deployment:

• Tunnel only carries OSPF. Reduce the interface MTU to the tunnel MTU. The cleanest answer.
• Tunnel carries OSPF + user traffic. Use mtu-ignore. The alternative (reducing the interface MTU) may affect the user traffic.
• Mixed tunnel types. Each tunnel type has its own MTU overhead. Use mtu-ignore if multiple tunnels with different MTUs share the same OSPF adjacency.

flowchart TB
  subgraph "Tunnel MTU 1400 (IPsec overhead)"
    T1["IPsec tunnel<br/>MTU 1400"]
  end
  subgraph "Physical interface MTU 1500"
    P1["eth0<br/>MTU 1500"]
  end
  T1 -. "OSPF DBD 1500<br/>dropped" .-> T1
  P1 -- "Fix option A: align MTU" --> P1
  P1 -- "Fix option B: mtu-ignore" --> P1

VLAN MTU reduction

A similar scenario occurs with VLAN tagging. Some switches treat VLAN-tagged frames with a slightly different MTU (typically the same 1500 bytes plus the 4-byte VLAN tag, for a total of 1504 on the wire but still 1500 of IP payload). Other switches reduce the IP MTU to 1496 to accommodate the VLAN tag.

The production playbook for VLAN MTU:

1. Verify the switch behaviour. Check the switch configuration for VLAN MTU handling.
2. Match the IP MTU on the router. Set the IP MTU on the router’s VLAN sub-interface to match the switch’s effective IP MTU.
3. Use mtu-ignore if matching is not possible. For example, a multi-vendor switch fabric where some switches reduce MTU and others do not.

Diagnostic sequence

The diagnostic sequence for an MTU-mismatch failure:

1. Confirm the adjacency is stuck in EXSTART. show ip ospf neighbor.
2. Compare the interface MTU on both sides. show ip ospf interface and show interfaces ethernet eth0 on both routers.
3. Verify the physical interface MTU. The physical interface MTU is the upper bound for IP packets.
4. Verify the OSPF interface MTU. The OSPF interface MTU (carried in the DBD) is what OSPF uses for the check.
5. Check the switch fabric. For VLAN scenarios, the switch may be reducing MTU.
6. Check the tunnel overhead. For tunnel scenarios, the tunnel interface has reduced MTU.
7. Apply the fix. Either align the MTU or use mtu-ignore.
8. Validate. show ip ospf neighbor shows Full; show ip ospf database shows the LSDB synchronised.

Production failure modes

• MTU mismatch silently drops DBD. The adjacency is stuck in EXSTART. The fix is align MTU or mtu-ignore.
• Tunnel MTU overlooked. The operator configures OSPF over a tunnel without checking the tunnel MTU. The fix is to check the tunnel MTU before configuring OSPF.
• Switch VLAN MTU inconsistent. The operator’s switch fabric has some switches with VLAN MTU reduction and some without. The fix is to standardise the VLAN MTU or use mtu-ignore.
• mtu-ignore on the wrong side. The operator configures mtu-ignore on the lower-MTU side. The OSPF still fails because the higher-MTU side is sending oversized DBD. The fix is to set mtu-ignore on the higher-MTU side.
• PMTUD blackhole. ICMP Fragmentation Needed messages are filtered by an upstream firewall. PMTUD cannot reduce the packet size. The fix is to allow ICMP Fragmentation Needed messages through the firewall.

Rollback

The rollback for an MTU-related change:

# Capture the running configuration
show configuration commands | save /tmp/ospf-mtu-$(date +%s).txt

# Compare
compare

# Commit with a short confirm window
commit-confirm 5

# Rollback if needed
rollback 1
commit

For an MTU alignment:

# Revert to the previous MTU
set interfaces ethernet eth0 mtu 1500
commit
save

For an mtu-ignore removal:

delete protocols ospf interface eth0 mtu-ignore
commit
save

Production discipline

Cross-course references

The MTU fundamentals lesson vyos-vii-03-mtu-on-interface covers the interface MTU configuration this lesson builds on. The OSPF configuration lesson vyos-xix-02-ospf-interface-config covers the per-interface OSPF parameters. The OSPF troubleshooting lessons vyos-xxii-01-neighbour-stuck (neighbour-stuck states) and vyos-xxii-04-authentication (authentication, a different EXSTART cause) cover related failure modes. The MTU and fragmentation lesson vyos-li-01-mtu-basics covers the IP-layer MTU and PMTUD mechanisms.

Quiz