OSPF SPF and cost — Dijkstra's metric, reference bandwidth, and tuning paths

OSPF’s SPF tree computes the lowest-cost path from the local router to every destination. The cost is an integer assigned to each interface; SPF sums interface costs along the path and chooses the minimum. Where the operator does not set a cost explicitly, FRR derives one:

cost = reference-bandwidth / interface-bandwidth

reference-bandwidth defaults to 100 Mbit/s. The quotient is rounded to the nearest integer and then clamped, so anything at or above the reference lands on the same floor.

That last sentence is the whole problem. With the default reference, every interface of 100 Mbit/s or faster gets cost 1. A 100 M link, a 1 G link and a 10 G link are indistinguishable to SPF. In a modern network the default reference bandwidth turns OSPF into a hop-count protocol.

This lesson explains the cost formula, the tuning knobs, the places where the derived cost is not trustworthy, and the production failure modes that show up as mis-tuned cost.

The cost formula and its trap

flowchart TB
  subgraph "Reference bandwidth = 100 Mbit/s (default)"
    E100["100M ethernet<br/>100/100 = 1"]
    E1G["1 Gbit/s ethernet<br/>100/1000 = 0.1, clamped to 1"]
    E10G["10 Gbit/s ethernet<br/>100/10000 = 0.01, clamped to 1"]
  end

The operational fix is to raise the reference bandwidth so the faster interfaces get distinguishable costs:

# 10 Gbit/s reference - the value is in Mbit/s
set protocols ospf auto-cost reference-bandwidth 10000

FRR accepts 1 to 4294967 Mbit/s. What the setting does to the derived costs:

Read the columns rather than the cells. The rule is that the reference bandwidth sets the point at which differentiation stops: anything at or above it collapses to cost 1. Pick a reference at least as large as the fastest link whose cost you want to be distinguishable, and expect everything faster than that to tie.

Where the derived cost is not trustworthy

The formula divides by the bandwidth FRR holds for the interface. That number is not always the link speed you would name if asked.

• On VyOS you can state it: set protocols ospf interface eth0 bandwidth 10000 tells OSPF the interface is 10 Gbit/s for costing purposes, whatever the driver reports.
• Where you do not state it, FRR uses what the routing daemon has learned about the interface. For a physical NIC that is usually the negotiated speed. For tunnels, bonds, VLAN sub-interfaces and virtual NICs it frequently is not, because there is no meaningful link speed to report in the first place.

The practical rule is: do not predict the cost, read it.

# The effective cost on every OSPF interface, after the formula,
# any bandwidth override and any explicit cost have been applied
show ip ospf interface

Each interface block reports Network Type ..., Cost: N. That N is the number SPF will use. If it does not match what you computed on paper, the bandwidth FRR holds is not what you assumed — and on a tunnel, that is the normal case rather than the exception.

Per-interface cost vs global reference-bandwidth

There are two tuning primitives and they compose, with the per-interface one winning.

Global: auto-cost reference-bandwidth

# 10 Gbit/s reference - modern datacenter
set protocols ospf auto-cost reference-bandwidth 10000

# 100 Gbit/s reference - backbone tier
set protocols ospf auto-cost reference-bandwidth 100000

The value is in Mbit/s, so 10000 is 10 Gbit/s and 100000 is 100 Gbit/s. It affects every interface on the router that does not carry an explicit cost.

Per-interface: set protocols ospf interface <name> cost <n>

# Pin this interface at 50 regardless of what it negotiates
set protocols ospf interface eth0 cost 50

# A tunnel, where the derived cost would be meaningless
set protocols ospf interface tun0 cost 100

# A slow backup that must lose to everything while the
# primary is alive
set protocols ospf interface eth1 cost 500

An explicit cost overrides the formula entirely: neither the reference bandwidth nor the interface bandwidth is consulted for that interface any more. This is the primitive for traffic engineering, and it is the one that survives a link renegotiating at a different speed.

A convention that holds up in production: pin the cost explicitly on every backbone and critical-path interface, and let the formula handle the rest. The configuration then documents the intended topology, and no cost anywhere depends on a reference bandwidth agreeing across the domain.

# Backbone: every link is 10G and deliberately equal
set protocols ospf interface eth0 cost 1
set protocols ospf interface eth1 cost 1

# Edge, costed by tier rather than by speed
set protocols ospf interface eth2 cost 10     # 10G to a partner
set protocols ospf interface eth3 cost 50     # 1G to a partner
set protocols ospf interface eth4 cost 500    # DSL backup

How SPF chooses between equal-cost paths

When two paths to a destination have the same total cost, OSPF offers both and FRR installs them as equal-cost multi-path. The kernel then hashes flows across the next-hops, so a given flow keeps one path and the split is per-flow rather than per-packet.

How many paths are installed is capped:

# Install at most 4 equal-cost next-hops per destination
set protocols ospf maximum-paths 4

flowchart LR
  R1["R1 (root)"] -->|"cost 10"| R2["R2"]
  R1 -->|"cost 10"| R3["R3"]
  R2 -->|"cost 10"| D["10.99.0.0/24"]
  R3 -->|"cost 10"| D

R1 reaches the destination at total cost 20 through either R2 or R3, so both are installed.

vyos@r1:~$ show ip route 10.99.0.0/24

Routing entry for 10.99.0.0/24
Known via "ospf", distance 110, metric 20, best
Last update 00:04:11 ago
* 10.0.0.2, via eth0, weight 1
* 10.0.0.3, via eth1, weight 1

The thing to read is the asterisks. One starred next-hop means one path is being used; two means the traffic is being split. There is no “backup” next-hop in an OSPF RIB entry — a higher-cost path is not installed at all, so it does not appear here until the better one goes away.

To collapse the ECMP into a single preferred path, make one side cheaper:

set protocols ospf interface eth0 cost 5

R1-R2 is now 5 + 10 = 15 and R1-R3 is 10 + 10 = 20. Only the R2 path is installed, and show ip route 10.99.0.0/24 drops to a single starred next-hop with metric 15.

How summarisation affects the SPF tree

Route summarisation collapses multiple specific prefixes into one at an ABR (area X range, covered in lesson 4). Inside the summarised area nothing changes — routers there still hold the full Type 1 and Type 2 detail and compute exact per-prefix costs. What changes is everyone else’s view: other areas see one Type 3 with one metric.

flowchart TB
  subgraph "Area 1"
    P1["10.1.0.0/24 - near ABR1"]
    P2["10.1.1.0/24 - near ABR1"]
    P3["10.1.2.0/24 - near ABR2"]
    P4["10.1.3.0/24 - near ABR2"]
    ABR1["ABR1<br/>range 10.1.0.0/22"]
    ABR2["ABR2<br/>range 10.1.0.0/22"]
  end
  subgraph "Area 0"
    R0["R0"]
  end
  ABR1 -->|"one /22 Type 3"| R0
  ABR2 -->|"one /22 Type 3"| R0

R0 sees two Type 3s for 10.1.0.0/22, one per ABR, and picks the cheaper. It has no way to know that two of the four /24s inside the range are much closer to the other ABR, because that information stopped at the area boundary.

RFC 2328 section 12.4.3 sets the summary’s metric to the largest cost among its component routes, unless the operator pins it with range ... cost. So the summary is priced by its worst member, which is safe but pessimistic.

The consequences for traffic engineering:

• Inside the area, OSPF knows every per-prefix cost exactly.
• Across the boundary, OSPF knows one number for the whole range.
• Traffic for a destination whose real best exit is ABR2 will take ABR1 if ABR1’s summary is cheaper overall.

SPF throttling — what happens after a cost change

Changing a cost re-originates LSAs and triggers SPF on every router that receives them. FRR does not run SPF immediately on each arrival; it throttles, with an initial delay and a hold time that backs off while churn continues.

# delay, initial-holdtime and max-holdtime, all in milliseconds
set protocols ospf timers throttle spf delay 200
set protocols ospf timers throttle spf initial-holdtime 400
set protocols ospf timers throttle spf max-holdtime 10000

show ip ospf reports the configured values along with when SPF last ran and how long it took:

show ip ospf
# Initial SPF scheduling delay ... millisec(s)
# Minimum hold time between consecutive SPFs ... millisec(s)
# Maximum hold time between consecutive SPFs ... millisec(s)
# SPF algorithm last executed ... ago
# Last SPF duration ... usecs

Those last two lines are the diagnostic that matters when a network feels unstable. “Last executed” repeatedly showing a very recent time, across successive checks, means SPF is being re-triggered continuously — something is flapping. The duration tells you what each run costs the CPU.

Operational commands

# Effective cost on every interface, after every override
show ip ospf interface

# The OSPF-computed routes and their costs, before the RIB
show ip ospf route

# What actually reached the routing table
show ip route ospf
show ip route 10.99.0.0/24

# SPF throttle configuration, last SPF run and its duration
show ip ospf

# The Type 3 summaries an ABR is producing
show ip ospf database summary

# Confirm what is actually committed
show configuration commands | match 'ospf.*cost'

# OSPF events in the system log
show log | match ospf

FRR has no show ip ospf topology command and show ip ospf route takes no prefix argument — it prints the whole OSPF route table. To ask about one destination, use show ip route with the prefix, which reports the winning route across all protocols along with its administrative distance and metric.

A healthy OSPF cost design shows:

• Every backbone interface with an explicit cost, and every other interface with a derived cost you have read rather than assumed.
• The same auto-cost reference-bandwidth on every router, or no reliance on it at all.
• show ip ospf route matching the paths the design intends for every destination that matters.

How it fails — production failure modes

All paths equal — reference bandwidth too low

A network of 1 G and 10 G links running the default reference bandwidth. Every interface derives cost 1, every path of equal hop count ties, and the intent “prefer the 10 G” was never expressed in any number OSPF can see.

Diagnostic: show ip ospf interface reports Cost: 1 on every interface regardless of speed. show ip route for the affected destinations shows two starred next-hops where one was expected.

Fix: set protocols ospf auto-cost reference-bandwidth 10000 — on every router in the domain, in one change. The 1 G links move to cost 10, the 10 G links stay at 1, and the preference re-emerges.

Per-interface cost pointing the wrong way

The operator set cost 50 on eth0 meaning “this link is slow”, without checking that the derived cost was already 10. The link did not become less preferred by a little; it became less preferred by 40, and traffic that should still have used it during normal operation moved off it entirely.

Diagnostic: show ip ospf interface eth0 shows Cost: 50; show ip ospf route shows the affected destinations leaving via an interface the design did not intend.

Fix: decide the cost from the topology rather than from a feeling about the link, and set it on both ends. Cost is directional — each router costs its own outbound interface — so a one-sided change produces asymmetric routing that looks correct from whichever router you happen to check first.

Sub-optimal routing across a summarised boundary

An area’s four /24s are summarised into a /22 at the ABRs. Downstream routers install the /22 with the advertising ABR’s metric and route every one of the four destinations the same way, including the two whose real best exit is the other ABR.

Diagnostic: show ip ospf database summary on a router in area 0 shows the /22 and none of the /24s. show ip route 10.1.2.0/24 resolves through the /22, via the ABR that is not closest to it.

Fix: accept it, or redraw the range so each range’s members share an exit. Un-summarising works too and defeats the purpose.

Cost churn after a flapping link

A link flaps. Each transition re-originates a Type 1, which triggers SPF everywhere, which shifts traffic; the next transition shifts it back.

Diagnostic: show ip ospf run twice a few seconds apart shows “SPF algorithm last executed” resetting to a very recent value each time. show log | match ospf shows repeated adjacency or interface events for the same interface.

Fix: stabilise the link. Raising the SPF hold times damps the symptom and can be the right call while you chase the cause, but it does not make the topology stop changing — it only makes the router notice more slowly. Do not raise a cost to “route around” a flap; the cost is honest and the link is the problem.

Tuning for traffic engineering — a worked example

A four-router fabric: R1 and R2 are edge routers with partner uplinks, R3 is the primary core and R4 the secondary. All the inter-router links are 10 G, so the derived costs would all tie at 1 and express nothing. The design intent — prefer R3, fall back to R4 — has to be written as cost.

# R3, primary core: cheap to everyone
set protocols ospf interface eth0 cost 1     # to R1
set protocols ospf interface eth1 cost 1     # to R2
set protocols ospf interface eth2 cost 1     # to R4

# R4, secondary core: expensive towards the edge, cheap to R3
set protocols ospf interface eth0 cost 100   # to R1
set protocols ospf interface eth1 cost 100   # to R2
set protocols ospf interface eth2 cost 1     # to R3

# R1 and R2, edge: cheap to R3, expensive to R4, so R3 wins
# while it is alive
set protocols ospf interface eth1 cost 1     # to R3
set protocols ospf interface eth2 cost 100   # to R4
set protocols ospf interface eth0 cost 10    # partner uplink

Note that the R1-to-R4 cost is set on both R1 (eth2) and R4 (eth0). Cost is per-direction, and a link costed on one end only is preferred in one direction and avoided in the other — a classic source of asymmetric paths that survive every review because each router’s configuration looks reasonable on its own.

With R3 up, R1 reaches R2 at cost 1 + 1 = 2 via R3, versus 100 + 100 = 200 via R4. When R3 fails, R1’s SPF recomputes and the R4 path at 200 is the only one left, so it is installed. The higher cost never blocked the path; it only ranked it.

Rollback

# Revert a reference-bandwidth or per-interface cost change
rollback 1
commit
save

# Or remove the specific node
delete protocols ospf interface eth0 cost
commit
save

A cost change needs no process restart. Committing it re-originates the router’s LSAs and every router that receives them re-runs SPF on its own throttle schedule; the network converges without further intervention. If a router is genuinely stuck with a stale LSDB — which is a bug, not a normal outcome — the escape hatch is FRR’s:

vtysh -c 'clear ip ospf process'

That is reached through vtysh and is not a VyOS operational-mode command in its own right. It drops every OSPF adjacency on the router it runs on, so it belongs in a maintenance window and not in a diagnostic sequence.

The discipline around cost changes: capture show ip ospf route before and after. The diff names every destination that moved. If destinations you did not intend to touch appear in it, the change had wider reach than the ticket described.

Production discipline

Cross-course references

• Part XVII (XVII-VyOS-RoutingFund) lesson 6 covers routing protocol metrics in general — interface cost is one implementation of the metric concept.
• Part XIV (XIV-VyOS-RoutingTables) covers the kernel’s installation of the OSPF-computed best path into the routing table.
• Part XVIII lesson 4 (vyos-xviii-04-areas) covers area X range, the primitive whose summaries this lesson prices.
• Part XXXVI (XXXVI-VyOS-ECMP) covers equal-cost multi-path handling in detail — the OSPF cost design determines when ECMP kicks in.
• Part XXXV (XXXV-VyOS-Summarisation) covers the BGP-side summarisation primitives that interact with OSPF when redistribution is in play.
• RFC 2328 section 8.1 for the SPF algorithm and section 12.4.3 for the metric of a summary LSA.

Quiz