BGP session states — Idle, Connect, Active, OpenSent, OpenConfirm, Established, and the Notification codes that knock a peer out of each

A BGP session between two routers progresses through a strict finite state machine with six top-level states: Idle, Connect, Active, OpenSent, OpenConfirm, and Established. The state machine is deterministic and strictly ordered: a session cannot skip from Idle to Established, and a session that drops out of Established always returns to Idle before trying again. The operator who understands the FSM can read show ip bgp summary and immediately know whether the problem is TCP-layer (Active), BGP-layer capability negotiation (OpenConfirm), or hold-time expiry (Established → Idle).

This lesson walks the FSM in order, names what each state requires to advance, and pairs each state with the failure mode that keeps a session stuck in it. The goal is not memorisation — the goal is the ability to look at a show ip bgp summary line and immediately say “Active 0:00:00 — the TCP three-way handshake is failing” or “OpenConfirm 0:01:23 — a capability is wrong” without running any other command.

Why a finite state machine and not just “connected”

BGP runs over TCP port 179. The transport layer is reliable: TCP handles retransmission, ordering, and connection state. BGP’s own FSM exists because BGP has session-scoped concerns that TCP does not cover: peer authentication, capability negotiation, route refresh handling, graceful restart signalling, and the hold-time contract. The FSM is the protocol-level overlay on top of TCP that ensures both peers agree on the same parameters before they exchange a single UPDATE.

The FSM is also the operator’s diagnostic surface. A session stuck in a state has a name for the failure: stuck-Active means TCP is failing, stuck-OpenConfirm means a capability is wrong, stuck-OpenSent means the peer is not responding to the OPEN. The state is the first hint; the rest is evidence in the log and the BGP NOTIFICATION codes.

The six states in order

stateDiagram-v2
  [*] --> Idle
  Idle --> Connect: peer configured, ConnectRetry timer fires
  Connect --> OpenSent: TCP three-way handshake completes
  OpenSent --> OpenConfirm: valid OPEN received, hold-time agreed
  OpenConfirm --> Established: KEEPALIVE received
  Established --> Idle: hold-time expires, NOTIFICATION sent, operator clears, peer resets TCP
  Connect --> Active: TCP handshake fails (passive)
  Active --> OpenSent: TCP handshake eventually succeeds
  Active --> Idle: ConnectRetry timer expires repeatedly
  OpenSent --> Active: the TCP connection fails before the peer's OPEN arrives
  OpenConfirm --> Idle: NOTIFICATION received or hold-time expires

The terminal state is always Idle. Every failure path returns the session to Idle, where the ConnectRetry timer waits before the next attempt. Its value is per neighbour rather than a single global constant — read the configured value out of show ip bgp neighbors <peer-ip> rather than assuming one, and set it deliberately with set protocols bgp neighbor <peer-ip> timers connect <seconds> if the default cadence is wrong for the design. What matters diagnostically is the shape, not the number: a peer cycling Idle → Connect → Active on a fixed cadence has a connectivity problem, while a peer stuck in OpenConfirm for hours has a protocol-layer problem.

State 1 — Idle

The initial state. The router is configured with the peer but is not trying to connect. BGP has refused any incoming connection. The state lasts until the configuration enables the peer — on VyOS, until delete protocols bgp neighbor <peer-ip> shutdown is committed — and the ConnectRetry timer fires.

On VyOS 1.5 LTS, a peer in Idle because of a configuration mistake usually means set protocols bgp neighbor <peer-ip> shutdown is present, or the peer sits in a different VRF from the BGP process that is meant to reach it, or the peer address is on an interface that is administratively down. The operator checks show configuration commands | match <peer-ip> for the shutdown clause.

State 2 — Connect

The router is attempting the TCP three-way handshake. BGP opens a TCP socket to the peer’s IP on port 179. If the handshake completes within the ConnectRetry timer, the state advances to OpenSent. If the handshake fails (no SYN-ACK, ICMP unreachable, ACL on the peer drops the SYN), the state falls back to Active.

Evidence in show ip bgp summary: the state column shows a number with a colon (Connect 0:00:05). The number is the time spent in the state. A session that cycles Idle → Connect → Active on the retry cadence spends most of its time in Connect, waiting for the SYN-ACK that never comes.

State 3 — Active

The router has failed the TCP handshake and is now retrying. BGP still owns the session; the peer is configured; the transport layer just is not cooperating. The router continues to attempt the TCP handshake periodically. If a handshake finally succeeds, the state advances to OpenSent.

A session stuck in Active is the canonical “TCP is failing” symptom. The most common causes:

• No route to the peer IP. show ip route <peer-ip> returns nothing. The BGP process can SYN, the SYN has no path, the SYN-ACK never comes.
• ACL on the peer side blocks TCP 179. The SYN reaches the peer, but the peer’s input filter drops it. tcpdump -nn -i any tcp port 179 on the local side shows SYNs going out, no SYN-ACKs coming back.
• Firewall between the two peers blocks TCP 179. Same symptom as the previous point but the firewall is not on the peer.
• The peer is administratively down or crashed. No process is listening on TCP 179.
• The peer’s BGP daemon has the local IP in its neighbor <ip> shutdown list.
• TCP-MD5 password mismatch. RFC 2385 authentication belongs to TCP, not to BGP. The kernel signs every outbound segment and verifies every inbound one; a segment whose signature does not match the configured key is discarded before TCP processes it. The three-way handshake therefore never completes, no OPEN is ever exchanged, and no BGP NOTIFICATION is sent in either direction. The peer oscillates between Connect and Active exactly as it would for a blocked port.

The operator who sees Active should walk these in order: route first, then ACL, then BGP process state, then the TCP-MD5 keys.

Telling a TCP-MD5 mismatch apart from any other TCP-layer failure is a kernel question, not a BGP question. The kernel keeps three counters for it:

nstat -az | grep -i md5

• TcpExtTCPMD5Failure — a segment arrived carrying a signature that does not match the local key. The two passwords differ.
• TcpExtTCPMD5NotFound — a segment arrived with no signature, but a key is configured for that peer. Only the local side has authentication configured.
• TcpExtTCPMD5Unexpected — a segment arrived carrying a signature, but no key is configured for that peer. Only the remote side has authentication configured.

Any of the three incrementing while the peer sits in Active is the signature. If all three stay at zero, authentication is not the problem and the operator goes back to routing and filtering.

State 4 — OpenSent

The TCP connection is up. The router has sent its OPEN message and is waiting for the peer’s OPEN. The OPEN carries the local AS number, BGP version (always 4 in 2026), router-id, hold-time, and the optional capabilities list (multiprotocol, route-refresh, graceful restart, ADD-PATH, and so on).

A session stuck in OpenSent usually means:

• The peer is slow. A busy router may take seconds to respond. OpenSent for a few seconds is normal.
• The peer sent an OPEN but BGP cannot parse it. Rare in modern FRR; usually indicates a corrupted TCP stream.
• The peer is in a different AS than expected and rejects the OPEN with NOTIFICATION 2/2 (Bad Peer AS). The peer sends NOTIFICATION, the local state drops to Idle, and the cycle restarts. The evidence is on the Last reset line of show ip bgp neighbors <ip>.
• BGP version mismatch. BGP-4 is the only version in use in 2026; this is historical.

State 5 — OpenConfirm

The router has sent its OPEN and received a valid peer’s OPEN. Both peers have agreed on hold-time and capabilities. The router is now sending periodic KEEPALIVE messages and waiting for the first KEEPALIVE from the peer.

A session stuck in OpenConfirm is the canonical “capability mismatch” symptom. Common causes:

• No address family in common. On VyOS 1.5 LTS a neighbour needs an explicit address-family node; a peer configured with remote-as alone negotiates no AFI/SAFI at all. FRR sends NOTIFICATION 2/7 (Unsupported Capability) and records the reset as No AFI/SAFI activated for peer.
• The peer demands a capability the local router does not advertise. A peer configured for a strict capability match tears the session down with NOTIFICATION 2/7 rather than settling for the intersection of the two capability sets.
• The local hold-time is larger than the peer’s hold-time. The peer may have sent a smaller hold-time in its OPEN and the local router refuses to lower its own hold-time. NOTIFICATION 2/6 (Unacceptable Hold Time). The fix: lower the local hold-time to match.

Two failures that are not on that list. RFC 5492 requires a speaker to ignore a capability it does not recognise, so advertising ADD-PATH or graceful restart to a peer that does not implement it does not by itself drop the session — the two sides negotiate the intersection and carry on. And a TCP-MD5 password mismatch never reaches OpenConfirm at all: it kills the session in Connect or Active, before the first OPEN is sent.

OpenConfirm should be brief (sub-second on healthy links). A session in OpenConfirm for more than 2-3 seconds has a real problem.

State 6 — Established

The session is fully operational. Both peers are exchanging UPDATE, KEEPALIVE, and (on route changes) WITHDRAWN messages. The state persists until one of three things happens:

• The hold-time expires without a KEEPALIVE or UPDATE. The router sends NOTIFICATION 4/4 (Hold Timer Expired) and drops to Idle. The default hold-time is 180 seconds in FRR; keepalive is 60 seconds. A session that cycles Established → Idle every 180 seconds has lost its peer and is hitting hold-time expiry.
• A NOTIFICATION is received. The peer sent a fatal error; the session drops. The reason is in the NOTIFICATION code and subcode.
• The operator clears the session. reset bgp <peer-ip> on VyOS (FRR’s own spelling is clear bgp <peer-ip>, reachable through vtysh) forces a NOTIFICATION 6/4 — Cease, subcode Administrative Reset — and drops the session.

Established is the only state in which UPDATEs are accepted. In every other state, BGP holds its updates and applies them only when the FSM reaches Established.

The six NOTIFICATION error codes

RFC 4271 defines the NOTIFICATION message format with a 1-byte error code and a 1-byte error subcode. There are six error codes, and the subcode space is per code:

The operator who sees a NOTIFICATION in show ip bgp neighbors <ip> reads the code/subcode and immediately knows what the peer is complaining about:

• Code 2 Subcode 2 (Bad Peer AS) — the local router’s AS in its OPEN does not match the peer’s neighbor <ip> remote-as. The fix is to correct the AS number.
• Code 2 Subcode 4 (Unsupported Optional Parameter) — the OPEN carried an Optional Parameter whose type the peer does not recognise. This is not the same as an unrecognised capability: capabilities travel inside Optional Parameter type 2, and RFC 5492 requires a speaker to ignore individual capabilities it does not understand rather than reject the OPEN. Modern peers rarely send 2/4.
• Code 2 Subcode 7 (Unsupported Capability) — the peer requires a capability the local router did not advertise, or the two capability sets leave no address family in common. On VyOS the usual cause is a neighbour configured with remote-as and no address-family node.
• Code 2 Subcode 6 (Unacceptable Hold Time) — hold-time negotiation failed. The fix is to lower the local hold-time.
• Code 3 Subcode 8 (Invalid NEXT_HOP Attribute) — the NEXT_HOP carried in an UPDATE is not a legal address for the receiver to use. RFC 4271 §6.3 requires that it be a valid IP host address, that it not be the receiving speaker’s own address, and that on a single-hop eBGP session it share a subnet with the receiver. This is an address-validity error, not a reachability error: a NEXT_HOP that is merely unresolvable in the receiver’s routing table produces no NOTIFICATION at all. The route is held as unusable, never installed, and the session stays Established.
• Code 4 Subcode 0 (Hold Timer Expired) — the peer did not send a KEEPALIVE or UPDATE inside the hold-time. Either the TCP connection has stalled or the peer’s CPU is too busy to generate one.
• Code 6 (Cease), subcode 2 (Administrative Shutdown) or 4 (Administrative Reset) — somebody shut the neighbour down in configuration, or cleared the session by hand. The fix is to find out who, and why. Subcode 1 (Maximum Number of Prefixes Reached) is the other one worth memorising: the peer hit its inbound prefix limit and tore the session down.

Two entries the operator will look for and not find. OPEN subcode 5 was “Authentication Failure” in RFC 1771 and is deprecated in RFC 4271: BGP has no authentication of its own, and the TCP MD5 Signature Option of RFC 2385 that replaced it sits below BGP entirely, so a password mismatch cannot be reported by a BGP message. UPDATE subcode 7 was “AS Routing Loop” in RFC 1771 and is deprecated too; a route whose AS_PATH already contains the local AS is silently discarded on receipt and the session is not touched.

The NOTIFICATION code is the BGP process’s way of saying “I am dropping the session because of X”. The operator who reads the code first has the answer — but only when there is a code to read. A failure below the BGP layer leaves no NOTIFICATION anywhere.

How the result is validated

show ip bgp summary
show ip bgp neighbors <peer-ip>
show ip bgp neighbors <peer-ip> advertised-routes
show ip bgp neighbors <peer-ip> received-routes
vtysh -c 'show debugging bgp'
nstat -az | grep -i md5
journalctl -u frr | grep <peer-ip>
monitor traffic interface <intf> filter 'tcp port 179'
tcpdump -nn -i any tcp port 179

The first command shows the state and the up/down timer for every peer. The second shows the FSM state for one peer and its Last reset line, which is where a NOTIFICATION sent or received is recorded. The third and fourth show the routes exchanged with that peer — note that received-routes prints nothing useful unless the neighbour has soft-reconfiguration inbound configured or the route-refresh capability was negotiated, because otherwise FRR does not keep a copy of what arrived before policy. If it answers with a message about inbound soft reconfiguration rather than a route list, that is what it is telling you. The fifth shows FRR’s debugging state. The sixth reads the kernel’s TCP-MD5 counters, the only place a password mismatch is visible. The seventh is the journal filtered by peer IP — the FSM transition log is here. The last two are the packet-level evidence for the TCP handshake and the BGP messages.

How it fails

The production failure modes the engineer must recognise:

• Session stuck in Active for hours with no NOTIFICATION recorded. TCP is failing and BGP never ran. The fix is to walk: route to peer → filter on the local router → filter on the peer → BGP process state on the peer → TCP-MD5 keys, confirmed against nstat -az | grep -i md5.
• Session stuck in OpenSent for more than 5 seconds. The peer is not responding to the OPEN. The fix is to check the peer’s BGP process and the AS number match.
• Session stuck in OpenConfirm for more than 2 seconds. Capability negotiation. The fix is to compare the capability lists on both sides with show ip bgp neighbors <ip> and identify the mismatch.
• Session cycles Established → Idle every 180 seconds with “Hold Timer Expired”. The link is dropping packets or the peer CPU is saturated. The fix is to check the link and the peer load.
• Session drops immediately after Established with “Cease”. The peer operator is clearing the session. The fix is to coordinate with the peer operator.
• Session stays Established but the peer’s routes never appear in the table. The NEXT_HOP is unresolvable, so the routes are held as unusable rather than installed. This is not a session failure and produces no NOTIFICATION. The fix is either next-hop-self on the advertising router or an IGP that carries the NEXT_HOP.
• Session has an unstable up/down timer (up 5s, down 30s, up 5s, …). The peer is bouncing. The fix is to look at the peer’s log or call the peer NOC.

Rollback

Session-state changes are operational changes, not configuration changes. The rollback is the same as the change: clear the session again or fix the underlying problem.

• reset bgp <peer-ip> — hard reset, from VyOS operational mode. Drops the TCP session and restarts the FSM from Idle.
• vtysh -c 'clear bgp 192.0.2.2 soft in' — re-request routes from the peer without dropping the session. Inbound soft reset needs either the route-refresh capability (negotiated by default between modern peers) or soft-reconfiguration inbound configured on the neighbour.
• vtysh -c 'clear bgp 192.0.2.2 soft out' — re-advertise this router’s routes to the peer without dropping the session.
• set protocols bgp neighbor <peer-ip> shutdown and the matching delete to administratively shut a peer and bring it back. Note the 1.5 shape: the local AS is a leaf (set protocols bgp system-as <asn>) and neighbours hang directly off protocols bgp, so there is no AS number in the neighbour path.

The hard reset is the operator’s “reboot the BGP session” lever: it forces the FSM to restart from Idle. It also withdraws every route learned from that peer for the duration, so on a transit session prefer the soft forms unless the FSM itself is the thing under test.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel TCP/IP stack. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the equivalent FRR FSM on the firewall side. The BGP lessons vyos-xxiv-01-bgp-config and vyos-xxiv-02-bgp-neighbour cover the configuration of the peer; vyos-xxiv-06-bgp-session-troubleshoot covers the broader session debugging. The lesson vyos-xxxii-03-bfd-with-bgp covers BFD integration that turns a silent failure into a fast failure.

Quiz