BGP Session Idle/Active

The incident

Nine days ago a second transit circuit was turned up at the branch site. The circuit tested clean, the peering /30 came up, and the neighbour stanza was pasted from the same turn-up template that had brought up the sibling border router to the same ISP a fortnight earlier.

The session has never reached Established. It cycles: Connect, Active, Connect, Active, with the up/down timer resetting about every thirty seconds. All traffic is still on the primary circuit, which is running hot at peak, and the second circuit is being billed for carrying nothing.

Three engineers have looked at it across the nine days.

• The first pinged the peer. It answers, both directions, no loss. The route to the peer is the connected /30 on eth1. Reachability was declared fine and the ticket moved to the firewall.
• The second reviewed the firewall twice. The local rule permitting TCP/179 from the peer address exists, and its packet counters are incrementing, so packets from the peer really are arriving and really are being accepted.
• The third asked the ISP. Their NOC has checked twice, reports the session configured on their side with the agreed AS numbers, says their state is also cycling, and says they have nothing in their logs.

Nobody has been able to explain the one thing everybody has seen and nobody has commented on: show bgp neighbor records no reason for the session being down. Not a hold-timer expiry, not a bad peer AS, not an administrative shutdown. Nothing at all.

Evidence

vyos@bdr-02:~$ show bgp summary

Neighbor        V   AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
192.0.2.2       4 65100         0         0        0    0    0 00:00:11 Active

The MsgRcvd and MsgSent columns are the ones that matter and the ones nobody read. In nine days this session has neither sent nor received a single BGP message.

vyos@bdr-02:~$ show bgp neighbor 192.0.2.2

BGP neighbor is 192.0.2.2, remote AS 65100, local AS 65000, external link
BGP state = Active
Last read ..., Last write never
Message statistics:
  Sent: opens 0, notifications 0, updates 0, keepalives 0
  Rcvd: opens 0, notifications 0, updates 0, keepalives 0

vyos@bdr-02:~$ ping 192.0.2.2 count 3

3 packets transmitted, 3 received, 0% packet loss

vyos@bdr-02:~$ sudo tcpdump -nn -i eth1 tcp port 179 -c 8

IP 192.0.2.1.50122 > 192.0.2.2.179: Flags [S], seq ..., options [... md5 valid ...]
IP 192.0.2.2.44310 > 192.0.2.1.179: Flags [S], seq ..., options [... md5 ...]
IP 192.0.2.1.50123 > 192.0.2.2.179: Flags [S], seq ..., options [... md5 valid ...]
IP 192.0.2.2.44311 > 192.0.2.1.179: Flags [S], seq ..., options [... md5 ...]

vyos@bdr-02:~$ nstat -az | grep -i md5

TcpExtTCPMD5NotFound            0                  0.0
TcpExtTCPMD5Unexpected          0                  0.0
TcpExtTCPMD5Failure             1826               0.0

The committed neighbour, for completeness:

vyos@bdr-02:~$ show configuration commands | match 'neighbor 192.0.2.2'
set protocols bgp neighbor 192.0.2.2 remote-as '65100'
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast
set protocols bgp neighbor 192.0.2.2 password 'REDACTED'

The secret is redacted here. On the router it is printed in full, which is worth remembering before pasting a configuration dump into a ticket.

Work the evidence before reading on

Four facts are in tension, and the tension is the whole exercise.

1. Packets arrive. The firewall counters increment and tcpdump sees the peer’s SYN land on eth1. Whatever is dropping traffic is not dropping it before the interface.
2. Nothing is answered. There is no SYN-ACK in either direction. Both ends are behaving as if the other end’s SYN never arrived.
3. BGP has recorded nothing. Zero messages, no reset reason. A protocol that cannot explain why it is down is telling you it was never asked to run.
4. The peer NOC’s report matches ours exactly. Both sides cycling, both sides with nothing logged. A one-sided fault does not usually produce a symmetric symptom.

Before continuing: if a segment arrives on the wire, is visible to tcpdump, and is nonetheless never answered by TCP, what sits between those two events - and where would that thing write down what it did?

Root cause

1. The authentication is below BGP, and so is the failure

The neighbour carries a password, which on VyOS 1.5 LTS configures the RFC 2385 TCP MD5 Signature Option. The digest covers the TCP pseudo-header, the TCP header, the segment data and a shared key. It is computed and verified by the kernel’s TCP stack, not by bgpd.

The key on this router does not match the key the peer holds for this session. Every segment the peer sends is discarded by the local kernel before TCP state processing; every segment this router sends is discarded by the peer’s kernel for the same reason. The three-way handshake cannot complete in either direction, so:

• no OPEN is ever sent, in either direction;
• no capability negotiation, AS check or hold-time negotiation ever happens;
• neither speaker has a session on which a NOTIFICATION could be raised.

That is the answer to the puzzle in fact 3. show bgp neighbor has no reset reason because there has never been anything to reset. RFC 4271 deprecated OPEN subcode 5, “Authentication Failure”, for exactly this reason: BGP handed authentication to TCP and stopped reporting on it. There is no BGP message that can say “your key is wrong”, and there never will be.

2. The state alternates because the ConnectRetry timer is doing its job

Connect is “attempting the TCP handshake”. Active is “the handshake failed, retrying”. With the ConnectRetry timer at its VyOS 1.5 LTS default of 30 seconds, a peer that can never complete a handshake cycles between those two states on a 30-second period forever. The oscillation is not instability; it is the FSM working correctly on a transport that will never open.

3. The kernel counters name the failure and name the side

The kernel keeps three counters for this, and they are the only place in the entire incident where the cause is written in plain language:

Here TcpExtTCPMD5Failure is climbing and the other two are at zero. Both ends are authenticating and the keys disagree. Had the counter been NotFound, the peer would simply never have configured a key and the conversation with the NOC would be a different one.

4. Why the template was the misdirection

The turn-up template is fine. What was wrong was one value inside it. The peering record for this ISP has a row per session, and the engineer at turn-up copied the row for the sibling border router’s session with the same provider. The key pasted into this router is a real, current, correct key - for a session on a different box.

This is why “the same template worked on the other router” felt like strong evidence and was worthless. It proved the shape of the configuration was right. Every per-session value inside it was untested.

Resolution

1. Confirm the finding before changing anything. Read nstat -az | grep -i md5 twice, sixty seconds apart, and confirm TcpExtTCPMD5Failure is climbing while the other two counters stay at zero. This is a thirty-second check and it decides whether the next hour is spent on keys or on the firewall.
2. Confirm out of band with the peer NOC which peering-record row governs this neighbour: local router, peer address, key identifier. Do not send or ask for the secret itself over the ticket or the chat channel.
3. Retrieve the correct secret for this session from the approved secret store.
4. Apply it: configure, then set protocols bgp neighbor 192.0.2.2 password with the retrieved value, then commit and save. The neighbour already exists, so this is a single-node change and compare should show exactly one line.
5. Watch the counters rather than the session. nstat -az | grep -i md5 should stop moving within one ConnectRetry interval. If it keeps climbing, the new key is also wrong, and the session state will not tell you that any faster than the counter will.
6. Confirm the session with show bgp summary, then read show bgp neighbor 192.0.2.2 for the negotiated hold time and the message counters. Non-zero opens and keepalives in both directions is the first proof BGP has ever run on this session.
7. Record the key identifier and rotation reference in the change record. Record the secret nowhere.

If the peering-record row cannot be confirmed inside the window, stop and hold. Leave the session down, leave the traffic on the primary circuit, and hand the hold to a named owner with an explicit end time. Trying a second key on a guess produces evidence that looks exactly like the current state, and the next engineer inherits a router where they cannot tell which of two wrong keys is loaded.

Verification

1. All three TCP-MD5 counters are unchanged across two reads sixty seconds apart. A session that establishes while TcpExtTCPMD5Failure still climbs means something else on this router is mis-keyed and is being masked.
2. show bgp summary shows Established, and the up/down timer climbs on successive reads rather than resetting.
3. show bgp neighbor 192.0.2.2 shows non-zero opens and keepalives sent and received, and a negotiated hold time consistent with the peering agreement.
4. The session survives longer than the negotiated hold time. Established is reached before the first hold interval has ever been tested, so a ten-second-old session proves the handshake worked and proves nothing about keepalive delivery.
5. The received prefix count matches what the peering agreement says the provider sends. A session that establishes and receives an unexpected count is a policy problem you have just uncovered, not a successful verification.
6. The prefixes are installed, not merely present. show route for a sample of received prefixes shows them in the routing table with the expected next-hop, and show ip route summary reflects the new source.
7. Traffic actually moves. Confirm the second circuit is carrying its share before closing the incident, because the business symptom was a hot primary circuit, not a red status line.

Prevention

• Read the absence of a NOTIFICATION as a positive finding. A session that has never left Connect or Active has exchanged no BGP message, so BGP cannot have recorded a reason. The empty Last reset line is a pointer to a layer below BGP, and it is one of the few diagnostics in this course that is delivered by something not being there.
• Put nstat -az | grep -i md5 on the BGP turn-up checklist, directly after the reachability test and before anyone is allowed to open a firewall ticket. It costs one command and it eliminates an entire branch of the search tree.
• Give every session its own row in the peering record, keyed by local router and peer address rather than by provider name. This incident is what a provider-keyed record produces the first time a second session to the same provider exists.
• Carry a key identifier alongside every secret, and record the identifier - never the secret - in change records and tickets. An identifier makes a mis-copied row visible during review; a masked secret does not.
• Treat “the same template worked elsewhere” as evidence about the template only. Templates are correct about shape and silent about values, and every value in a peering stanza is per-session.
• Plan key rotation as a routing change with a window and a peer cutover, because TCP-MD5 has no key overlap. A rotation improvised on a live transit session produces this incident deliberately.

Related material

• vyos-xxxi-01-session-states — the BGP finite state machine, the NOTIFICATION code table, and the kernel TCP-MD5 counters.
• vyos-xxiv-06-bgp-session-troubleshoot — the five-layer diagnostic model this scenario walks.
• vyos-xxiv-04-bgp-authentication — the password node, TCP-MD5 versus TCP-AO, and the coordinated rotation sequence.
• vyos-xxiv-05-ebgp-multihop — the other common reason a directly connected peer sits in Active, and why raising TTL is not a response to Active on its own.