high riskservice affecting~45 min

Runbook: Troubleshoot a BGP Session

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The symptom is stated as a STATE and a duration, not as "BGP is down". show bgp summary prints one of Idle, Connect, Active, OpenSent, OpenConfirm or a prefix count, together with an Up/Down timer. Those two values already narrow the fault to one layer, and everything in this runbook is ordered by them
· The peer actually appears in show bgp summary. A neighbour that is absent entirely is a configuration problem - not committed, wrong VRF, or shut down - and no amount of transport debugging will find it
· Every other session on this router has been read at the same time, not only the one you were paged about. One peer down is a peer problem; several peers down together is a local problem and the investigation starts somewhere else
· The change history has been read rather than assumed: show configuration commit. A session that went down within minutes of a commit is a rollback, and vyos-rb-rollback-bad-config gets there faster than any diagnosis
· You know whether this session carries the route to your own management station. A hard reset on the session that holds your access path is a self-inflicted lockout, and it is not obvious from the neighbour stanza
· You know what this session is the only path for. A transit session with no backup and a peering session with three alternates justify completely different urgency and completely different containment
· The current neighbour state has been captured to the incident record BEFORE any reset. show bgp neighbor 203.0.113.2 carries the Last reset line, the negotiated timers, the capability list and the message counters, and the next reset overwrites all of it
· The case has been separated from its neighbours: a session that is Established but carries no prefixes is vyos-rb-troubleshoot-missing-route, and a session that establishes and drops on a cadence is a flap, which is diagnosed by its cadence rather than by its state

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read show bgp summary and record the state and the Up/Down timer for every peer. Write both into the incident record now; the timer is the only thing that distinguishes a session that has been down since Tuesday from one that resets every ninety seconds.
2Read show bgp neighbor 203.0.113.2 and go to the Last reset line FIRST. When a NOTIFICATION caused the reset, the line names the direction - sent or received - and the code and subcode, and that is most of the answer for the price of one command.
3If there is no reset reason to read at all, treat that absence as the finding. It means no BGP message has been exchanged in either direction, so the failure is below BGP, and every protocol-layer check you are about to run is guaranteed to be silent.
4Read what was committed rather than what you remember typing: show configuration commands | match "bgp|neighbor|update-source|password". Check remote-as, update-source, VRF, the address-family node and any shutdown before touching the network.
5For a session in Active or Connect, walk transport in cost order: route to the peer, route back to your source, ping from the configured source address, then the socket, then a capture. The route check is free and is the answer more often than the capture is.
6Read the capture by DIRECTION, not by volume. No outbound SYN is a local source or socket problem. SYN out with no SYN-ACK back is a path or a filter. A completed handshake followed by a reset is transport authentication or the BGP contract. Each of those points at a different owner.
7Rule TCP-MD5 in or out from the kernel rather than from BGP: nstat -az | grep -i md5. A password mismatch produces no NOTIFICATION anywhere, because the kernel discards the mis-signed segments before BGP ever runs. Three counters distinguish three different mistakes.
8For a session in OpenSent, compare the OPEN parameters: the configured remote-as against what the peer believes its AS to be, and the router-id for a collision. NOTIFICATION 2/2 (Bad Peer AS) on the Last reset line settles it immediately.
9For a session in OpenConfirm, compare capabilities and address families. On VyOS a neighbour configured with remote-as and no address-family node negotiates no AFI/SAFI at all, and FRR records the reset as no AFI/SAFI activated for the peer.
10For a session that establishes and drops on a regular cadence, stop treating the state and start reading the cadence. Seconds points at the interface, sub-second at BFD, the hold time at the path or the peer CPU, and anything matching a cron schedule at your own automation.
11Change ONE contract at a time, with compare and commit-confirm, and read the state after each. A change to source, TTL, timer, password and policy in a single commit produces a session whose behaviour nobody can attribute.
12Prefer a soft reset to a hard one: reset bgp ipv4 203.0.113.2 soft in re-requests the table without dropping the session. Use the hard reset bgp ipv4 203.0.113.2 only when the FSM itself is what you are testing, and know that it withdraws every route from that peer for the duration.
13Verify with the same commands you started with, take a second reading a few minutes later to prove the session is stable rather than merely up, then save and record what was actually wrong - not what you tried.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show bgp summary shows Established, and a SECOND reading several minutes later shows an Up/Down timer that has increased by the elapsed time. A session that is Established the moment you look at it may be four seconds into its next cycle
✓The received prefix count matches what this session is contracted to carry, not merely something above zero. A transit session showing eleven prefixes is up and useless
✓The Last reset line in show bgp neighbor 203.0.113.2 now attributes the last reset to your own action and nothing newer has appeared since
✓The routes are present in the BGP table AND installed in the FIB - show ip bgp and show ip route - because a session that is up with routes held as unusable looks identical in the summary
✓The service the session exists for works end to end, measured from where the traffic actually originates rather than from the router
✓No other session on the router changed state during the work. Diagnostic resets, timer changes and policy edits reach further than the neighbour they were aimed at
✓The kernel TCP-MD5 counters in nstat -az | grep -i md5 are static across two readings, if authentication was ever a hypothesis
✓show configuration commit shows the archive entry for the fix, and diff /config/config.boot <(show configuration commands) is empty so the fix survives a reboot

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Roll back the narrowest thing that was changed, not the router. Wrong neighbour address: delete and recreate that neighbour. Wrong source: restore the previous update-source and prove its route. Wrong timer: restore the previous value and read the negotiated hold time
↶Wrong AS: restore the previously negotiated value on ONE side and re-read the peer state before touching the other. Changing both ends at once produces a mismatch that looks exactly like the one you started with
↶Wrong secret: restore the previous key through the secret process, never by pasting it into the incident record. The configuration containing a BGP password is a credential artefact and the incident record is not a credential store
↶Wrong policy: remove the neighbour-specific or address-family override you added, not the global baseline it was overriding. Deleting the baseline to clear an override is how one session outage becomes an estate-wide one
↶Run compare N before rollback N, then rollback N, compare, commit-confirm, verify, confirm, save. The diff covers every commit since that archive entry, which during an incident regularly includes somebody else responding to the same incident
↶Remove any diagnostic ebgp-multihop or ttl-security that was added to test a hypothesis and did not fix anything. Both are permanent security exceptions, both look harmless in a green status page, and neither belongs in a configuration as a leftover
↶POINT OF NO RETURN: a hard reset that has already run cannot be taken back. Every prefix learned from that peer was withdrawn and re-advertised, everything downstream of you saw the withdrawal, and restoring the configuration does not restore the traffic that moved in between
↶POINT OF NO RETURN: the pre-policy view of what the peer sent is gone once the session resets, and it was only ever retained where soft-reconfiguration inbound was configured in advance. Enabling it now shows you what the peer is sending now, which is a different question

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the peer NOC when the evidence stops at the demarcation: your SYN leaves and nothing returns, or the peer sends a NOTIFICATION you did not provoke. Give them the state, the timer, the reset reason and the capture direction - not "BGP is down"
· Escalate to network engineering when the session is one of a redundant pair or is the only transit path. Whether to keep trying or to fail traffic away is a routing-design decision with a business cost, and it is not a call the on-call operator should be making alone
· Escalate to security when the fault is authentication. A TCP-MD5 mismatch is either a key rotation that half-completed or a credential changed without a ticket, and both need somebody to say which before the key is replaced
· Escalate to whoever owns the physical layer when interface errors or carrier transitions correlate with the session flap. A BGP session riding a failing optic is a symptom and no BGP change fixes it
· Escalate to the change owner, and switch to the rollback runbook, when the session state changed within minutes of a commit. Diagnosing a self-inflicted outage from first principles wastes the window in which reverting is trivial
· Escalate rather than continue after two hypotheses have been tested and neither moved the state. Two failures usually means the layer was misidentified at the start, and a third change on the same model makes the evidence harder to read for whoever comes next

BGP is unusually generous to whoever is debugging it. The finite state machine is strictly ordered, every failure path returns to the same place, and the state a session is stuck in names the layer that is broken. A session in Active is failing TCP. A session in OpenConfirm is failing capability negotiation. A session cycling out of Established on the hold time is losing packets or talking to a router too busy to answer.

Two commands read that out: show bgp summary for the state and the timer, and show bgp neighbor for the Last reset line. They cost about fifteen seconds together, and they are skipped constantly — because a page that says “BGP down” produces a reflex to capture packets, and a packet capture will happily show you an hour of correct TCP behaviour on a session whose remote-as is one digit wrong.

The second thing this runbook is for is the reset. Clearing a session is the most available action in the incident and it destroys the evidence for it. Do it last, deliberately, and knowing what it costs everyone downstream.

When to use this runbook

A configured BGP neighbour on a VyOS router is not in Established, or does not stay there. That covers a session stuck in a state, a session flapping on a cadence, and a session that establishes and drops when a specific thing happens.

When not to use it

The session is Established and the prefixes are missing. Nothing about the session is wrong; the route is being filtered, not selected, or not installed. That is vyos-rb-troubleshoot-missing-route.
The session is Established and carrying far too much. That is vyos-rb-respond-bgp-route-leak, which is a policy incident with a clock.
The neighbour does not appear in show bgp summary at all. It was never committed, it is in another VRF, or it is shut down — a configuration problem, answered by vyos-rb-configure-bgp-neighbour.
The state changed at the moment of a commit. Revert first (vyos-rb-rollback-bad-config) and diagnose from a known state afterwards.

Blast radius

Reading is free. Everything else here is not, and the sizes are unequal enough to be worth knowing before you choose.

A hard reset withdraws every prefix learned from that peer, drops everything you were advertising to it, and re-exchanges the table on re-establishment. On a full-table session that is a CPU event locally and a convergence event for everyone downstream of you. A soft reset re-requests or re-sends the table without touching the session, and costs a fraction of that.

A timer change is negotiated with the peer and applies to the session, not just to your side. A multihop or TTL change is a permanent security exception. A policy change on a session that is not currently up will not be visible until it comes up, which is the classic way two faults get fixed into one confusing state.

And if this session is the path your management traffic takes, the blast radius includes you.

Step 1: the state and the timer, for every peer

Read-only / Safefifteen seconds, most of the answer

show bgp summary

Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down    State/PfxRcd
203.0.113.2     4  65100         0        0  00:31:07    Active
198.51.100.9    4  64500      2841     2790  19:22:07       312
192.0.2.130     4  65000     14022    13998  6d02h11m      4108

Illustrative output

Read all three rows even though you were paged about one. Two peers down together is a local fault — an interface, a VRF, the routing daemon — and starting from the neighbour stanza of one of them wastes the first twenty minutes. One peer down among healthy siblings is a fault on that adjacency.

Then read the state against this map, which is the whole reason the FSM is worth knowing:

State	What has already succeeded	What is failing
Idle	Nothing; not trying	Administratively shut down, wrong VRF, or the peer address sits on a down interface
Connect / Active	The peer is configured and BGP is trying	The TCP three-way handshake. Route, filter, peer process, or TCP-MD5
OpenSent	TCP is up, OPEN sent	The peer’s OPEN. AS mismatch, router-id collision, peer not answering
OpenConfirm	Both OPENs exchanged	Capability or address-family negotiation, or hold-time agreement
Established, cycling	Everything, repeatedly	Whatever is interrupting it. Read the cadence, not the state

Step 2: the Last reset line

Read-only / Safethe peer's own account of why it left

show bgp neighbor 203.0.113.2

This output is long and the operator only needs three parts of it on the first pass: the FSM state, the negotiated timers, and the Last reset line. That line records how long ago the session last dropped and why — and when a NOTIFICATION was involved, it names the direction and the code and subcode.

The codes worth recognising without looking anything up:

2/2 Bad Peer AS — the AS in the OPEN does not match the configured remote-as. Fix the number; stop debugging.
2/6 Unacceptable Hold Time — hold-time negotiation failed. Lower the local hold time.
2/7 Unsupported Capability — on VyOS, almost always a neighbour with remote-as and no address-family node, which negotiates no address family at all.
4/0 Hold Timer Expired — you stopped hearing from the peer inside the hold time. The path is dropping packets or the peer’s CPU cannot generate a KEEPALIVE.
6/2 or 6/4 Cease, Administrative Shutdown or Reset — a human did this. Find out which human before undoing it.
6/1 Cease, Maximum Number of Prefixes Reached — the peer hit its inbound limit because of what you were sending. That is an outbound policy incident.

Step 3: read what was committed

Read-only / Safethe configuration, not the memory of it

show configuration commands | match "bgp|neighbor|update-source|password"
show configuration commit

Check five things: the remote-as value, the update-source, the VRF the neighbour lives in, whether an address-family node exists, and whether a shutdown is set. Each of them produces a distinct stuck state and each of them is invisible from the summary.

show configuration commit is here for a different reason. If the newest archive entry lands within a few minutes of the session going down, this is a self-inflicted outage, and reverting it is faster and more certain than diagnosing it.

Step 4: transport, in cost order

For Connect or Active, the layers are cheap to expensive and the cheap one is frequently the answer.

Read-only / Saferoute first, capture last

show route 203.0.113.2
show route 192.0.2.1

Both routes, not just the outbound one. A session needs a path to the peer and a path back to whatever source address the router will use — and where update-source points at a loopback, the peer needs a route to that loopback through whatever policy sits between you.

Then the capture, read by direction rather than by volume:

Read-only / Safeprove the source address is usable, then look at the socket

# Substitute your own values before running:
PEER=203.0.113.2
SRC=192.0.2.1

ping -c 3 -I "$SRC" "$PEER"
ss -tnp | grep ':179'

ping is not a BGP test and a successful one proves nothing about the session. A failed one, sourced from the address the neighbour is configured to use, explains why BGP cannot start — which is the more useful of the two outcomes.

Read-only / Safedirection is the diagnosis

# Substitute your own value before running:
PEER=203.0.113.2

sudo tcpdump -nn -i any "host $PEER and tcp port 179"

What the capture shows	What it means	Who owns it
No outbound SYN at all	Local: no route, wrong source, socket not opened, neighbour shut down	You
SYN out, no SYN-ACK back	Path or filter between here and the peer, or the peer is not listening	You, then the peer
Handshake completes, then a reset before a stable OPEN	Transport authentication, or the BGP contract itself	You and the peer together
Handshake, OPEN, then a NOTIFICATION	BGP-layer. Go back to the Last reset line; it has the code	Whoever the code names

Active is not an argument for `ebgp-multihop`

The reflex when a loopback-sourced session sits in Active is to raise the TTL and see if it helps. Resist it.

A missing source route, a return-path ACL or a peer pointing at the wrong address are all fixed by finding the real fault, and none of them are fixed by raising the TTL. What raising it does do is create a permanent exception — your router will now accept a BGP session from further away than the design says it should — and that exception survives the incident, sits in the configuration looking deliberate, and is nobody’s to remove.

Prove the path, the source, the return route and the peer address first. If the design genuinely is a routed loopback peer, then ebgp-multihop is correct, belongs in the ticket, and gets a hop count that matches the actual topology rather than a round number.

Step 5: rule TCP-MD5 in or out from the kernel

Read-only / Safethe only place a password mismatch is visible

nstat -az | grep -i md5

TCP-MD5 authentication belongs to TCP, not to BGP. The kernel signs every outbound segment and verifies every inbound one, and a segment whose signature does not match is discarded before TCP processes it. So a password mismatch produces no NOTIFICATION in either direction — the session sits in Connect or Active looking exactly like a blocked port.

Three counters separate three different mistakes:

TcpExtTCPMD5Failure — a signed segment arrived and the signature did not match. Both sides have a key and the keys differ.
TcpExtTCPMD5NotFound — an unsigned segment arrived and a key is configured here. Only your side has authentication.
TcpExtTCPMD5Unexpected — a signed segment arrived and no key is configured here. Only their side has it.

Any of them incrementing while the peer sits in Active is the signature. All three static rules authentication out, and you go back to routing and filtering.

Restoring the key is a credential operation. Take the previous value from wherever secrets are held, not from a screenshot, and do not paste it into the incident record — a configuration containing a BGP password is a credential artefact, and so is a ticket comment quoting one.

Step 6: the flap is diagnosed by its cadence

A session that reaches Established and drops is a different problem from one that never gets there, and the state column will not tell you which cause you have. The interval will.

Cadence	Cause	Where to look
Seconds, regular	The physical interface is flapping	Interface state and error counters, the link partner
Sub-second	BFD, configured faster than the link or the CPU can sustain	BFD peer state, the configured interval
Matching the hold time	Packet loss on the path, or a peer too busy to send KEEPALIVE	Loss along the path, peer CPU
Matching a schedule	Your own automation clearing the session	The FRR log, the task scheduler, the config-sync job
Irregular, with prefix loss	A prefix limit being hit and released	Cease subcode 1 on the reset line

The trap in this table is the hold-time row. A session cycling every ninety seconds against a hundred-and-eighty-second hold time is not evidence that ninety is the right hold time. It is evidence that the peer stops hearing you about halfway through the interval, and shortening the timer makes the session fail faster rather than less.

Step 7: change one contract, with a net

Service impact possibleone hypothesis, one commit

configure
set protocols bgp neighbor 203.0.113.2 remote-as 65100

compare
commit-confirm 10
exit

One change, then read the state, then decide. A commit that alters the source, the TTL, a timer, the password and a policy at once will very likely produce a working session, and you will not know which of the five did it, which means you cannot fix the next router and cannot write the incident up.

When you finally need a reset, prefer the soft form:

Service impact possiblesoft first; hard is a convergence event

reset bgp ipv4 203.0.113.2 soft in
reset bgp ipv4 203.0.113.2 soft out

soft in re-requests the peer’s table without dropping the session. soft out re-advertises yours. Neither disturbs the FSM. The hard reset bgp ipv4 203.0.113.2 is for when the FSM itself is what you are testing, and it withdraws everything from that peer while it runs.

Step 8: verify stability, not liveness

Established at the moment you look is not the same as stable. Read the summary once, wait several minutes, read it again, and check that the Up/Down timer increased by the elapsed time rather than resetting.

Then check the prefix count against what the session is contracted to carry. A transit session showing eleven prefixes is up and useless, and it will page somebody else in an hour when the traffic it should be carrying is still going the wrong way.

Then check the routes reached the FIB, not only the BGP table — a route can be best and valid and still not installed — and that the thing the session exists for works from where its traffic starts.

Escalation

Escalate when:

Your SYN leaves and nothing comes back: the evidence has crossed the demarcation, and the peer NOC needs the state, the timer, the reset reason and the capture direction.
The session is one of a redundant pair, or the only transit.
The fault is authentication — that is a credential question before it is a routing one.
Interface errors or carrier transitions correlate with the flap.
The state changed at a commit timestamp. Revert, then diagnose.
Two hypotheses have failed to move the state.

Why this matters

Almost every BGP session incident is solved by the first two commands, and almost none of them start there.

The reason is worth naming, because it is not ignorance. Under pressure the attractive action is the one that produces a lot of output, and tcpdump on port 179 produces output immediately. show bgp summary produces three lines, one of which says Active, and it does not feel like progress. So the operator who knows perfectly well what Active means still opens a capture, and an hour later has a great deal of correct TCP behaviour and a peer whose remote-as is one digit wrong.

The second failure is worse because it is irreversible. Clearing a session is the most available thing in the incident: it is one command, it sometimes works, and it makes the graph move. What it also does is destroy the Last reset line, the message counters and the negotiated capability list — the exact evidence that would have named the fault — and, on a transit session, withdraw every prefix from the peer and re-exchange the table on the way back up. An operator can clear a session three times in ten minutes, learn nothing, and have caused three convergence events downstream while doing it.

The discipline that fixes both is small and unglamorous: read the state, read the reset reason, and treat “there is no reset reason” as a finding rather than as a dead end. That last one is the piece experienced operators use constantly and rarely explain. Silence from BGP is not missing evidence. It is BGP telling you it never got to run.