medium riskservice affecting~45 min

Runbook: Investigate Packet Loss

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The loss claim has a measurement behind it, not a screenshot. ping -c 100 or mtr -r -c 100 to the FINAL destination, with the loss percentage recorded. Red on an intermediate mtr hop with a clean final hop is not loss; it is a router declining to answer probes
· The affected flow is named as a five-tuple, not as "the network". Which source, which destination, which protocol, which port, entering this router on which interface and leaving on which
· Loss has been checked in both directions. A one-way measurement cannot tell a lost request from a lost reply, and the two have different owners
· The loss is proportional at every packet size. If it is a cliff - clean below a size, total above it - stop: this is an MTU fault and vyos-rb-investigate-mtu-issue is the runbook
· A baseline exists, or its absence is recorded. Interface and kernel drop counters are cumulative since boot, so a non-zero value proves nothing on its own. What matters is whether it is climbing right now
· The change history has been read rather than assumed: show log commit | last 10 and show configuration commit. Loss that started at a timestamp usually has a commit near it
· Whether the traffic is being dropped deliberately has been considered before anything is tuned. A default-deny firewall doing its job looks exactly like loss to the person whose packets it is dropping

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Measure loss end to end and write down the number: mtr -r -c 100 DEST and ping -c 100 DEST. Read the FINAL hop. Intermediate-hop loss with a clean final hop is ICMP rate limiting on that hop, not a fault.
2Rule out the MTU cliff before anything else: a sized do-not-fragment ping at a small size and at 1472. Proportional loss at both sizes is loss; clean at one and dead at the other is MTU.
3Establish direction. Measure from both ends where a host at each end exists, and confirm with a capture on this router that the request arrives and the reply leaves - monitor traffic interface ethN filter for the five-tuple.
4Ask whether the drops are policy or resource. Read show firewall ipv4 forward filter and show firewall statistics and look for a default-action or state-invalid counter climbing in step with the loss. A deliberate drop needs a policy conversation, not a tuning one.
5Take a two-sample reading of the interface counters, sixty seconds apart, and work on the DELTA: show interfaces ethernet ethN and ip -s link show ethN. Absolute counters since boot are not evidence.
6Separate link-layer errors from queue drops. RX errors, CRC errors, frame errors and carrier transitions are physical - cable, optic, switch port, duplex. They are not fixed by tuning and must be handed to whoever owns the physical layer.
7If the drops are queue drops, identify WHICH queue. Read /proc/net/softnet_stat for the per-CPU input queue, /sys/class/net/ethN/statistics/ for the driver view, and ethtool -S ethN for the driver-specific breakdown. The three answer different questions and disagree usefully.
8Correlate with CPU: mpstat -P ALL 1 5. A single core at high %soft while the others idle is interrupt distribution; every core saturated is capacity. The two have completely different fixes and only one of them is a configuration change.
9Match the fix to the counter that is actually climbing. Ring full raises the ring; per-CPU backlog full raises net.core.netdev_max_backlog; socket buffer full raises net.core.rmem_max and the per-protocol buffer. Raising all three because one is full is how a diagnosis becomes three untested changes.
10Apply one change at a time, with compare then commit-confirm 5, and confirm from the operational shell before running confirm. One change per measurement is the only way to attribute the improvement.
11Re-measure the same way you measured at the start: the same destination, the same count, and a fresh two-sample counter delta. The counter must stop CLIMBING; it will not go back to zero.
12Run save, then record the drop layer, the counter, the fix, the before and after loss percentage, and the new baseline. A tuned value with no recorded reason is the next operator reverting it.

4 · Verification

Confirm the procedure actually fixed the problem.

✓End-to-end loss measured the same way as at the start - same destination, same count, same tool - has fallen to the agreed threshold, and the measurement is in the incident record next to the original
✓The specific counter that was climbing is no longer climbing: two samples sixty seconds apart show no increase. An absolute value above zero is expected and is not a failure
✓No new counter started climbing in exchange. Raising a ring buffer or a backlog moves the pressure; re-read the whole set, not the one you fixed
✓mpstat -P ALL 1 5 shows the softirq load distributed as intended, if interrupt distribution was part of the fix
✓The application symptom that produced the page is gone, measured on the application - the call quality, the transfer, the session - and not only on ping
✓Any firewall rule identified as the real cause was resolved as a policy decision with a named owner, not silently widened
✓compare from configure produces no output, and configure, load /config/config.boot, compare, discard produces no diff: the change is saved, not only live
✓The new baseline is recorded - the counter values, the loss percentage, and the tuned settings - so the next incident starts from a number instead of a guess

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Sysctl changes: delete system sysctl parameter KEY, then commit and save. The kernel applies the default immediately; nothing needs to restart
↶Ring buffer: delete interfaces ethernet ethN ring-buffer rx, then commit. Resizing a ring re-initialises the NIC queue and briefly interrupts traffic on that interface - the rollback costs the same interruption the change did
↶Live ethtool changes made during triage do not survive a reboot and are not in the configuration tree. Write down every one you made, because the next operator reading show configuration will not see them
↶Run compare N BEFORE rollback N. The diff covers everything committed since that archive entry, which on a shared router is frequently more than your change
↶If the interface did not come back after a ring or offload change, recover from the console. Do not attempt a second remote change against an interface that just proved it can go away
↶POINT OF NO RETURN: packets already dropped are gone. Anything that failed during the incident - a call, a backup, a replication window - must be re-driven by its owner; tuning the router does not replay it
↶POINT OF NO RETURN: a larger ring buffer or a larger backlog trades latency for tolerance of bursts. Rolling the value back restores the latency profile but does not undo any decision taken downstream while the router was buffering more deeply

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to whoever owns the physical layer as soon as RX errors, CRC errors or carrier transitions are climbing. No sysctl fixes a bad optic, and time spent tuning around one is time the fault stays in place
· Escalate to network engineering when the loss is on a router that is one half of a redundant pair. Tuning one half of a pair changes which member is the better path, which is a failover decision rather than a performance one
· Escalate to capacity planning when every core shows sustained high softirq. That is the router being smaller than the traffic, and it is a procurement conversation; software tuning at that point buys single-digit percentages
· Escalate to the security or firewall owner when the drops turn out to be a deliberate policy rule. Widening a rule to stop a page is a change of intent and belongs to the person who wrote the intent
· Escalate to the upstream provider when loss is present at the final hop from more than one internal source and this router shows clean counters in both directions. Give them the five-tuple, the loss percentage and the measurement window, not a screenshot
· Escalate rather than keep tuning after two changes have failed to move the number. Two failed attempts usually means the layer was misidentified, and a third guess makes the evidence worse

Most packet-loss incidents open with a screenshot of mtr that has red on hop three. Almost none of them are what the screenshot appears to say, because a router that declines to answer a probe while forwarding every packet perfectly produces exactly that picture, and it produces it on a healthy network every day.

So the first job is not to find the loss. It is to establish that there is any, and then to find out which of five completely different things is causing it. A VyOS router can drop a packet at the physical layer, at the NIC ring, at the kernel’s per-CPU input queue, at a socket receive buffer, or because a firewall rule told it to. Each one has its own counter, and each one has a fix that does nothing at all for the other four.

The expensive version of this incident is the one where somebody raises the ring buffer, the backlog and the socket buffer in a single commit, the number improves for unrelated reasons, and the router now carries three permanent settings that nobody can justify.

When to use this runbook

Traffic across a VyOS router is being lost or degraded proportionally — a percentage of packets, spread across sizes — and the loss is measurable at the destination rather than only at an intermediate hop.

When not to use it

Loss is a cliff rather than a slope: everything below some size is clean and everything above it fails. That is vyos-rb-investigate-mtu-issue.
The path is completely down. This runbook assumes packets are mostly getting through; if none are, start at vyos-rb-troubleshoot-missing-route.
The router itself is slow rather than lossy — the CLI is sluggish, the routing daemons are behind. Start at vyos-rb-investigate-router-cpu and come back if the evidence points at the data path.

Blast radius

Every diagnostic step here is read-only, with one important exception: a packet capture on a busy interface is not free. Capturing on a saturated data path adds work to exactly the softirq pass that is already struggling, and it can deepen the loss you are measuring. Capture with a tight filter and a packet count, or capture on a switch SPAN port instead.

The remediation is not read-only either. Resizing a ring buffer re-initialises the NIC queue and briefly interrupts the interface. Do it with a timer and, where the interface carries your session, from the console.

Step 1: prove the loss is real and end-to-end

Read-only / Safethe last hop is the only hop that counts

# Substitute your own value before running:
DEST=192.0.2.50

mtr -r -c 100 "$DEST"
ping -c 100 "$DEST"

Start: 2026-08-18T09:14:02+01:00
HOST: r1                          Loss%  Snt  Last  Avg  Best  Wrst StDev
1. 198.51.100.1                   0.0%  100   0.4   0.4   0.3   0.6   0.1
2. 198.51.100.9                  38.0%  100   1.2   1.4   1.0   4.1   0.4
3. 192.0.2.50                     0.0%  100  12.4  12.6  12.0  15.1   0.7

Illustrative output

That output shows no loss. Hop 2 is losing 38% of the probes addressed to hop 2, and forwarding 100% of the traffic that passes through it. Routers generate ICMP responses on the control plane, deprioritise that work, and rate limit it; a busy transit router answering fewer than half the probes aimed at it is behaving normally.

The rule is simple and it saves hours: read the final hop. Loss that does not appear at the destination did not happen to your traffic. Loss that appears at a hop and at every hop after it is real, and the first hop where it appears is where to look.

Step 2: establish direction

A single measurement from one end cannot tell you whether the request was lost or the reply was. Those have different owners and different fixes, and assuming the wrong one sends the investigation to the wrong team.

Read-only / Safewatch the flow arrive and leave

monitor traffic interface eth0 filter 'host 192.0.2.50 and icmp'
monitor traffic interface eth1 filter 'host 192.0.2.50 and icmp'

If the request appears on the ingress interface and does not appear on the egress interface, this router is where the packet stopped, and the rest of the runbook applies to this router. If it leaves cleanly and the reply never returns, the loss is downstream or on the return path, and Part LII’s return-path discipline applies: measure again from the far end before doing anything here.

Step 3: policy drops or resource drops?

Ask this before touching a tuning knob, because the answer changes who owns the incident.

Read-only / Safecounters, not intentions

show firewall ipv4 forward filter
show firewall ipv4 input filter
show firewall statistics

A default-action counter climbing in step with the reported loss means the traffic is being dropped on purpose by a rule set that is doing its job. So does a state invalid drop counter, which typically means the router is seeing one direction of a flow whose other direction went somewhere else — asymmetric routing, not resource exhaustion.

That is a policy conversation with a named owner. Widening a rule to make a page go away, without that conversation, is how a firewall stops being a statement of intent.

Step 4: read the counters as a delta, never as a total

Read-only / Safetwo samples, sixty seconds apart

show interfaces ethernet eth0
ip -s link show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
  RX: bytes  packets  errors  dropped  missed  mcast
  9812734511 18342991  0       21884    0      4412
  TX: bytes  packets  errors  dropped  carrier collisions
  8123998210 17993412  0       0        0      0

Illustrative output

Every counter here is cumulative since the interface came up. A router with 400 days of uptime and 21,884 dropped frames has dropped one frame in every 838,000 — which is nothing, and which looks alarming to somebody reading the number for the first time at three in the morning.

Sample it, wait sixty seconds, sample it again, and work on the difference. A counter that is not moving is not your incident, however large it is. A counter moving at a thousand a second is, however small the total.

Read the whole row, because the columns mean different things

errors, dropped, missed and carrier sit next to each other and point at completely different subsystems.

RX errors, CRC errors, frame errors — the frame arrived damaged. Cable, optic, connector, duplex mismatch, or a failing switch port. Physical.
carrier — the link itself flapped. Also physical, and worth correlating against the interface’s up/down history before anything else.
dropped — the frame arrived intact and the kernel had nowhere to put it. A queue problem, and the one this runbook can actually tune.
missed — the NIC could not hand the frame to the kernel at all, usually because the receive ring was full. Hardware-side queue pressure.
collisions — should be zero on any full-duplex link. Non-zero means a duplex mismatch, which is a physical-layer fix and not a tuning one.

Errors and drops climbing together is a different incident from drops climbing alone, and no amount of sysctl helps the first one.

Step 5: if it is a queue drop, find out which queue

Three buffers sit between the wire and a consumer, and each has its own counter. Reading all three takes two minutes and is the difference between one correct change and three speculative ones.

Read-only / Safethe per-CPU input queue

awk '{ printf "CPU %d: processed=%-12d dropped=%-10d squeezed=%d
", NR-1, strtonum("0x" $1), strtonum("0x" $2), strtonum("0x" $3) }' /proc/net/softnet_stat

CPU 0: processed=418223941   dropped=1204431   squeezed=88213
CPU 1: processed=402118773   dropped=0         squeezed=0
CPU 2: processed=399884102   dropped=0         squeezed=0
CPU 3: processed=401220518   dropped=0         squeezed=0

Illustrative output

The columns are cumulative hex, one row per CPU: packets processed, packets dropped because the per-CPU input queue was full, and times the softirq pass ran out of budget before draining it. The shape above — one core dropping, three idle — is the signature of a NIC whose interrupts all land on a single core, and the fix is interrupt distribution rather than a bigger queue.

Read-only / Safethe driver's own view

# Substitute your own value before running:
IFACE=eth0

cat "/sys/class/net/$IFACE/statistics/rx_dropped"
cat "/sys/class/net/$IFACE/statistics/rx_missed_errors"
cat "/sys/class/net/$IFACE/statistics/tx_dropped"

sudo ethtool -S "$IFACE"
sudo ethtool -g "$IFACE"

Counter climbing	Where the packet died	The fix	What it costs
`rx_missed_errors`, driver ring counters	NIC receive ring was full	Raise the ring with `ethtool -G`, persist in the configuration	Deeper queueing means a longer worst-case latency; a ring resize interrupts the interface
`dropped` in `softnet_stat`, one core only	Per-CPU input queue on the core the NIC interrupts	Distribute interrupts across cores (RSS)	A real change to how the NIC is driven; verify with `/proc/interrupts` afterwards
`dropped` in `softnet_stat`, every core	Per-CPU input queues, all of them	Raise `net.core.netdev_max_backlog` for bursts — and accept that sustained loss here is capacity	Absorbs bursts; buys nothing against sustained overload
Socket-level drops, `Recv-Q` non-zero in `ss -m`	A consumer is not draining its socket	Raise `net.core.rmem_max` and the per-protocol buffer, or fix the consumer	The ceiling rises; the actual buffer only grows if the application or protocol asks for it
RX errors, CRC, frame, carrier	Physical layer	Not a tuning problem	Hand it to whoever owns the cable

Step 6: correlate with the CPU before you conclude

Read-only / Safewho is actually busy

mpstat -P ALL 1 5
top -bn1 | head -20
grep eth0 /proc/interrupts

Read the %soft column. One core high while the others idle is a distribution problem: the NIC’s interrupts are landing on a single core and that core’s softirq pass cannot keep up. Every core high is a capacity problem, and no configuration change on this router fixes it — that is a hardware or a traffic conversation, and saying so early is more useful than a week of tuning.

Drops and CPU utilisation also decouple, which surprises people. A router at 30% CPU can be dropping steadily, because the softirq pass on the specific core that matters is being starved by something else with the CPU — a routing daemon working through an update burst, an SNMP walk, a capture you started during triage. Spare capacity on the box as a whole is not evidence that the packet path has any.

Step 7: change one thing, with a net

Service impact possibleone knob, one timer, one measurement

set system sysctl parameter net.core.netdev_max_backlog value 10000

compare
commit-confirm 5
exit

For a ring-buffer change, test the value live with ethtool -G first — it takes effect immediately and does not survive a reboot, which makes it a genuinely cheap experiment — and only then persist it:

Service impact possibleresizing a ring interrupts the interface

set interfaces ethernet eth0 ring-buffer rx 2048

compare
commit-confirm 5
exit

Leave configure mode, re-measure from the operational shell while the timer runs, and only then confirm and save. If the interface does not come back, do nothing: the timer restores the previous state without a second decision from an operator who cannot currently see the box.

Step 8: verify on the delta, not on the total

Re-measure exactly as you measured at the start — same destination, same count, same tool — and take a fresh two-sample counter reading. The counter will not return to zero; it is cumulative. What must change is that it stops climbing.

Then check that nothing else started climbing in exchange. Raising a ring buffer moves pressure into the per-CPU queue behind it; raising the backlog moves it into the socket buffers. Re-read the whole set, not the one you fixed.

Finally, record the new baseline: the counter values at rest, the loss percentage, the tuned settings and why. A tuned value with no recorded reason is a value the next operator reverts.

Common patterns

Symptom	Likely cause	Where to look
Red on a middle mtr hop, clean final hop	ICMP rate limiting on that hop	Final-hop loss only; ignore the middle
Loss clean at small sizes, total at large	MTU, not loss	Sized do-not-fragment ping
One core high `%soft`, others idle	NIC interrupts on a single core	`softnet_stat` per-CPU rows, `/proc/interrupts`
Every core high `%soft`	Capacity, not configuration	Escalate to capacity planning
`rx_missed_errors` climbing, `softnet_stat` clean	NIC receive ring too small	`ethtool -g`, then `ethtool -G`
RX errors and drops climbing together	Physical layer	Cable, optic, switch port, duplex
Drops appeared with a commit timestamp	The change, not the hardware	`show log commit`, `compare N`
Loss only on flows through one interface	That interface, not the router	Per-interface delta, not the aggregate
`state invalid` counter climbing	Asymmetric routing, not resource loss	Return path, and the firewall’s view of it
Loss got worse when you started capturing	The capture is competing with the packet path	Tight filter and a packet count, or a SPAN port

Escalation

Escalate when:

Errors, CRC or carrier counters are climbing.
The router is one half of a redundant pair.
Every core shows sustained high softirq.
The drops turn out to be a deliberate firewall rule.
The final hop loses packets from multiple internal sources while this router’s counters are clean in both directions.
Two changes have failed to move the number.

Why this matters

Packet loss is the incident class where the evidence most often points at the wrong place, and it does so for a structural reason: the tools an operator reaches for first measure the control plane, and the problem is in the data plane.

mtr asks routers to talk about themselves. Routers answer that question with whatever attention they have left over, which on a busy transit hop is not much. ping measures a 56-byte packet, which is the one packet size almost nothing can go wrong with. Interface counters report a total since boot on a box that may have been up for a year. Three tools, three different ways to be confidently wrong, and all three are the first thing anyone runs.

The discipline that fixes it is unglamorous. Read the last hop. Take two samples and subtract. Change one thing. None of that requires knowing anything about the router, and all of it is faster than the alternative, which is a tuning session that ends with three sysctls nobody can justify and a fault that was in the optic the whole time.

And name the drop layer out loud before touching a knob. The five layers are genuinely different systems, and the reason experienced operators fix these quickly is not that they know more values — it is that they spend the first five minutes deciding which of the five they are in, and then only own that one.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When to use this runbook

When not to use it

Blast radius

Step 1: prove the loss is real and end-to-end

Step 2: establish direction

Step 3: policy drops or resource drops?

Step 4: read the counters as a delta, never as a total

Step 5: if it is a queue drop, find out which queue

Step 6: correlate with the CPU before you conclude

Step 7: change one thing, with a net

Step 8: verify on the delta, not on the total

Common patterns

Escalation

References