QoS troubleshooting — latency, jitter, packet loss, the diagnostic flow

QoS failures typically manifest as high latency, jitter, or packet loss in the traffic that is supposed to be prioritised. Diagnosing these requires a systematic flow from symptom to evidence to root cause. The diagnostic tools are tc (Linux traffic control statistics), tcpdump (packet-level analysis), and iperf3 (throughput verification).

This lesson is the diagnostic reference for QoS. It covers the common failure modes (latency, jitter, packet loss, misclassification), the diagnostic flow for each, and the operational commands that reveal the state.

The diagnostic flow

The systematic flow for any QoS failure:

1. Symptom — what the operator sees (voice latency, video jitter, slow bulk transfers).
2. Class state — tc -s class show on the affected interface. The class statistics reveal drops, backlog, and throughput per class.
3. Queue state — tc -s qdisc show. The qdisc state reveals per-queue latency (sojourn time) and drops.
4. Classification — verify the packets are in the right class. Compare capture DSCP with expected.
5. Bandwidth — verify the link is congested (or not). If not, the QoS may not be the issue.

# The minimal QoS diagnostic set

# 1. Class state
vyos@R1:~$ tc -s class show dev eth0
# Lists classes with stats (bytes, packets, drops, backlog)

# 2. Queue state
vyos@R1:~$ tc -s qdisc show dev eth0
# Lists qdiscs with stats (drops, overlimits, requeues)

# 3. Filter state
vyos@R1:~$ tc -s filter show dev eth0
# Lists filters with match counts

# 4. Captures on the affected interface
vyos@R1:~$ tcpdump -ni eth0 -c 100 -vv
# Reveals DSCP markings, packet sizes, jitter

# 5. Test with controlled traffic
iperf3 -c <server> -P 4 -t 60
# Tests sustained throughput

# 6. Test with VoIP-like traffic (small UDP packets)
# Use iperf3 UDP mode to simulate voice
iperf3 -c <server> -u -b 100K -l 200 -t 60
# 100 Kbps, 200-byte packets (typical voice)

Failure mode 1 — High latency

Symptom: voice traffic has high latency. mtr or ping shows high round-trip times under load.

Diagnostic:

# Test under load
iperf3 -c <server> -u -b 1M -l 200 -t 60 &
# Background UDP traffic at 1 Mbps to load the link

# Measure latency
ping -i 0.1 <peer>
# Ping with 100 ms interval; look for high RTT

# Inspect the classes
tc -s class show dev eth0
# Look for backlog in the voice class; if backlog is large, latency is high

Common causes:

1. Voice class rate too low. The voice class can’t keep up; packets backlog.
2. Voice class not getting its priority. The QoS policy is not honouring the marking.
3. Leaf scheduler not fq_codel. SFQ or FIFO doesn’t control latency.
4. Bufferbloat. The bulk class has a large queue; packets wait in the bulk queue behind the voice.

flowchart TD
  A["Symptom: high latency"] --> B{"Class backlog<br/>high?"}
  B -- "yes" --> C["Class rate too low<br/>or bufferbloat"]
  B -- "no" --> D{"Voice class getting<br/>priority?"}
  D -- "no" --> E["Fix classification<br/>or policy"]
  D -- "yes" --> F{"Leaf scheduler<br/>is fq_codel?"}
  F -- "no" --> G["Use fq_codel"]
  F -- "yes" --> H["Other issue<br/>(hardware, link)"]

Failure mode 2 — Jitter

Symptom: voice quality is poor; the latency varies significantly. mtr shows varying round-trip times.

Diagnostic:

# Continuous ping to measure jitter
ping -i 0.05 <peer> | awk '{print $7}' | uniq -c
# Ping every 50 ms; count distinct RTT values

# Inspect the queue sojourn time
# (Requires kernel with CoDel stats; sometimes visible via /proc/net/...)
# Or use ss or netstat with detailed socket stats

# Or use a probe (e.g., smoke ping, OWAMP) for accurate jitter measurement

Common causes:

1. Voice class not isolated. Bulk bursts affect voice; voice is in a class that competes with bulk.
2. Multiple flows in the voice class. Each TCP flow waits for the other; voice packets queue behind TCP retransmissions.
3. Bursty traffic from the LAN. The link is intermittent; bursts cause queue build-up.

The fix:

1. Verify the voice class is isolated from bulk.
2. Verify the leaf scheduler is fq_codel (or similar AQM).
3. Verify the QoS policy is enforcing the priority.

Failure mode 3 — Packet loss

Symptom: voice or video has audible artefacts (loss); iPerf3 shows retransmissions; TCP throughput is low.

Diagnostic:

# Inspect the class drops
tc -s class show dev eth0
# Look for "dropped" counters in classes

# Inspect the qdisc drops
tc -s qdisc show dev eth0
# Shows drops and overlimits

# Test with controlled traffic
iperf3 -c <server> -P 4 -t 60
# Multiple parallel streams; look for retransmissions

# Test with UDP (no retransmission)
iperf3 -c <server> -u -b 50M -l 1400 -t 60
# 50 Mbps UDP; look for lost packets

Common causes:

1. Class ceiling too low. The class can’t handle the burst; excess is dropped.
2. QoS not applied to the right interface. Traffic enters the WAN interface but the QoS policy is on the LAN.
3. Link capacity exceeded. The total traffic exceeds the link capacity.

The fix depends on the cause:

• If ceiling too low: increase it.
• If wrong interface: apply the QoS policy to the correct interface.
• If link overloaded: more bandwidth (or accept the loss).

Failure mode 4 — Misclassified traffic

Symptom: traffic that should be high priority is treated as bulk (and vice versa).

Diagnostic:

# Capture on the WAN; look at the DSCP markings
tcpdump -ni eth0 -c 20 -vv
# Check the DSCP field (tos byte)
# EF (46) for voice; AF41 (34) for video; BE (0) for default

# Inspect the filter matches
tc -s filter show dev eth0
# Lists filters with match counts; verify each filter matches the expected traffic

# Inspect the class state
tc -s class show dev eth0
# Look for traffic in classes it doesn't belong to

Common causes:

1. Marking rules wrong. The mangle rule marks the wrong traffic.
2. Filter match wrong. The filter matches the wrong traffic.
3. Re-marking at WAN. The WAN re-marking rules are wrong; external markings are still honoured.

The fix:

• Verify the marking rules match the intended traffic.
• Verify the filter rules match the intended class.
• Verify the WAN re-marking is in place.

Failure mode 5 — Class saturation

Symptom: a class is consistently full; queues are deep; drops are occurring.

Diagnostic:

# Inspect the class state
tc -s class show dev eth0
# Look at "backlog" — the bytes waiting in the queue
# High backlog indicates sustained saturation

# Test with sustained load
iperf3 -c <server> -P 8 -t 300
# 8 parallel streams for 5 minutes; check the class saturation

Common causes:

1. Class rate too low. The class is allocated less rate than its traffic demands.
2. Class ceiling too low. Even with spare capacity on the link, the class cannot use more.
3. Misclassification. Traffic that should be in another class is in this class, saturating it.

The fix:

• Increase the class rate (if link has spare capacity).
• Increase the ceiling (if rate increase is not possible).
• Re-check the classification (if misclassification is the cause).

A complete diagnostic flow

# Symptom: Voice has high latency / jitter / loss

# Step 1: Capture during the issue
tcpdump -ni eth0 -c 100 -w /tmp/cap.pcap &
# Run the capture during the issue

# Step 2: Inspect the class state
tc -s class show dev eth0
# Look at backlog (queue depth) and drops

# Step 3: Inspect the qdisc state
tc -s qdisc show dev eth0
# Look at overlimits (queue exceeded the limit) and drops

# Step 4: Inspect the filter state
tc -s filter show dev eth0
# Look at match counts; verify each filter matches

# Step 5: Inspect the marking
iptables -t mangle -L -v
# Verify the marking rules are hitting the expected traffic

# Step 6: Run iPerf3 UDP
iperf3 -c <server> -u -b 100K -l 200 -t 60 -i 1
# 100 Kbps UDP for 60 seconds; observe jitter and loss

# Step 7: Check link utilisation
iperf3 -c <server> -t 60 -i 1
# TCP throughput; observe link utilisation

# Step 8: Apply the fix and verify
# (Modify the QoS policy; commit; re-test)

Production failure modes

Ceiling too low

The voice class has ceiling 100mbit but voice traffic bursts to 200 Mbps. The class drops excess.

Diagnostic: tc -s class show dev eth0 shows drops in the voice class.

Fix: increase the ceiling.

Rate too low for sustained load

The voice class has rate 50mbit but voice traffic sustains at 80 Mbps. The class cannot keep up; backlog grows.

Diagnostic: backlog in the voice class is high; latency increases.

Fix: increase the rate (or reduce voice traffic / add bandwidth).

Leaf scheduler not fq_codel

The leaf scheduler is SFQ or FIFO. Bufferbloat causes high latency.

Fix: change to fq_codel.

Filters not matching

The filter does not match the intended traffic. The traffic falls through to the default class (bulk).

Fix: verify the filter matches the intended traffic.

Host marking bypasses trust boundary

A host marks its own DSCP to EF; the QoS policy honours the marking; legitimate voice is starved.

Fix: implement the trust boundary (re-mark on ingress).

Rollback

# Capture the diagnostic output
tc -s class show dev eth0 | save /tmp/vyos-qos-diag-$(date +%s).txt
iperf3 -c <server> -t 30 | save /tmp/vyos-qos-iperf-$(date +%s).txt

# Roll back to a previous configuration
configure
load /tmp/vyos-qos-backup-$(date +%s).conf
commit
save

The rollback restores the previous QoS policy.

Production discipline

Cross-course references

• Part LII-04 (LII-VyOS-Troubleshoot / subsystem by subsystem) covers the wider diagnostic methodology.
• Part XLIX-04 (XLIX-VyOS-Monitoring / VRRP telemetry) covers the monitoring integration.
• Part LI-06 (LI-VyOS-MTU / MTU validation) covers a related troubleshooting approach for tunnels.

Quiz