QoS Misconfiguration Starves Critical Traffic

Reported symptoms

edge-hq is the head-office VyOS router. eth0 is a 1 Gbps fibre handoff to the ISP; eth1 faces the office LAN, including the voice VLAN. Eight days ago a QoS policy went live on the WAN egress. The project had one stated goal: protect voice during the busy hour.

Since then the ticket queue has collected four complaints that nobody has connected to each other.

The phone system is worse than it was — choppy audio and dropouts during business hours, with jitter and loss on the media streams that the platform’s own metrics did not show in the month before the change. SSH sessions to remote sites freeze for seconds at a time, correlating with somebody moving a large file rather than with any time of day. The nightly backup to the DR site now runs for about nine hours where it used to finish inside one, and it is exactly as slow at 03:00, with an empty office, as it is at midday. And the QoS policy itself looks like it is doing nothing: the voice class counters are effectively flat while the phone platform reports its normal volume of calls.

Two things have already been tried. On day two an engineer raised the voice class ceiling from 200 Mbps to 400 Mbps, on the theory that the class was running out of room; it changed nothing at all. On day four the ISP was asked to check the circuit and reported it clean, which the router agrees with — the port is up at 1000 Mbps full duplex, the interface error and drop counters are zero, and a ping to the ISP gateway sits at a steady 3 ms.

So the working theory on the bridge is that there are three separate problems: a voice problem, a WAN capacity problem, and a QoS policy that is somehow not engaging. None of the three has moved in a week.

Evidence provided

vyos@edge-hq:~$ show configuration commands | match qos

set qos policy shaper WAN-OUT default bandwidth 100mbit
set qos policy shaper WAN-OUT default burst 15k
set qos policy shaper WAN-OUT default queue-type fq-codel
set qos policy shaper WAN-OUT class voice match dscp 46
set qos policy shaper WAN-OUT class voice priority 0
set qos policy shaper WAN-OUT class voice queue-type fq-codel
set qos policy shaper WAN-OUT class voice ceiling 400mbit
set qos policy shaper WAN-OUT class video match dscp 34
set qos policy shaper WAN-OUT class video priority 2
set qos policy shaper WAN-OUT class video queue-type fq-codel
set qos policy shaper WAN-OUT class bulk match all
set qos policy shaper WAN-OUT class bulk priority 7
set qos policy shaper WAN-OUT class bulk queue-type fq-codel
set interfaces ethernet eth0 traffic-policy out WAN-OUT

vyos@edge-hq:~$ show configuration commands | match mangle

set firewall ipv4 mangle rule 10 description 'Mark VoIP with EF'
set firewall ipv4 mangle rule 10 protocol udp
set firewall ipv4 mangle rule 10 destination port 5060
set firewall ipv4 mangle rule 10 set dscp 46
set firewall ipv4 mangle rule 20 description 'Mark video with AF41'
set firewall ipv4 mangle rule 20 protocol tcp
set firewall ipv4 mangle rule 20 destination port 1720
set firewall ipv4 mangle rule 20 set dscp 34
set firewall ipv4 mangle rule 30 description 'Mark bulk with BE'
set firewall ipv4 mangle rule 30 protocol tcp
set firewall ipv4 mangle rule 30 destination port 22
set firewall ipv4 mangle rule 30 set dscp 0

vyos@edge-hq:~$ tc -s class show dev eth0

class htb 1:1 root rate 100Mbit ceil 100Mbit burst 15Kb cburst 1600b
Sent 41122839104 bytes 31882144 pkt (dropped 0, overlimits 0 requeues 0)
rate 94208Kbit 8106pps backlog 0b 0p requeues 0
class htb 1:10 parent 1:1 leaf 110: prio 0 rate 100Mbit ceil 400Mbit burst 15Kb cburst 1600b
Sent 611904 bytes 2318 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
class htb 1:20 parent 1:1 leaf 120: prio 2 rate 100Mbit ceil 100Mbit burst 15Kb cburst 1600b
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
class htb 1:30 parent 1:1 leaf 130: prio 7 rate 100Mbit ceil 100Mbit burst 15Kb cburst 1600b
Sent 41122227200 bytes 31879826 pkt (dropped 118442, overlimits 0 requeues 0)
rate 94208Kbit 8106pps backlog 1874112b 1238p requeues 0

vyos@edge-hq:~$ tc -s qdisc show dev eth0

qdisc htb 1: root refcnt 2 r2q 10 default 0x30 direct_packets_stat 0
qdisc fq_codel 110: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn
qdisc fq_codel 120: parent 1:20 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn
qdisc fq_codel 130: parent 1:30 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn

vyos@edge-hq:~$ tcpdump -ni eth1 -c 6 -vv udp

11:42:01.004881 IP (tos 0xb8, ttl 64, id 9931, offset 0, flags [DF], proto UDP (17), length 902)
  192.0.2.51.5060 > 198.51.100.77.5060: SIP, length: 874
11:42:03.118427 IP (tos 0x0, ttl 64, id 34122, offset 0, flags [DF], proto UDP (17), length 200)
  192.0.2.51.16418 > 198.51.100.77.19236: UDP, length 172
11:42:03.138455 IP (tos 0x0, ttl 64, id 34123, offset 0, flags [DF], proto UDP (17), length 200)
  192.0.2.51.16418 > 198.51.100.77.19236: UDP, length 172
11:42:03.158402 IP (tos 0x0, ttl 64, id 34124, offset 0, flags [DF], proto UDP (17), length 200)
  192.0.2.51.16418 > 198.51.100.77.19236: UDP, length 172

vyos@edge-hq:~$ cat /sys/class/net/eth0/speed

1000

vyos@edge-hq:~$ iperf3 -c 198.51.100.90 -t 10

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   112 MBytes  94.1 Mbits/sec  241   sender
[  5]   0.00-10.01  sec   112 MBytes  94.0 Mbits/sec        receiver

Work the evidence before reading on

Four complaints are on the ticket. Before reading on, decide how many faults you think there are, and in which subsystem each one lives.

1. The backup is slow at 03:00 on an empty circuit. QoS decides who goes first when there is contention. What can a prioritisation problem not explain, and what does that eliminate in one step?
2. cat /sys/class/net/eth0/speed says 1000. The iperf3 run tops out at 94 Mbits/sec and does so repeatably. Where else in the evidence does a number close to 100 appear?
3. The voice class has sent about 600 kB in eight days. The phone platform says call volume is normal. Both statements can be true at once — how?
4. Look at the two tos values in the LAN capture. One is 0xb8 and one is 0x0. Work out which DSCP each represents, then work out which of the two flows is the audio.
5. Raising the voice ceiling from 200 to 400 Mbps changed nothing. What does a ceiling do to a class that has no packets in it?

Then the question that resolves the incident: the QoS project was meant to protect voice during congestion, and voice got worse. What did committing this policy do to the amount of congestion on this circuit?

Root cause

1. The shaper is rate-limiting a gigabit circuit to 100 Mbps

set qos policy shaper WAN-OUT default bandwidth 100mbit. The port negotiated 1000 Mbps. The policy came from the branch-office template, where 100 Mbps was the right number, and the one line that had to change for this site did not.

The consequence is larger than a slow backup, because HTB is a rate limiter, not a congestion detector. It does not measure whether the circuit is busy and then intervene; it shapes to the configured rate at all times. From the moment this was committed, edge-hq has had a 100 Mbps circuit — at 03:00 with an empty office exactly as much as at 15:00.

That is the whole explanation of the backup, and the iperf3 plateau at 94 Mbits/sec on an idle link is the same fact measured from the other end. The gap between 100 and 94 is unremarkable: framing overhead and the shaper’s own accounting.

It is also, and more importantly, the explanation for the congestion. Before the change, a gigabit circuit carrying an office’s worth of traffic was rarely contended, which is why voice was fine on plain FIFO and why nobody had ever needed QoS. The QoS deployment created the bottleneck it was meant to manage.

2. The voice class has never carried a voice packet

The mangle rule sets DSCP 46 on UDP with destination port 5060. Port 5060 is SIP: the signalling that sets up, modifies and tears down a call. It is a handful of packets per call, and in the capture it is the 902-byte datagram with tos 0xb8 — which is 46 shifted left by two bits into the DSCP position, so the marking rule is working exactly as written.

The audio is not on 5060. It is RTP, and RTP runs on a UDP port pair negotiated per call inside the SIP session description — in the capture, a stream of 200-byte datagrams from port 16418 to port 19236, every 20 ms, which is the shape of a single G.711 call. Its tos is 0x0. Nothing marked it, because no rule matches it.

So class voice match dscp 46 matches the signalling and nothing else. That is the 611904 bytes in class 1:10 over eight days: call setup and teardown, which is real traffic and is not what anyone meant by “voice”. Everything else falls through to class bulk match all — the audio, the SSH sessions (which the template explicitly marks BE), and the backup, all in one queue.

3. Why the two together are worse than either alone

The wrong bandwidth on its own would have produced one complaint: everything across the WAN is evenly slower than it should be, and someone would have found the 100mbit line inside a day, because one symptom pointed at one place.

The broken classification on its own would have produced no complaint at all. On an uncongested gigabit circuit, being in the bulk class costs nothing — there is no queue to be at the back of. The defect would have sat there until the first genuinely busy hour, months later.

Committed together, they compose. The bandwidth defect manufactures sustained congestion; the classification defect puts the traffic least able to tolerate it — 20 ms audio frames, no retransmission, a jitter budget in the tens of milliseconds — behind a backup stream that is by design always willing to consume every byte available. The backlog line in class 1:30 is 1.8 MB. At 94 Mbits/sec that is well over a hundred milliseconds of queuing delay, applied to packets that needed to arrive 20 ms apart.

That is the answer to the gating question. The policy did not fail to protect voice; it created the congestion and then put voice at the back of it.

4. What the two attempted fixes tell you

Raising the voice ceiling changed nothing because a ceiling limits what a class may consume, and the class was consuming almost nothing. A knob with no effect is a measurement of which class the traffic is really in; it was filed as a failed repair. The clean ISP report is the same kind of information. When every party outside this router checks out, the fault is in the thing between them.

Resolution

1. Decide first whether you are firefighting or repairing. During business hours with calls in progress, the goal is to stop the damage, not to deliver the project. Those are different changes with different risks and, usually, different people.
2. The firefighting action is to take the policy off the interface: delete interfaces ethernet eth0 traffic-policy out WAN-OUT, committed with commit-confirm 5. The circuit returns to its full rate and to the FIFO behaviour it had before the project. Backups, SSH and most of the call quality recover within seconds.
3. Be clear about what that costs. It removes voice protection during a genuine congestion event — but that protection has never worked here, so nothing real is given up. What it does cost is a conversation: the QoS project has been live for eight days and was, in that period, a 100 Mbps circuit.
4. If the shaper must stay on for policy reasons, the narrower change is set qos policy shaper WAN-OUT default bandwidth corrected to the circuit rate. That fixes the bottleneck without removing the classification defect, so voice is still in the bulk class — acceptable only because the class is no longer congested.
5. Establish the correct rate before you set it: from the ISP service description, confirmed by measurement at a quiet hour. Not from cat /sys/class/net/eth0/speed — the negotiated link speed and the rate the carrier delivers are different numbers, and a shaper set above the delivered rate pushes the queue into carrier equipment you can neither see nor schedule.
6. Fix the classification to mark the media rather than the signalling. Get the RTP port range from whoever owns the phone platform and mark that range, or — where the criteria are compound, such as the voice VLAN source subnet together with a UDP port range — set an fwmark in the mangle table and match the fwmark in the class, which is what fwmark is for.
7. If the phone platform owner is not reachable, hold this half. Do not guess a port range. A mangle rule that marks a range you have not confirmed will mark something else EF and hand you a second incident on top of this one. Holding is a decision, so it gets a named owner and a review time, not an implicit intention to look at it tomorrow.
8. Keep the SIP marking. Signalling is worth protecting; the defect was that signalling was the only thing being protected. Re-commit with commit-confirm and a short timer — you are changing the rate limiter on the office's only WAN circuit, from a session that traverses it.

Verification

1. The rate is what you intended. Read it back from the kernel with tc -s class show dev eth0 and check the HTB root rate and ceil, rather than trusting the configuration you just typed. The configuration was never in doubt; the number in it was.
2. The rate is reachable. Run a throughput test at a quiet hour and require it to reach the intended figure rather than the old 94 Mbits/sec plateau. Do this in addition to the previous check, not instead of it — a slow far end and a shaper look identical from a single throughput number.
3. The voice class carries voice. Place a real call and watch the byte counter for the voice class in tc -s class show dev eth0 climb while the call is up and stop when it ends. This is the check that has teeth, and it was available on day one.
4. Confirm the same thing at packet level. tcpdump -ni eth0 'ip[1] & 0xfc >> 2 == 46' during a call must capture the RTP stream, not only the SIP exchange: 200-byte datagrams every 20 ms are the audio, a burst of 900-byte datagrams at setup is not.
5. Verify under congestion, in a maintenance window. A QoS policy is inert on an idle link and therefore proves nothing there. Load the circuit with parallel TCP streams and, at the same time, run the voice traffic profile — small UDP packets at call rate — and require the latency and loss on that stream to stay acceptable.
6. Read the drops correctly during that test. Drops in the bulk class while voice stays clean is the shaper doing its job; drops in the voice class are a real finding.
7. Confirm the other two symptoms cleared, since they were the same fault: the backup completes in its historical window, and SSH no longer stalls behind a large transfer.
8. If you held the classification fix, the verification is a calendar entry rather than a command: the owner is named and the review time is set.

Prevention

• Make the circuit rate a per-site value that cannot be inherited from a template. It is the one number in a shaper that is certain to be wrong when the policy is copied, and copying the policy is how everyone builds the second site.
• Read the class counters after every QoS change, before you call it done:

tc -s class show dev eth0

A priority class that has not moved while the traffic it names is flowing is a classification failure, visible in one command, on the first day.

• Alert on it rather than relying on someone remembering. Sample the voice class byte counter and alert when it stays flat during business hours. That turns this incident from eight days into an afternoon.
• Mark the media, not the port whose name you recognise. Signalling and media are separate flows on separate ports, and only one of them is the call. The same trap exists for any protocol with a control channel and a data channel.
• Rehearse QoS under manufactured congestion before trusting it. Deployment windows are quiet by design, and a policy validated in a quiet window has been validated in the one condition where it does nothing.
• Record somewhere an operator will find at 03:00 that a shaper is unconditional. “Is the link congested?” and “did we congest the link?” are different questions, and this incident is the distance between them.