Lab: QoS Classification and Shaping

Objective

QoS is the one subject where a configuration review tells you almost nothing. Every class you write parses, every match commits, and a policy that classifies nothing at all looks identical on the page to one that works. The only evidence is counters and measurements, and this lab is built entirely out of them: you will take four throughput-and-latency measurements, and every configuration change you make will be judged by what it did to a number rather than by how sensible it reads.

You will also do the thing QoS material usually skips. A shaper that honours DSCP is a shaper that believes whatever the hosts behind it write into the IP header — so you will let a host claim Expedited Forwarding for a flood of 1,400-byte packets, watch the router believe it, and then install the trust boundary that takes the claim back. That is the difference between a QoS deployment and a QoS deployment somebody can abuse from a laptop.

Architecture

Three VyOS routers. edge carries the shaper on its WAN egress; client is a host behind it; server is the far end and also the target for both traffic generators.

   +----------------+                 +----------------+                 +----------------+
   |     client     |                 |      edge      |                 |     server     |
   |  eth1 .5       |-----------------| eth1 .1        |                 |                |
   |                |   LAN           | eth2 203.0.113.1|----------------| eth1 203.0.113.2|
   |                | 192.168.10.0/24 |                 |  WAN           |                |
   +----------------+                 +----------------+ 203.0.113.0/30  +----------------+
                                             ^
                                             |
                                    shaper attached here,
                                    egress on eth2

There is deliberately no NAT in this lab. Every address is visible end to end, so a capture taken on either side shows the same addresses and the same TOS byte, and nothing you observe has to be corrected for a translation.

eth0 on each router is a management interface on whatever bridge your hypervisor already uses. No task configures it and no route points at it.

Requirements

• A hypervisor with roughly 3 GiB of free RAM and 24 GiB of free disk — three VMs at 1 GiB and 8 GiB each.
• The VyOS 1.5 LTS ISO. The QoS tree changed shape between 1.3 and 1.4; none of this commits on 1.3.
• Two isolated layer-2 segments — bridges with no physical port.
• Console access to all three routers. No task in this lab cuts management, but Task 2 makes the WAN twenty times slower and you will want a second way in if you mistype the interface name.
• Roughly 100 minutes, of which about half is measurement.
• Two consoles open on client throughout: one runs a generator, the other measures. Almost every task needs both at once.

Scenario

A branch office has a 20 Mbit/s Internet circuit and a complaint that has been open for months: “the line goes to treacle whenever anyone uploads anything.” Voice calls break up, SSH sessions type in bursts, and the monitoring graph shows the circuit at 100% utilisation and reports it as healthy, because from the graph’s point of view it is.

Nobody has measured anything. Your job is to measure it, fix the part that is fixable, and be able to say what the fix costs — because QoS does not create bandwidth, and every guarantee you hand one class is taken from another.

Tasks

Configuration blocks are written for the [edit] prompt. Where a block opens with configure and you are already at [edit], skip that line. Blocks tagged bash are the operational-mode shell, as their comments say.

Task 1 — Build the baseline and measure the link you have

Install three routers, give each a hostname and a management address on eth0, and capture the starting state before any lab set.

# Run from operational mode on each of the three routers.
JOURNAL="$HOME/lab20"
mkdir -p "$JOURNAL"
show configuration commands > "$JOURNAL/pre-lab-config.txt"
ip -brief address show > "$JOURNAL/pre-lab-addresses.txt"

edge:

configure
set system host-name edge
set interfaces ethernet eth1 address 192.168.10.1/24
set interfaces ethernet eth1 description 'LAN - trust boundary lives here'
set interfaces ethernet eth2 address 203.0.113.1/30
set interfaces ethernet eth2 description 'WAN - shaper attaches here'
commit
save

client:

configure
set system host-name client
set interfaces ethernet eth1 address 192.168.10.5/24
set protocols static route 0.0.0.0/0 next-hop 192.168.10.1
commit
save

server:

configure
set system host-name server
set interfaces ethernet eth1 address 203.0.113.2/30
set protocols static route 192.168.10.0/24 next-hop 203.0.113.1
set service ssh port 22
commit
save

Now take the baseline. Two measurements, taken at the same time, because a throughput number without a concurrent latency number is exactly the number the branch’s monitoring graph has been reporting for months.

On server, start the bandwidth-test listener:

monitor bandwidth-test accept

On client, console one — the load:

monitor bandwidth-test initiate 203.0.113.2

On client, console two — the measurement, running while the load runs:

ping 203.0.113.2 count 30

$ ping 203.0.113.2 count 30

--- 203.0.113.2 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29041ms
rtt min/avg/max/mdev = 0.402/0.688/1.913/0.221 ms

Write both numbers into the journal as measurement 1: throughput, and min/avg/max/mdev latency under that load. On virtual NICs the throughput will be some large number and the latency will be small, because the link you are measuring is a bridge in RAM. That is not a problem — it is the reason the next task exists.

Task 2 — Make the link 20 Mbit, and meet bufferbloat

A shaper is how you turn a fast interface into the circuit you actually bought. Build one with nothing but a default class, and deliberately give it the queue type that is the problem rather than the solution.

configure
set qos policy shaper WAN-OUT bandwidth 20mbit
set qos policy shaper WAN-OUT default bandwidth '100%'
set qos policy shaper WAN-OUT default ceiling '100%'
set qos policy shaper WAN-OUT default priority 7
set qos policy shaper WAN-OUT default queue-type drop-tail
set qos interface eth2 egress WAN-OUT
compare
commit
save

Two things about that block are the whole shape of VyOS QoS. set qos policy shaper defines a policy that is attached to nothing; set qos interface eth2 egress WAN-OUT is what puts it on an interface, and it is a separate tree. A policy defined and never attached commits cleanly, appears in the configuration, and does nothing whatsoever — which is the QoS equivalent of the health check with no rule.

Verify from the kernel, not from the configuration. VyOS 1.5 has no show qos operational command — the 1.3-era show queueing and show interface ethernet ethX queue are gone — so the evidence is tc:

# On edge, operational mode.
sudo tc qdisc show dev eth2
sudo tc class show dev eth2

$ sudo tc class show dev eth2

class htb 1:1 root rate 20Mbit ceil 20Mbit burst 1600b cburst 1600b
class htb 1:2 parent 1:1 prio 7 rate 20Mbit ceil 20Mbit burst 1600b cburst 1600b

If tc class show dev eth2 prints nothing, the policy is not attached. Go back and read show configuration commands | match 'qos interface' before changing anything else — an unattached policy is the single commonest reason a QoS change “did nothing”.

Now repeat Task 1’s paired measurement, unchanged. Record it as measurement 2.

$ ping 203.0.113.2 count 30

--- 203.0.113.2 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29088ms
rtt min/avg/max/mdev = 0.611/94.284/181.507/48.902 ms

Throughput has dropped to roughly 20 Mbit, which is what you asked for. Latency under load has gone up by two orders of magnitude, which you did not ask for and which nothing in the configuration mentions.

Task 3 — Change the leaf, not the rate

One node. Nothing else in the policy changes.

configure
set qos policy shaper WAN-OUT default queue-type fq-codel
compare
commit
save

Re-run the paired measurement. Record it as measurement 3.

The throughput should be within noise of measurement 2 — you did not change the rate, and fq-codel does not create bandwidth any more than the shaper did. The loaded latency should fall dramatically, because fq-codel does two things a FIFO does not: it gives each flow its own queue and serves them fairly, so a probe is not stuck behind a bulk transfer’s backlog; and it applies CoDel, which drops from a queue that has been standing too long, which is the signal TCP needed in order to stop filling it.

Write into the journal, in one sentence, what changed between measurement 2 and measurement 3 and what did not. That sentence is the one you will use the next time somebody proposes buying bandwidth to fix a latency problem.

Task 4 — Add classes, and prove classification from counters

Now the requirement that justifies classes: the branch’s latency-sensitive traffic must keep its own queue and its own guarantee, and bulk uploads must not be allowed to fill the circuit.

configure
set qos policy shaper WAN-OUT class 10 description 'latency-sensitive, marked EF'
set qos policy shaper WAN-OUT class 10 match EF-MARKED ip dscp EF
set qos policy shaper WAN-OUT class 10 bandwidth '20%'
set qos policy shaper WAN-OUT class 10 ceiling '30%'
set qos policy shaper WAN-OUT class 10 priority 1
set qos policy shaper WAN-OUT class 10 queue-type fq-codel

set qos policy shaper WAN-OUT class 20 description 'bulk transfer, capped'
set qos policy shaper WAN-OUT class 20 match BULK ip protocol tcp
set qos policy shaper WAN-OUT class 20 match BULK ip destination port 5001
set qos policy shaper WAN-OUT class 20 bandwidth '50%'
set qos policy shaper WAN-OUT class 20 ceiling '60%'
set qos policy shaper WAN-OUT class 20 priority 7
set qos policy shaper WAN-OUT class 20 queue-type fq-codel

set qos policy shaper WAN-OUT default bandwidth '30%'
compare
commit
save

Read the shapes in that block, because three of them are where people go wrong:

• Classes are numbers. class 10, not class voice. The human-readable name goes in description, and the filter gets a name — EF-MARKED, BULK — which is what match <name> is.
• A match is a named filter with criteria hanging off it. Two match BULK lines with different criteria are one filter matching TCP and port 5001. Two differently-named filters on the same class would be two filters, matching either.
• bandwidth and ceiling are different promises. bandwidth is the guarantee: this class gets at least this much when it wants it. ceiling is the cap: it may borrow idle capacity up to here and no further. The three guarantees must not sum above 100%, or the shaper is promising capacity that does not exist.

Now prove the classification, from counters rather than from the rule set. On client, send twenty EF-marked probes:

ping 203.0.113.2 tos 184 count 20 size 1400

tos sets the whole 8-bit TOS byte, and DSCP occupies its top six bits: DSCP 46 shifted left by two is 184, which is 0xb8. That is why the number in a ping command and the number in an RFC never match, and why a capture prints tos 0xb8 when the class you are matching is called EF.

On edge, read which class moved:

$ sudo tc -s class show dev eth2

class htb 1:1 root rate 20Mbit ceil 20Mbit
Sent 41822 bytes 43 pkt (dropped 0, overlimits 0 requeues 0)

class htb 1:10 parent 1:1 prio 1 rate 4Mbit ceil 6Mbit
Sent 28840 bytes 20 pkt (dropped 0, overlimits 0 requeues 0)

class htb 1:20 parent 1:1 prio 7 rate 10Mbit ceil 12Mbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)

class htb 1:30 parent 1:1 prio 7 rate 6Mbit ceil 20Mbit
Sent 12982 bytes 23 pkt (dropped 0, overlimits 0 requeues 0)

Twenty packets in the class built from class 10, none in the one built from class 20, and the rest — your SSH session, ARP, everything else — in the default class. VyOS assigns the kernel class handles, so read the rate and ceil on each line to work out which of your classes you are looking at rather than assuming the minor number matches the number you typed.

Then run the same probe without tos 184 and confirm the twenty packets land in the default class instead. Two runs, two counter readings: that is a classification proof. A configuration review is not.

Now measure the ceiling doing its job. Re-run the paired measurement from Task 1, and record it as measurement 4.

The bandwidth test targets TCP port 5001, so it is now class 20, whose ceiling is 60% of 20 Mbit. Expect roughly 12 Mbit rather than 20 — the link is idle and the class is still capped, because a ceiling is a cap and not a fallback.

Task 5 — Let a host mark itself, and watch the router believe it

The policy you have built matches ip dscp EF and asks no questions about who set it. That is the default posture of every DSCP-aware device, and it is a security decision that nobody made on purpose.

On client, generate sustained load that claims Expedited Forwarding for itself:

# On the client, operational mode. sudo is required for a sub-200ms interval.
# 1400 bytes every 2 ms is roughly 5.6 Mbit/s, all of it claiming EF.
sudo ping -Q 184 -i 0.002 -s 1400 -c 5000 203.0.113.2

While that runs, on edge:

sudo tc -s class show dev eth2

Class 1:10 is now carrying megabytes. The class you built for the site’s latency-sensitive traffic is being filled by a workstation, at priority 1, with a guarantee the operator granted and a host claimed. Start the bandwidth test at the same time and record what class 20 achieves now compared with measurement 4.

Confirm what is on the wire rather than inferring it from counters:

# On edge, operational mode.
sudo tcpdump -ni eth2 -v -c 5 icmp

$ sudo tcpdump -ni eth2 -v -c 5 icmp

IP (tos 0xb8, ttl 63, id 4711, offset 0, flags [DF], proto ICMP (1), length 1428)
  192.168.10.5 > 203.0.113.2: ICMP echo request, id 21, seq 1, length 1408

tos 0xb8 is DSCP 46. The router received that value from the LAN, matched on it, gave the packet a priority queue, and forwarded the marking onward to the provider — where, if the provider honours DSCP at all, it will be believed a second time.

Task 6 — Install the trust boundary

The fix is not in the shaper. The shaper’s job is to honour a marking; deciding which markings are true is a different job, and it belongs at the point where traffic enters your control — the LAN interface.

On VyOS 1.5 that is a policy route rule set applied to the ingress interface, with set dscp as its action. There is no mangle node under firewall to do this with.

configure
set policy route MARK-LAN description 'QoS trust boundary - this site marks, hosts do not'

set policy route MARK-LAN rule 10 description 'the only class this site marks EF'
set policy route MARK-LAN rule 10 protocol udp
set policy route MARK-LAN rule 10 destination port 5060
set policy route MARK-LAN rule 10 set dscp 46

set policy route MARK-LAN rule 20 description 'everything else leaves as best effort, whatever the host claimed'
set policy route MARK-LAN rule 20 set dscp 0

set policy route MARK-LAN interface eth1
compare
commit-confirm 5

Three things decide whether this works, and all three have bitten people:

• The binding is on the interface traffic arrives on. eth1, the LAN. A rule set bound to eth2 never sees the outbound traffic you were trying to re-mark, commits cleanly, and does nothing.
• Rules are evaluated in ascending order and the first match wins. Rule 20 carries no match criteria, so it matches everything — which is exactly what a catch-all needs to be, and exactly why it has to be numbered last. Numbered 5, it would re-mark the site’s own EF traffic to best effort and the rest of the policy would be dead weight.
• set dscp is a rewrite, not a verdict. It does not accept or drop the packet; the packet continues through the normal forwarding path with a different DSCP value.

Confirm and save, then re-run the flood from Task 5 unchanged, and read the same two pieces of evidence:

# On edge, operational mode. The claim should no longer survive the LAN.
sudo tcpdump -ni eth2 -v -c 5 icmp
sudo tc -s class show dev eth2

$ sudo tcpdump -ni eth2 -v -c 5 icmp

IP (tos 0x0, ttl 63, id 4711, offset 0, flags [DF], proto ICMP (1), length 1428)
  192.168.10.5 > 203.0.113.2: ICMP echo request, id 22, seq 1, length 1408

Class 1:10’s packet counter should now be static while the flood runs, and the default class’s counter should be climbing. That pair — the TOS byte on the wire and the class counter that stopped moving — is the proof. Either one alone is suggestive; both together are conclusive.

Task 7 — Read the whole picture back

Rebuild the measurement table from the four runs and put it in one place.

Then write the one-page class plan that is this lab’s real deliverable. For each class: its number, its match, its bandwidth, its ceiling, and one sentence justifying each number. A class plan without those sentences is a set of percentages somebody chose because they added up, and the first time the circuit is upgraded nobody will know which of them should change.

Close with the question the whole lab has been building to: of the four measurements you took, which ones would have looked identical if the shaper had been defined and never attached? Write the answer down. It is the reason sudo tc class show appears in every task on this page.

Validation

• Four throughput-and-latency pairs exist in the journal, each taken with the load and the probe running at the same time.
• sudo tc class show dev eth2 on edge lists an htb root at 20 Mbit and three child classes with rates matching the percentages you configured.
• The latency in measurement 3 is substantially lower than in measurement 2, and the throughput is within noise of it.
• Two sudo tc -s class show dev eth2 captures exist for Task 4’s probe — one with tos 184 and one without — and a different class counter moved in each.
• Measurement 4’s bandwidth-test throughput is close to 60% of the shaper rate, and your journal names ceiling on class 20 as the reason.
• Two sudo tcpdump -ni eth2 -v captures of the same flood exist, one showing tos 0xb8 and one showing tos 0x0, and the class counters agree with them.
• The written class plan carries one justifying sentence per number.

Expected Outcome

A VyOS router shaping its WAN egress to the circuit’s real rate, with an fq-codel leaf that keeps loaded latency low, three classes whose guarantees sum to the link, and a trust boundary on the LAN interface that decides what a marking means before the shaper is asked to honour it.

More usefully: a set of measurements that let you say what each piece of that bought you. Most of the latency improvement came from one node — the leaf scheduler — and the classes bought a cap and a guarantee that you can now state in numbers rather than adjectives.

Troubleshooting

sudo tc class show dev eth2 prints nothing. The policy is defined and not attached. show configuration commands | match 'qos interface' settles it in one line. This is the QoS failure that looks most like “QoS does not work on this platform”.

A class counter never moves. The match is not matching. Check the shape before the content: a filter is match <filter-name> ip <criterion> <value>, and criteria on the same filter name are ANDed. ip dscp EF and ip dscp 46 are the same thing; dscp 46 with no ip is not a valid leaf.

The commit is rejected on the class number. Classes are numeric on 1.5. A name where a number belongs is the single most common paste-from-Part-XLV error.

Throughput is far below the shaper rate with no class involved. Check that bandwidth on the policy and bandwidth/ceiling on default are what you think. A default ceiling below 100% caps everything unclassified, which on a freshly built policy is nearly all of it.

The trust boundary re-marks nothing. Read the binding first — show configuration commands | match 'policy route MARK-LAN interface' — and confirm it names the LAN. Then confirm the catch-all rule is numbered after the specific rules; a bare rule with no criteria numbered first swallows everything.

Latency under load is still high after switching to fq-codel. Confirm the change actually reached the leaf: sudo tc qdisc show dev eth2 should show fq_codel under the HTB classes. If it shows pfifo, the commit did not take or you edited a different class.

The paired measurement is noisy run to run. It will be, on virtual NICs sharing a host. Take each measurement three times and record the median; a QoS conclusion drawn from a single sample is a coin toss with extra steps.

Cleanup

Everything is on isolated bridges, so cleanup restores a known state rather than service.

Step 1. If you are keeping the topology for the QoS break/fix scenario, stop here and save on each router. Note in the journal that edge is shaping to 20 Mbit with a trust boundary on eth1, so the next person is not surprised by a “slow” link.

Step 2. To return edge to the unshaped state Task 1 built, remove the attachment before the policy — a policy deleted while still attached is a different and less predictable ordering:

configure
delete qos interface eth2
delete qos policy shaper WAN-OUT
delete policy route MARK-LAN
compare
commit
save

Step 3. Confirm the removal from the kernel rather than the configuration:

# On edge. Expect the default qdisc back, and no htb classes at all.
sudo tc qdisc show dev eth2
sudo tc class show dev eth2

Step 4. To reset a router entirely, load the configuration captured in Task 1:

configure
load /config/pre-lab-20.boot
compare
commit
save

Step 5. To remove the topology, delete the three VMs and the two bridges on the hypervisor.

What You Learned

• QoS is only ever judged by measurement. Four paired throughput-and-latency readings told you more than any amount of reading the policy back, and one of them — the ceiling holding class 20 at 12 Mbit on an idle link — was a fault you could only have found by measuring.
• The shaper and the attachment are separate trees. set qos policy shaper builds a policy; set qos interface puts it somewhere. A policy that is never attached is the QoS analogue of a health check with no rule.
• The leaf scheduler is the highest-value single change. Switching one node from drop-tail to fq-codel removed most of the loaded latency without touching the rate, and without anybody having to agree a class plan.
• bandwidth and ceiling are different promises. One is a floor the class is guaranteed; the other is a cap it cannot cross even when the link is idle. Confusing them produces a circuit that under-performs quietly.
• Classification is proved by counters. sudo tc -s class show and two runs of the same probe with different markings is the whole method. The rule set tells you what you meant.
• A DSCP-aware router believes whatever the LAN writes. You watched a host take the priority class with one command, and you watched a policy route re-marking rule set on the ingress interface take it back — proved on the wire by one byte, and in the kernel by a counter that stopped moving.