Performance baselines — pbench, record baselines, alert on deviation, golden image

A performance baseline is a snapshot of the router’s measured behaviour at a known state. The baseline is the operator’s reference for “normal”; a deviation from the baseline is an early warning of a regression (a configuration change that hurt throughput, an upgrade that increased latency, a new workload that the router cannot sustain). Without a baseline, the operator cannot distinguish “the router has always been at 30% CPU” from “the router just started running at 30% CPU because of a regression”.

This lesson is the production reference for performance baselines on VyOS 1.5 LTS: capturing baselines with pbench and sysstat, golden-image comparison, alert thresholds for deviation, and the production discipline that turns baselines into an early-warning system.

What to baseline

A performance baseline has five components:

flowchart LR
  subgraph BASELINE["Performance baseline"]
    CPU["CPU utilisation<br/>(userspace, softirq, hardirq)"]
    MEM["Memory utilisation<br/>(used, cached, slab)"]
    TPUT["Throughput<br/>(iperf3, wrk, fio)"]
    LAT["Latency<br/>(ping, http latency)"]
    CONV["Convergence<br/>(BGP/OSPF reconvergence time)"]
  end

Each component is measured at a known state (a “golden image” — the router’s known-good configuration and workload) and recorded for comparison after any change.

CPU utilisation. mpstat -P ALL 1 for 60 seconds under a known load (e.g., 1 Gbps of iperf3 traffic). The baseline is the average %usr + %sys + %soft per core and the peak (P99).

Memory utilisation. free -m and cat /proc/meminfo for the used, cached, slab, and kernel-stack values. The baseline is the working-set size and the slab growth.

Throughput. iperf3 -c <peer> for TCP and UDP throughput, with -P 4 (parallel streams) for multi-core measurement. The baseline is the peak TCP throughput, the UDP jitter, and the packet-loss rate.

Latency. ping -c 100 <peer> for round-trip-time (RTT) and wrk -t 4 -c 100 -d 30s <url> for HTTPS request latency. The baseline is the RTT P50, P95, P99, and the HTTP request latency P50, P95, P99.

Convergence. clear ip bgp <peer> soft for BGP, or clear ip ospf process for OSPF; measure the time until show ip route summary returns to its pre-clear state. The baseline is the convergence time in seconds.

The operator records all five into a structured baseline document:

# Performance baseline — R1 (edge router 1)
# Captured: 2026-08-15 13:42 UTC by operator@example.com
# Image: VyOS 1.5-rolling-20260815
# Configuration: production-baseline (commit 7a3f8e9)
# Workload: 1 Gbps sustained iperf3, 5000 BGP routes, 100 OSPF routes

CPU:
  avg_userspace: 8.5%
  avg_softirq:   5.2%
  peak_userspace: 35.0%
  peak_softirq:   25.0%

Memory:
  used:   1024 MB
  cached: 4096 MB
  slab:    256 MB
  total:  8192 MB

Throughput:
  tcp_throughput: 9.4 Gbps (iperf3 -P 4, 1500-byte packets)
  udp_throughput: 9.8 Gbps (iperf3 -u -b 10G)
  udp_jitter:     0.05 ms
  udp_loss:       0.01%

Latency:
  rtt_p50:        0.5 ms
  rtt_p95:        0.8 ms
  rtt_p99:        1.2 ms
  http_p50:       5 ms
  http_p95:       10 ms
  http_p99:       25 ms

Convergence:
  bgp_soft_clear: 12 seconds (until full table restored)
  ospf_restart:   5 seconds (until all neighbours Adjacent)

Capturing baselines with sysstat

sysstat (specifically sadc — the system activity data collector) records the kernel’s per-second metrics to a file. The operator schedules sadc to run continuously and harvests the data with sar (system activity reporter).

VyOS 1.5 LTS includes sysstat by default. The operator enables it:

configure
set system syslog global facility local7
set system syslog global archive file /var/log/local.log
set system package repository community components main
set system package repository community url https://repo.vyos.net
set system package update interval add 7
set system package update interval update 7
set system package auto-sync disable
commit
# Install sysstat
sudo apt-get update
sudo apt-get install -y sysstat

Configure sysstat to record every 10 seconds (the default is every 10 minutes, too coarse for performance baselining):

$ sudo vi /etc/cron.d/sysstat
# Change the interval from 10 minutes to every 10 seconds
*/1 * * * * root /usr/lib/sysstat/sadc 10 /var/log/sysstat/sa$(date +%d)

Harvest the data:

$ sar -u -r -n DEV 1 60
# Captures 60 samples (1 per second) of CPU, memory, and network utilisation

Save the output as part of the baseline:

$ sar -u -r -n DEV 1 60 > baseline-sar-20260815.txt

Capturing baselines with pbench

pbench is a toolkit for orchestrating repeatable performance benchmarks. It wraps iperf3, fio, wrk, netperf, and other tools into a “test plan” that captures the command, the configuration, the system state, and the result into a single directory.

The operator installs pbench on a benchmarking host (not the router):

$ sudo apt-get install -y pbench

The operator registers the router as a target:

$ pbench-register-tool --label=iperf3
$ pbench-register-tool-set --label=iperf3

The operator runs a benchmark:

$ pbench-run-iperf3 --target 198.51.100.1 --threads 4 --runtime 60
# Runs iperf3 from the benchmarking host to 198.51.100.1 (the router)
# 4 parallel streams, 60 seconds

The result is captured in a directory with the timestamp:

$ ls /var/lib/pbench-agent/iperf3-results/2026-08-15_13-42-00/
result.json  iperf3.txt  system.config.txt

The operator archives the directory into the baseline:

$ cp -r /var/lib/pbench-agent/iperf3-results/2026-08-15_13-42-00/ baseline-pbench-20260815/

Golden image

A golden image is the router’s known-good configuration and software version, captured as a VyOS image. After every configuration or software change, the operator can restore the golden image and re-run the baseline; the difference between the post-change baseline and the golden-image baseline is the regression caused by the change.

The operator captures the golden image with add system image:

configure
# ... apply the known-good configuration ...
commit
save
add system image https://repo.vyos.net/vyos/1.5-rolling-20260815/vyos-1.5-rolling-20260815-amd64.iso

The image is added as a bootable alternative. After a configuration or upgrade change, the operator can boot the golden image and verify the regression is configuration-induced, not software-induced.

Alert thresholds

The alert thresholds are derived from the baseline. A common pattern:

# Alert thresholds for R1
# Triggered when deviation from baseline exceeds:
cpu_userspace:   baseline + 20%
cpu_softirq:     baseline + 20%
memory_used:     baseline + 30%
throughput_tcp:  baseline - 10%
latency_rtt_p99: baseline + 50%
convergence:     baseline + 50%

The thresholds are encoded into the operator’s monitoring system (Prometheus, Grafana, Nagios, Zabbix). The operator is alerted when the threshold is breached.

The discipline: do not alert on absolute values; alert on deviation from the baseline. An idle router at 5% CPU is not abnormal; a busy router at 30% CPU is not abnormal; a router that was at 5% and is now at 30% is abnormal.

Production failure modes

The performance-baseline failure modes the operator encounters:

• No baseline exists. A regression is detected by user complaint (the network is slow), not by automated alert. The operator must rebuild the baseline from scratch, which is expensive and may not match the pre-regression state. Fix: capture the baseline before the next change.
• Baseline captured under unrealistic load. The baseline is captured at idle; the production load is 5x higher; the alert thresholds are wrong. Fix: capture the baseline under realistic load (production traffic replay or synthetic load generation).
• Baseline not updated after a legitimate change. A configuration change legitimately increases CPU (because it adds a feature); the baseline is not updated; the operator is alerted on the legitimate change. Fix: update the baseline after every accepted change.
• Golden image lost. The router was upgraded and the golden image was not preserved; the operator cannot compare against the pre-upgrade state. Fix: keep the golden image on the router (add system image) and on a remote backup.
• Baseline too coarse. The baseline is captured every 10 minutes (sysstat default); a transient spike is missed. Fix: capture every 10 seconds; keep the raw data for postmortem analysis.
• Thresholds set too tight. The alert fires on every minor variation; the operator learns to ignore the alert. Fix: set thresholds wider; review after a month.

Rollback

Baseline changes are about audit, not rollback. The discipline:

• Capture a new baseline after every accepted change. The previous baseline is archived; the new baseline is the reference.
• Do not delete old baselines unless the configuration has been rolled back and re-validated. The archive is the operator’s regression history.
• For software upgrades, capture the baseline before the upgrade and after; the difference is the upgrade’s regression footprint. If the footprint is unacceptable, the upgrade is rolled back.

Production discipline

Cross-course references

• Part L-01 (L-VyOS-Performance / CPU saturation) covers the diagnostic method that uses baselines to identify CPU saturation.
• Part XLIX (XLIX-VyOS-Monitoring) covers the monitoring and observability integration that consumes the baseline data (Prometheus exporters, Grafana dashboards).
• Part LVI (LVI-VyOS-ImageManagement) covers image management and the add system image primitive for golden-image preservation.
• The Observability course covers the telemetry side (baseline dashboards in Grafana, alert configuration in Prometheus / Alertmanager).
• The Proxmox course covers the VM-side performance baseline (pbench runs in a VM to capture per-VM baselines).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (baseline capture automated via Ansible playbooks).

Quiz