Performance testing and benchmarking: validating your cluster is fast enough

SLOs are promises. Benchmarks are how you verify those promises hold. This lesson covers the practical benchmarking discipline for a PVE cluster: what to measure, how to measure it, and how to keep the results honest over time.

What to benchmark

Three layers, each telling a different story:

The synthetic layer tells you hardware capability. The component layer tells you how well PVE virtualises that hardware. The application layer tells you what users actually experience.

All three are needed. A synthetic benchmark that shows “10 Gbps throughput” doesn’t help if the VM running the database sees microsecond latency spikes.

Synthetic CPU benchmarking

stress-ng is the standard Linux stress test:

apt install -y stress-ng

# CPU stress: 4 workers, 60 seconds
stress-ng --cpu 4 --timeout 60 --metrics

# Output:
# stress-ng: info:  [12345] dispatching hogs: 4 cpu
# stress-ng: info:  [12345] successful run completed in 60.00s
# stress-ng: metrics:
#    stressor      bogo ops
#    cpu            1234567
#    wall-clock seconds    60.00
#    user-time seconds     239.5
#    system-time seconds    0.5
#    bogo ops/s (real)     20576
#    bogo ops/s (usr+sys)   100175

# Compare to baseline:
stress-ng --cpu 4 --timeout 60 --metrics 2>&1 | grep 'bogo ops/s (real)'
# 20576

# Now compare to a year-ago baseline
# If this drops to 12000, the host has lost 40% CPU performance

Run on every host quarterly, save the output to version control, alert on >20% regression.

For more realistic CPU testing, run a multi-VM workload:

# Create 4 VMs, each with 2 vCPUs
for i in 100 101 102 103; do
  qm create $i --memory 2048 --cores 2 --net0 bridge=vmbr0
  # ... boot and install ...
done

# Run stress-ng inside each VM in parallel
for i in 100 101 102 103; do
  ssh pve-01 "ssh vm$i 'stress-ng --cpu 2 --timeout 60 --metrics'"
done

This tests the actual contention PVE creates with multiple VMs.

Synthetic memory benchmarking

# Memory bandwidth (STREAM benchmark)
apt install -y stream
cd /tmp
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
gcc -O2 -fopenmp stream.c -o stream
export OMP_NUM_THREADS=4
./stream
# Output:
# Function    Best Rate MB/s  Avg time     Min time     Max time
# Copy:           12543.4     0.013        0.013        0.013
# Scale:          12539.1     0.013        0.013        0.013
# Add:            12611.5     0.013        0.013        0.013
# Triad:          12601.2     0.013        0.013        0.013

# Memory latency: lmbench is not packaged, so build it from source.
# It measures L1, L2, L3 and RAM latency in nanoseconds.

For VM memory performance specifically:

# Inside a VM, test memory allocation
stress-ng --vm 1 --vm-bytes 1G --timeout 60 --metrics
# Generates 1 GB of memory pressure; measures how fast the VM
# can allocate, write, and free

# Test NUMA effects
numactl --hardware
numactl --membind=0 stress-ng --vm 1 --vm-bytes 512M --timeout 30
numactl --membind=1 stress-ng --vm 1 --vm-bytes 512M --timeout 30
# Compare — if NUMA binding makes a big difference, your VM is
# crossing NUMA boundaries

Synthetic disk benchmarking

# fio as covered in the storage lesson
fio --name=baseline --filename=/var/lib/vz/test.bin \
    --size=4G --bs=4k --rw=randread \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json > /var/log/fio/baseline.json

# Random read IOPS (host-level)
jq '.jobs[0].read.iops' /var/log/fio/baseline.json
# 87432

# For VM disk performance, run inside a VM
ssh vm100 "fio --name=baseline --filename=/var/lib/vz/test.bin \
    --size=2G --bs=4k --rw=randread \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json > /var/log/fio/vm100-baseline.json"

The VM number should be within 10–15% of the host number. If much lower, there is virtio overhead that needs tuning.

Synthetic network benchmarking

# iperf3 between two hosts
iperf3 -s    # On one host
iperf3 -c <other-host> -t 60 -P 4 -i 1
# -P 4: 4 parallel streams
# -i 1: report every second

# UDP (jitter, packet loss)
iperf3 -c <other-host> -t 60 -u -b 1G
# 1 Gbps UDP; measure jitter and loss

# VM-to-VM (same host, different bridges)
iperf3 -s    # In VM 100
iperf3 -c 10.0.0.100 -t 60 -P 4
# Should approach host bridge speed

For storage network (Ceph, NFS, iSCSI):

# On the storage host
iperf3 -s

# On a PVE node connecting to the storage
iperf3 -c <storage-host> -t 60 -P 4
# Should match the storage network bandwidth (10 GbE, 25 GbE, etc.)

Application-level benchmarks

Synthetic benchmarks prove hardware capability. Application benchmarks prove that the user’s workload works.

For a database:

# sysbench OLTP benchmark
apt install -y sysbench
sysbench --db-driver=pgsql --pgsql-host=<host> \
    --pgsql-db=test --pgsql-user=bench \
    --tables=10 --table-size=100000 \
    --threads=16 --time=300 \
    oltp_read_write run

# Reports: transactions per second, latency p99, etc.

For a web server:

# wrk HTTP benchmark
wrk -t 4 -c 100 -d 60s --latency http://web.example.com/

# Reports: requests/sec, latency percentiles

For PBS backup throughput:

# Time a backup
time vzdump 100 --storage pbs-main --mode snapshot

# Reports: backup size / elapsed time = throughput MB/s

Application benchmarks should be run before going to production (baseline) and periodically in production (regression check).

Tracking baselines

The most important discipline: save baselines in version control and re-run periodically.

# /usr/local/bin/run-baseline.sh
#!/bin/bash
DATE=$(date +%Y-%m-%d)
mkdir -p /srv/baselines/$DATE
cd /srv/baselines/$DATE

# Synthetic
stress-ng --cpu $(nproc) --timeout 60 --metrics > stress-ng.txt 2>&1
iperf3 -c <storage-host> -t 30 -P 4 > iperf3.txt
fio --name=randread --filename=/var/lib/vz/test.bin \
    --size=4G --bs=4k --rw=randread \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json > fio.json

# Git commit
git add .
git commit -m "baseline $DATE"

Run this monthly via cron, with monitoring that alerts if the current baseline differs from a year ago by >20%.

Interpreting results

The “drop pattern” tells you where to look:

• All metrics drop: thermal throttling, power issue, noisy neighbour at the host level
• One metric drops: component-level issue (specific disk, NIC, CPU)
• Drop correlates with new kernel/driver: regression from an update — pin to the previous version

Production considerations

• Benchmark in production carefully. Synthetic benchmarks write test files and stress resources. Run during maintenance windows or on canary VMs.
• Compare like-for-like. Always compare the same hardware, same kernel, same BIOS settings. Differences outside the cluster are noise.
• Distrust vendor specs. Vendor-published performance numbers are best-case, with optimal BIOS settings, in their test lab. Your results will differ. Measure in your environment.
• Application benchmarks over synthetic. Synthetic benchmarks prove capability. Application benchmarks prove you can run the user’s workload at the SLO.

Common mistakes

• Benchmarking in a test environment only. The real hardware, real BIOS, real kernel, and real workload matter. Test in production.
• Using only single-stream tests. Real workloads are concurrent. Use multi-stream (-P 4).
• Comparing across hardware generations. A baseline from an older host isn’t meaningful on newer hardware. Tag baselines with hardware info.
• Not testing under failure conditions. Test “what happens to performance when one disk fails” or “when network is at 50% packet loss”. The answers often surprise.

Key takeaways

• Three layers: synthetic, component, application. All three needed.
• Run benchmarks quarterly and save baselines in version control.
• Multi-stream tests reflect real workload concurrency.
• Compare against same-hardware baselines; alert on >20% regression.

Knowledge check