Interrupt affinity — /proc/irq, RSS, NUMA, IRQBALANCE_BANNED_CPUS

The default interrupt distribution on Linux is a single IRQ to a single core. For a 10 Gbps NIC on a multi-core router, this is catastrophic: one core’s softirq saturates at a fraction of the line rate while the other cores idle. The fix is interrupt distribution: spread the NIC’s queues across the available cores so the kernel can process packets in parallel.

This lesson is the production reference for interrupt affinity on VyOS 1.5 LTS: Receive Side Scaling (RSS), the /proc/irq/<n>/smp_affinity mask, NUMA-aware pinning on multi-socket systems, and the irqbalance discipline that prevents automatic rebalancing from undoing deliberate pinning at boot.

The single-IRQ-to-single-core default

When the Linux kernel initialises a NIC, it allocates one interrupt per NIC and assigns the IRQ to CPU 0 by default. Every packet that arrives on that NIC triggers an IRQ on CPU 0; CPU 0’s softirq pass processes the packet; the other cores are not involved.

For low-rate interfaces (a 1 Gbps management NIC, a serial console), this is acceptable. For a 10 Gbps data path, it is a bottleneck. The single core can process somewhere between 1 and 3 million packets per second (depending on the NIC’s offload features and the kernel version); at 10 Gbps with 64-byte packets, the line rate is 14.88 million packets per second. The single core can do at most 20% of the line rate.

flowchart LR
  subgraph BEFORE["Before RSS"]
    NIC1["eth0 (10Gbps)"] --> IRQ0["IRQ 32"]
    IRQ0 --> CPU0["CPU 0<br/>(softirq 100%)"]
    CPU1["CPU 1"] --> x["idle"]
    CPU2["CPU 2"] --> y["idle"]
    CPU3["CPU 3"] --> z["idle"]
  end

The fix is to give the NIC multiple queues and assign each queue to a different core. The NIC’s RSS hash function (typically Toeplitz on the 5-tuple) distributes incoming packets across the queues; each queue has its own IRQ; each IRQ is pinned to its own core. The result is parallel packet processing.

flowchart LR
  subgraph AFTER["After RSS = 4"]
    NIC["eth0 (10Gbps)"] --> HASH["RSS Toeplitz hash<br/>(5-tuple)"]
    HASH --> Q0["Queue 0<br/>IRQ 32"]
    HASH --> Q1["Queue 1<br/>IRQ 33"]
    HASH --> Q2["Queue 2<br/>IRQ 34"]
    HASH --> Q3["Queue 3<br/>IRQ 35"]
    Q0 --> C0["CPU 0"]
    Q1 --> C1["CPU 1"]
    Q2 --> C2["CPU 2"]
    Q3 --> C3["CPU 3"]
  end

Step 1 — Inspect the current distribution

The operator reads /proc/interrupts to see how many IRQs the NIC has and which cores they are pinned to:

$ cat /proc/interrupts | grep eth0
  32:  1234567  0          0          0    IR-IO-APIC   eth0:TxRx-0
  33:  1234568  0          0          0    IR-IO-APIC   eth0:TxRx-1
  34:  1234569  0          0          0    IR-IO-APIC   eth0:TxRx-2
  35:  1234570  0          0          0    IR-IO-APIC   eth0:TxRx-3

The four queues (TxRx-0 through TxRx-3) are all on CPU 0 — every packet’s interrupt count is on CPU 0’s column. The NIC supports four queues; the kernel is using all of them; but they are all pinned to CPU 0.

The operator inspects each queue’s affinity mask:

$ cat /proc/irq/32/smp_affinity
01

$ cat /proc/irq/33/smp_affinity
01

$ cat /proc/irq/34/smp_affinity
01

$ cat /proc/irq/35/smp_affinity
01

The mask 01 (one bit set, the low bit, corresponding to CPU 0) confirms the diagnosis. The operator’s goal: change each mask to a different CPU.

Step 2 — Configure RSS and affinity

The operator first enables RSS in VyOS:

configure
set system ethernet eth0 receive-hashing on
set system ethernet eth0 hardware-queue rx 4
commit

Then the operator reads the negotiated queue count:

$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       4
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       4

Four combined queues are now active. The operator reads the current RSS indirection table:

$ ethtool -x eth0 indirection
RX:    indirection table for 4 queues, length 4
0:  0 1 2 3

The four queues are being distributed evenly. If the table were all zeros (0 0 0 0), the operator would need to set it explicitly:

$ ethtool -X eth0 equal 4

This sets the indirection table to {0, 1, 2, 3} (each queue receives an equal share of the hash space).

The operator then sets each queue’s IRQ affinity to a different core:

# Distribute the four eth0 queues across four cores
echo 01 > /proc/irq/32/smp_affinity   # Queue 0 → CPU 0
echo 02 > /proc/irq/33/smp_affinity   # Queue 1 → CPU 1
echo 04 > /proc/irq/34/smp_affinity   # Queue 2 → CPU 2
echo 08 > /proc/irq/35/smp_affinity   # Queue 3 → CPU 3

The mask format is hex, with each bit representing a CPU. 01 is CPU 0, 02 is CPU 1, 04 is CPU 2, 08 is CPU 3. For more CPUs, the mask extends (0f is CPUs 0-3, 10 is CPU 4, f0 is CPUs 4-7).

Step 3 — Validate the distribution

After the changes, the operator reads /proc/interrupts again:

$ cat /proc/interrupts | grep eth0
  32:  1234567  0          0          0    IR-IO-APIC   eth0:TxRx-0
  33:  0          1234568  0          0    IR-IO-APIC   eth0:TxRx-1
  34:  0          0          1234569  0    IR-IO-APIC   eth0:TxRx-2
  35:  0          0          0          1234570  IR-IO-APIC   eth0:TxRx-3

The interrupt counts are now distributed across all four CPUs. The operator verifies the softirq distribution:

$ mpstat -P ALL 1 5
13:55:00  CPU    %usr   %nice   %sys  %soft  %steal  %idle   %wait
13:55:01  all   5.50    0.00   1.25  25.00    0.00  68.25    0.00
13:55:01    0   5.00    0.00   1.00  25.00    0.00  69.00    0.00
13:55:01    1   5.50    0.00   1.50  26.00    0.00  67.00    0.00
13:55:01    2   6.00    0.00   1.50  24.50    0.00  68.00    0.00
13:55:01    3   5.50    0.00   1.00  24.50    0.00  69.00    0.00

The softirq is now balanced across all four cores. Throughput should approach the line rate.

NUMA awareness

On a multi-socket server (NUMA architecture), a NIC is physically attached to one socket’s PCIe lanes. Packets that arrive on a NIC attached to socket 0 must traverse the inter-socket link (QPI/UPI) to be processed by a core on socket 1, which adds latency and consumes inter-socket bandwidth.

flowchart LR
  subgraph S0["Socket 0"]
    C0["CPU 0"]
    C1["CPU 1"]
    N0["NIC eth0<br/>(PCIe on socket 0)"]
  end
  subgraph S1["Socket 1"]
    C2["CPU 2"]
    C3["CPU 3"]
  end
  N0 --> C0
  N0 --> C1
  C0 -. QPI .-> C2
  C1 -. QPI .-> C3

The discipline: pin the NIC’s IRQs and queues to cores on the same socket as the NIC’s PCIe attachment. The kernel’s first-touch allocation usually does this automatically, but the operator must verify with numactl and lscpu:

$ lscpu | grep NUMA
NUMA node(s):       2
NUMA node0 CPU(s):  0-15
NUMA node1 CPU(s):  16-31

$ cat /sys/class/net/eth0/device/numa_node
0

# Pin eth0's queues to NUMA node 0 (CPUs 0-15)
echo 0f > /proc/irq/32/smp_affinity   # CPUs 0-3
echo f0 > /proc/irq/33/smp_affinity   # CPUs 4-7
# ... etc.

A common mistake is to spread the IRQ affinity evenly across all CPUs in a NUMA system without checking the NIC’s socket. The result is a QPI-bound softirq path that is slower than the original single-core design.

irqbalance and IRQBALANCE_BANNED_CPUS

The irqbalance daemon monitors interrupt distribution and rebalances IRQs across CPUs to spread the load. By default, irqbalance moves IRQs to whichever cores have the most idle time. This is good for general-purpose servers but bad for routers, where the operator has deliberately tuned the IRQ distribution and does not want irqbalance to undo it.

The operator configures irqbalance with a banned-CPUs mask:

$ cat /etc/default/irqbalance
IRQBALANCE_ARGS="--banirq=32 --banirq=33 --banirq=34 --banirq=35"

Or with a banned-CPUs mask (less common; bans all IRQs on those CPUs):

$ cat /etc/default/irqbalance
IRQBALANCE_BANNED_CPUS="00000000,0000000f"
# Banned CPUs are bits 0-3 (the cores handling eth0)
# IRQs on those CPUs are not rebalanced

The VyOS way to configure this:

configure
set system irqbalance banned-cpus "0-3"
set system irqbalance enabled
commit

After this, irqbalance will not move eth0’s IRQs off CPUs 0-3. The operator’s deliberate distribution is preserved.

Production failure modes

The interrupt-affinity failure modes the operator encounters:

• Single-core softirq saturation despite RSS. RSS is configured, but the NIC’s driver does not honour the multi-queue assignment. Verify with ethtool -l <nic> and check the driver’s documentation. Some drivers require a module parameter to enable multi-queue.
• NUMA-cross softirq. The NIC is pinned to socket 0’s cores, but the softirq crosses to socket 1 because the operator distributed the affinity across all cores. Result: QPI-bound packet path. Fix: re-pin to socket 0 only.
• irqbalance undoes the pinning. The operator sets the affinity manually; irqbalance moves the IRQ back after a few minutes. Fix: configure IRQBALANCE_BANNED_CPUS or use the VyOS system irqbalance knobs.
• Affinity lost after reboot. The operator sets the affinity manually; the router reboots; the affinity is back to CPU 0. Fix: persistence via irqbalance or systemd unit.
• RSS hash clustering. The 5-tuple hash distributes unevenly because the traffic has a few large flows (e.g., one TCP session dominates). Result: one queue saturates while others idle. Fix: change the hash policy (ethtool -X <nic> hkey or use tcp/ip indirection); or use flow-steering to direct specific flows to specific queues.
• IRQ storm. A misbehaving device fires an IRQ thousands of times per second; the softirq cannot keep up. Fix: identify the device with cat /proc/interrupts and disable or replace it.

Rollback

Interrupt-affinity changes are immediate but ephemeral; the rollback discipline:

• Manual /proc/irq/<n>/smp_affinity writes — restore the original mask. If the original was the default 01 (CPU 0), reset with echo 01 > /proc/irq/<n>/smp_affinity.
• ethtool -L <nic> combined <n> — restoring to 1 puts the NIC in single-queue mode; packets stop being distributed.
• system irqbalance knobs — use delete system irqbalance banned-cpus and commit. This lets irqbalance resume normal rebalancing.
• For all changes, use commit-confirm 5 so the auto-rollback fires if the change has unintended consequences.

configure
# ... make the change ...
commit-confirm 5
# If the change causes throughput regression or peer loss,
# the auto-rollback restores the previous configuration.

Production discipline

Cross-course references

• Part L-01 (L-VyOS-Performance / CPU saturation) covers the diagnostic method that identifies single-core softirq saturation.
• Part L-03 (L-VyOS-Performance / packet drops) covers softnet_stat and drop reasons, which are the next step after interrupt distribution is in place.
• Part VII (VII-VyOS-Interfaces) covers interface configuration, including the set system ethernet eth0 mtu knob that interacts with IRQ sizing.
• The Linux course’s V-Linux-NetConfig and XXII-Linux-NetTroubleshoot parts cover the same primitives from the host perspective (NUMA, IRQ subsystem, sysfs).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (IRQ distribution automation with Ansible).
• The Proxmox course covers NUMA-aware VM configuration that affects interrupt distribution inside VMs.

Quiz