Interrupts, softirqs, and steal time - hardware and VM signals

Hardware interrupts (IRQs) and softirqs are how the kernel responds to hardware events. Steal time is a VM-specific signal. All three can show up as CPU saturation but have different causes.

Hardware interrupts

cat /proc/interrupts

Output:

           CPU0       CPU1       CPU2       CPU3
  0:         18          0          0          0   IO-APIC   2-edge      timer
  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi
 16:        123        456        789        234  IO-APIC  16-fasteoi   ehci_hcd:usb1
...

Each line is an IRQ. Columns:

• IRQ number.
• Per-CPU counts.
• Trigger type (edge, level).
• Source device.

To find interrupt storms:

watch -n 1 'cat /proc/interrupts'

An IRQ that grows rapidly is suspect.

Softirqs

Softirqs are deferred interrupt processing. Common ones:

• NET_RX: network packet receive.
• NET_TX: network packet send.
• TIMER: timer expiration.
• SCHED: scheduler.
• RCU: read-copy-update.

cat /proc/softirqs

To check softirq usage as a CPU percentage:

mpstat 1 5
# Look at %hi (hardware interrupts) and %si (softirqs)

High %si (softirq) often indicates network packet processing is the bottleneck. Check with:

ss -tinp | head    # listening sockets with backlogs

Steal time

Steal time is the percentage of CPU the hypervisor gave to other VMs. On a non-VM host, steal is always 0.

top -bn 1

%Cpu(s): 80.0 us,  5.0 sy,  0.0 ni,  5.0 id,  0.0 hi,  0.0 si, 10.0 st

%st = 10% means 10% of the time, the VM wanted CPU but the hypervisor gave it to another VM. Cause: hypervisor overcommit.

Fixes:

• Move the VM to a less-loaded host.
• Reduce the VM’s vCPU count.
• Notify the cloud provider.

Interrupt affinity

Interrupts are bound to CPUs. Network interrupts on CPU 0 mean packets are processed by CPU 0. For multi-queue NICs:

# Show IRQ affinity
cat /proc/irq/<N>/smp_affinity

# Set affinity
echo "f" > /proc/irq/<N>/smp_affinity    # all CPUs
echo "1" > /proc/irq/<N>/smp_affinity    # CPU 0 only

For high-throughput network, distribute interrupts across CPUs.

Softirq mitigation

Measure before you tune. /proc/net/softnet_stat is one line per CPU, hexadecimal, and the first three columns are the ones that decide what to change:

# col 1 = packets processed
# col 2 = packets DROPPED because the input backlog was full
# col 3 = time_squeeze: the NET_RX softirq ran out of budget
#         with work still queued
awk '{ printf "cpu%-3d processed=%d dropped=%d squeeze=%d\n", \
       NR-1, strtonum("0x"$1), strtonum("0x"$2), strtonum("0x"$3) }' \
    /proc/net/softnet_stat

Read the result before touching a knob:

• dropped is growing - packets arrived faster than the softirq drained the per-CPU backlog. Raise net.core.netdev_max_backlog.
• squeeze is growing - the softirq is being cut off mid-drain. Raise net.core.netdev_budget (default 300).
• neither is growing - your %si is real work, not a queue problem. Spread the interrupts (see IRQ affinity above) or coalesce them; do not touch the queue lengths.

# Only if col 2 (dropped) is non-zero and rising:
sudo sysctl -w net.core.netdev_max_backlog=4000   # default 1000

# Only if col 3 (squeeze) is non-zero and rising:
sudo sysctl -w net.core.netdev_budget=600         # default 300

# Fewer interrupts per packet, at the cost of a little latency
ethtool -C eth0 adaptive-rx on rx-usecs 50

Common patterns

Knowledge check