Run queue and context switching - measuring contention

The kernel scheduler maintains a run queue of runnable processes. When more processes are runnable than CPUs can handle, the queue grows. Context switches are the cost of moving between processes. Both metrics reveal contention.

Run queue

vmstat 1

Output:

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs  us sy id wa st
 4  0      0 10000  5000 40000    0    0     5    20  500 1000  80  5  5 10  0
 4  0      0 10000  5000 40000    0    0     0     0  500 1000  82  4  4 10  0

r is the run queue length (processes waiting for CPU). b is blocked (waiting for I/O).

Healthy:

• r < number of CPUs.
• r trends down when load decreases.

Saturation:

• r > number of CPUs sustained: CPU saturation.
• r rising: incoming workload exceeds capacity.

Context switches

Two kinds:

• Voluntary (cswch/s): process gives up CPU (waits for I/O, mutex).
• Involuntary (nvcswch/s): kernel preempts process (timeslice expired).

pidstat -w 1

Output:

14:30:00  UID    PID   cswch/s  nvcswch/s  Command
14:30:01  1000    1234     500.00      1.50  java
14:30:01  1000    1235     800.00      2.10  java

A healthy system has voluntary cswch for I/O wait. High involuntary cswch (nvcswch) indicates:

• CPU saturation (processes are being preempted).
• Thread scheduling (lock contention; many threads waiting).
• CPU affinity issues (processes bouncing between CPUs).

Spot contention

High context switches (millions per second) plus low CPU utilisation often indicates lock contention. The CPUs are idle because processes are waiting on each other.

# Total context switches per second
cat /proc/<pid>/status | grep ctxt

A process with a high ctxt (total context switches) but low CPU usage is spending most of its time waiting.

Thread scheduling

Linux threads share CPU time. With many threads:

• Lock contention: threads spend time waiting for locks.
• Thread imbalance: one thread is the bottleneck.

pidstat -t shows per-thread stats:

pidstat -t -p <pid> 1

Output:

14:30:00  UID    PID   TID    %usr  %system   %guest  %wait  %CPU  Command
14:30:01  1000  1234   1234   45.00    2.00     0.00   0.10  47.00  java
14:30:01  1000  1234   1235    2.00    0.50     0.00   0.05   2.50  java
14:30:01  1000  1234   1236    2.00    0.50     0.00   0.05   2.50  java
...

If one thread is at 47% CPU and others at 2.5%, that thread is the bottleneck. Often a lock or a serialisation point.

Common patterns

Knowledge check