bpftrace and bcc - sysadmin use cases and examples

bpftrace and bcc are the practical tools for sysadmin eBPF work. This lesson covers the most useful patterns.

bpftrace one-liners

# Count syscalls by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Histogram of read() latency, in microseconds.
# nsecs is nanoseconds - divide, or the bucket labels lie by 1000x.
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @usecs = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}'

# Top 10 files by open count
bpftrace -e '
tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }'

bcc tools

bcc ships pre-built tools. The catch is that no distribution installs them under their upstream names, so every example you find online fails on a stock host with command not found - usually mid-incident.

Common ones:

# Block I/O latency histogram
sudo biolatency

# Per-I/O tracing, one line per block request
sudo biosnoop

# TCP connection attempts
sudo tcpconnect

# New process execution
sudo execsnoop

# File open tracing
sudo opensnoop

# Syscall counts, system-wide or per process (-P)
sudo syscount
sudo syscount -P

# TCP retransmits
sudo tcpretrans

# DNS and other name-resolution latency (traces getaddrinfo/gethostbyname)
sudo gethostlatency

syscount and syscount-bpfcc are not two tools. They are the same tool under two packaging names - the second is simply what Debian and Ubuntu call the first.

Common use cases

“Where is latency coming from?”

# Substitute your own values before running:
PID=1234

# Block I/O latency histogram, in milliseconds
sudo biolatency -m

# Which files and processes are driving that I/O
sudo filetop -p "$PID"

# Syscall latency, not just counts
sudo syscount -L
sudo syscount -L -p "$PID"

biolatency measures block device latency, not syscall latency. If it looks clean, the delay is above the block layer - in the filesystem, in a lock, or in the application - and syscount -L is the tool that narrows it down.

“Why are packets being dropped?”

# Where in the kernel the skb was freed. Not a bcc tool: dropwatch
# uses the kernel dropmon netlink interface, and it is interactive -
# type 'start' at its prompt, and 'stop' to end.
sudo dropwatch -l kas

# Same question with bpftrace, non-interactive and scriptable
sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack] = count(); }'

# Interface-level counters first - they are free and often enough
ip -s link show dev eth0
nstat -az | grep -i drop

Start with ip -s link. If the drops are counted on the NIC itself, the answer is ring-buffer sizing or offload behaviour, and no amount of kernel stack tracing will show it.

“What is process X doing?”

# Substitute your own values before running:
PID=1234

# Files opened by one process
sudo opensnoop -p "$PID"

# Syscalls it is making, with latency
sudo syscount -L -p "$PID"

# Outbound connections it is attempting
sudo tcpconnect -p "$PID"

Note the space in -p <pid>. opensnoop-p is not a tool; it is a typo that produces command not found, which is easy to misread as “bcc is not installed” when you are under pressure.

“Why is the system slow?”

# Substitute your own values before running:
PID=1234

# Scheduler run-queue latency: how long runnable tasks wait for CPU
sudo runqlat 2 5          # 2-second intervals, 5 samples

# Off-CPU time: where processes are blocked rather than running
sudo offcputime -p "$PID"

# Which syscalls dominate, by count and by time
sudo syscount
sudo syscount -L

runqlat reports waiting for CPU, not CPU usage - a high run-queue latency means you have more runnable work than cores, whereas a slow process with an idle run queue is blocked on something else, which is what offcputime answers. For plain CPU usage, top and pidstat are still the right tools.

bpftrace cookbook

The bpftrace repository has many examples. Some favourites:

# Bytes read per process. sum(args->count), not count():
# count() tallies how many read() calls were made, which is a very
# different number from how many bytes they asked for.
bpftrace -e 'tracepoint:syscalls:sys_enter_read { @bytes[comm] = sum(args->count); }'

# read() calls per process, when that is what you actually want
bpftrace -e 'tracepoint:syscalls:sys_enter_read { @calls[comm] = count(); }'

# TCP connect attempts by process. args->uservaddr is a POINTER into
# userspace to a struct sockaddr - it is not a string, so %s on it
# prints garbage. Count the attempts by process instead, and use
# tcpconnect (bcc) when you need the destination addresses.
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { @connects[comm] = count(); }'

# Disk I/O size histogram
bpftrace -e 'tracepoint:block:block_rq_issue { @bytes = hist(args->bytes); }'

# TCP retransmits
bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[args->sport] = count(); }'

Knowledge check