The cost of tracing - eBPF overhead, dropped events and production safety · Linux

eBPF is often described as having negligible overhead. That is true of a well-chosen probe and false by several orders of magnitude of a badly chosen one. The difference between a one-liner that costs 0.1% and one that halves the throughput of a production database is not the tool - it is which event you attached to and what you did in the handler.

The verifier guarantees a BPF program cannot crash the kernel or loop forever. It guarantees nothing about how often the program runs.

Cost is frequency times per-event work

overhead  =  events per second  x  cost per event

Both factors vary enormously, and the first one varies more.

Attachment	Rough per-event cost	Notes
Tracepoint	Tens of nanoseconds	Stable ABI, compiled-in, cheapest
kprobe on a cold function	Tens to low hundreds of ns	Breakpoint plus trap handling
kretprobe	Roughly double a kprobe	Needs an entry probe plus a return trampoline
uprobe	Microseconds	A trap into the kernel from user space and back
uretprobe	Worse than a uprobe	Same, twice

The uprobe row is the one that surprises people. A uprobe costs roughly one to two microseconds per hit, because each hit is a full user-to-kernel-to-user round trip. That is fine on a function called a hundred times a second. On a function called a million times a second it is one to two seconds of CPU per second per core - the process simply stops making progress.

Frequency is the part you can look up before attaching:

# How often does this actually fire? Count for 10 seconds first.
sudo timeout 10 bpftrace -e '
  tracepoint:raw_syscalls:sys_enter { @[probe] = count(); }'

@[tracepoint:raw_syscalls:sys_enter]: 4128841

400,000 events per second. At even 50 ns each, that is 2% of one core just to count them - and this is the cheap probe type. Attaching a uprobe at that rate would be catastrophic.

A uprobe on a hot libc function is the classic self-inflicted outage

# DO NOT run this on a production host.
sudo bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { @ = count(); }'

That attaches to every malloc() in every process on the host that maps libc. A busy application allocates millions of times per second, so the host acquires several microseconds of kernel work per allocation, across every process at once.

The observed behaviour is a host that goes from healthy to unresponsive in seconds, with CPU pinned in kernel time and no single process to blame. Detaching fixes it immediately, which is the one mercy - but if you attached it over a session that then becomes unresponsive, getting to a shell to press Ctrl-C is its own problem.

Three rules that prevent it:

Scope to one process with -p PID, always, for uprobes.
Prefer a USDT probe or a tracepoint if one exists for what you want. A tracepoint at the same place is 20-50x cheaper.
Count the frequency first with a short, scoped run before attaching anything that does work per event.

And keep a second session open on the host before you attach anything on a hot path. That habit costs nothing and has saved many reboots.

Aggregate in the kernel

The second-biggest cost is what your handler does, and the dominant factor there is whether every event has to reach user space.

# BAD on a busy path: one line of output per event.
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_openat {
    printf("%s %s\n", comm, str(args.filename));
  }'

# GOOD: aggregate in kernel, transfer a summary at exit.
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

The first copies a string and pushes a record through the perf ring buffer for every event; the second increments a counter in a kernel map, and the map is read once when the program exits. On a high-rate path that is the difference between a usable tool and a tool that is itself the load.

The aggregating builtins - count(), sum(), avg(), min(), max(), hist(), lhist(), stats() - all do their work in the kernel. Reach for hist() in particular: a latency distribution is almost always more useful than a stream of individual latencies, and it costs a fraction as much.

Read the warnings - bpftrace tells you when it dropped data

When the ring buffer overflows, bpftrace says so:

Lost 1288 events

When a map fills, it says that too:

WARNING: Map @start has reached its max size of 4096 entries.

Neither stops the program, and both mean the output you are about to reason from is incomplete. A latency histogram missing its slowest events is worse than no histogram, because it is convincingly wrong.

The relevant limits are environment variables, not flags:

sudo BPFTRACE_PERF_RB_PAGES=256 BPFTRACE_MAX_MAP_KEYS=131072 \
  bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm, pid] = count(); }'

The defaults on a current build:

BPFTRACE_MAX_MAP_KEYS      [default: 4096]  max keys in a map
BPFTRACE_PERF_RB_PAGES     [default: 64]    pages per CPU for the ring buffer
BPFTRACE_MAX_PROBES        [default: 1024]  max number of probes
BPFTRACE_MAX_STRLEN        [default: 1024]  bytes on BPF stack per str()

Raising them costs kernel memory: BPFTRACE_PERF_RB_PAGES is per CPU, so 256 pages on a 64-core host is 64 MB of pinned memory. Raise deliberately, and prefer reducing cardinality instead - keying a map on comm rather than on (comm, pid, tid) is usually the better fix, because an unbounded key space fills any map you give it.

Bound what you attach

bpftrace expands wildcards into one probe per match, and the count can be startling:

sudo bpftrace -l 'kprobe:vfs_*'

kprobe:vfs_clone_file_range
kprobe:vfs_copy_file_range
kprobe:vfs_create
...

Count before attaching:

sudo bpftrace -l 'kprobe:tcp_*' | wc -l

A pattern like kprobe:* matches tens of thousands of functions. Attaching them takes minutes, holds locks while it does so, and BPFTRACE_MAX_PROBES exists specifically to stop you:

ERROR: Can't attach to 41284 probes because it exceeds the current limit
of 1024 probes.

That error is a guardrail. Raising the variable to get past it, on a production host, is almost always the wrong response.

--dry-run compiles and attaches everything and then exits immediately, which is the cheapest way to find out whether a script will attach cleanly:

sudo bpftrace --dry-run -e 'kprobe:tcp_sendmsg { @ = count(); }'

Bound the run

A tracing session left attached is the failure that outlives the investigation. Bound every one:

# Stop after 30 seconds regardless of what happens to your terminal.
sudo timeout 30 bpftrace -e 'tracepoint:block:block_rq_issue { @ = count(); }'

Or inside the script, which also gives you control over the output:

interval:s:30 { exit(); }

exit() runs the END block and prints the maps, so a bounded run still produces its summary.

Check afterwards that nothing was left behind:

sudo bpftool prog list | tail -20
sudo bpftool map list | tail -20

Programs pinned by a tool that died without cleaning up hold kernel memory and keep running their probes. bpftool prog show gives the load time and process name, which is usually enough to identify an orphan.

Things to refuse

Some attachments are not worth the risk on a production host at all:

Attachment	Why not
uprobe on `malloc`, `memcpy` or similar, host-wide	Millions of events per second at microsecond cost
kprobe on `_raw_spin_lock` or scheduler internals	Extremely hot, and probing the scheduler from the scheduler is its own problem
Any `printf` per event on a path exceeding a few thousand events per second	Ring buffer saturation and lost events
`kprobe:*` or an unbounded wildcard	Minutes of attach time and lock contention
`--unsafe` anything	It enables destructive actions - see below

bpftrace --unsafe enables functions such as system(), which runs a shell command from a probe handler. On a probe that fires a thousand times a second, that is a thousand forks a second. It exists for deliberate, scoped automation and has no place in an exploratory session.

kprobes attach to symbols, and symbols are not an ABI

A tracepoint is a stable interface the kernel promises to keep. A kprobe attaches to an arbitrary kernel function by symbol name, and those change without notice between kernel versions - which is why a one-liner from a blog post fails on your kernel:

ERROR: Unrecognized probe type: kprobe:tcp_set_state

or, worse, attaches to a function whose arguments have changed and reads the wrong field, producing plausible numbers that are wrong. A kprobe reading arg2 as a size on a kernel where an argument was inserted before it will report confident nonsense.

Two consequences for how you work:

Prefer tracepoints whenever one exists for what you want. bpftrace -l 'tracepoint:*' before reaching for a kprobe.
Verify the symbol exists on this kernel before trusting a script you did not write:
```
sudo bpftrace -l 'kprobe:tcp_sendmsg'
grep -w tcp_sendmsg /proc/kallsyms
```

Any kprobe-based tool must be re-validated after a kernel upgrade. Put that in the kernel upgrade checklist, because a monitoring agent that silently stops matching is not going to tell you.

A safe default shape

Putting the habits together, an exploratory tracing session on a production host looks like this:

# 1. Does the probe exist on this kernel?
sudo bpftrace -l 'tracepoint:block:block_rq_issue'

# 2. How often does it fire? Short, cheap, bounded.
sudo timeout 5 bpftrace -e 'tracepoint:block:block_rq_issue { @ = count(); }'

# 3. Will the real script attach cleanly?
sudo bpftrace --dry-run /usr/sbin/biolatency.bt

# 4. Run it bounded, aggregating in kernel, with a second session open.
sudo timeout 30 /usr/sbin/biolatency.bt

# 5. Confirm nothing was left attached.
sudo bpftool prog list | tail -5

Five commands, four of which are cheap. The discipline is not paranoia - eBPF genuinely is safe enough to run on production, and that is exactly why it gets run on production, which is why the failure modes matter.