Tracepoints, kprobes, and uprobes - eBPF attachment points

eBPF programs attach to events. There are three kinds of attachment: tracepoints (stable kernel events), kprobes (arbitrary kernel functions), and uprobes (user-space functions). Each has trade-offs.

Tracepoints

Tracepoints are stable kernel instrumentation points. They are added by kernel developers to mark interesting events. The list of tracepoints is stable across kernel versions (within reason).

# List tracepoints
bpftrace -l 'tracepoint:*' | head

# Trace a specific tracepoint
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", comm); }'

Common tracepoints:

• tracepoint:syscalls:sys_enter_* and sys_exit_*: syscall entry and exit.
• tracepoint:net:net_dev_xmit: network transmit.
• tracepoint:block:block_rq_issue: block I/O issue.
• tracepoint:sched:sched_switch: scheduler switch.

kprobes

Kprobes attach to arbitrary kernel functions. They are more flexible than tracepoints but less stable (function names may change between kernels).

# Attach to a kernel function
bpftrace -e 'kprobe:vfs_read { printf("%s\n", comm); }'

# With arguments
bpftrace -e 'kprobe:do_sys_open { printf("%s %s\n", comm, arg0); }'

Common uses:

• Trace a specific function for latency.
• Inspect arguments to understand behaviour.
• Debug kernel subsystems that lack tracepoints.

Risk: kprobes on critical functions can crash the kernel if the program is buggy. Test in a non-production environment.

uprobes

Uprobes attach to user-space functions. Useful for tracing applications without modifying them.

# Substitute your own values before running:
PID=1234

# Trace a function in a binary
bpftrace -e 'uprobe:/usr/bin/python3.6:_PyEval_EvalCodeWithGlobals { printf("%s\n", comm); }'

# With PID
bpftrace -e 'uprobe:/path/to/binary:function { ... }' --pid "$PID"

Common uses:

• Trace application calls.
• Profile specific functions.
• Investigate library calls.

Choose the right attachment

Tracepoints are stable and safe. Kprobes and uprobes are powerful but riskier.

Knowledge check