eBPF overview - in-kernel observability for sysadmins

eBPF (extended Berkeley Packet Filter) is a Linux kernel subsystem that runs sandboxed programs in the kernel without changing kernel source or loading kernel modules. It is the foundation of modern Linux observability.

What eBPF is

eBPF allows programs to:

• Run sandboxed in the kernel (verified by the verifier).
• Attach to events: tracepoints, kprobes, uprobes, network hooks, LSM.
• Read and modify kernel state (with limits).
• Communicate with userspace via maps.

eBPF programs are written in C or high-level languages like bpftrace. The kernel verifier ensures safety (no infinite loops, no out-of-bounds access).

Why eBPF matters

Before eBPF:

• Add a kernel module: risky, requires rebuild.
• Use /proc and /sys: limited, slow.
• Use strace or perf: powerful but heavyweight.

With eBPF:

• Run programs without kernel rebuild.
• Trace any kernel or user function.
• Custom programs for specific observability needs.
• Low overhead (programs run in kernel, no context switch).

Common eBPF tools

For a sysadmin, the most useful are bpftrace and bcc.

bpftrace basics

# Count syscalls by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Trace open() calls
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Histogram of read() latency
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @latency = hist(nsecs - @start[tid]);
    delete(@start[tid]);
}'

bpftrace is a high-level language that compiles to eBPF. Like awk for kernel tracing.

bcc tools

Pre-built tools in bcc package:

# biolatency: block I/O latency histogram
sudo biolatency

# biosnoop: per-I/O tracing
sudo biosnoop

# tcpconnect: trace TCP connections
sudo tcpconnect

# execsnoop: trace new processes
sudo execsnoop

# opensnoop: trace file opens
sudo opensnoop

These are the equivalent of strace for everything.

eBPF and sysadmin use cases

• Latency investigation: per-I/O latency, syscall latency, scheduler latency.
• Network troubleshooting: dropped packets, TCP retransmits, DNS latency.
• Security: detect suspicious syscalls, file access, or process behaviour.
• Capacity planning: identify which workloads are growing.

Caveats

• eBPF programs must be verified by the kernel. Complex programs may be rejected.
• High-overhead programs (e.g. trace every syscall) can slow the system.

Will these tools actually run here?

“Older kernels have limited support” is true and useless at 3am. Four things decide whether the commands in this lesson work on the host in front of you, and all four are checkable in a few seconds:

uname -r                                # >= 4.9 for anything useful; >= 5.x for comfort

ls /sys/kernel/btf/vmlinux 2>/dev/null \
  && echo "BTF present - CO-RE works" \
  || echo "No BTF - tools need kernel headers matching $(uname -r)"

sysctl kernel.unprivileged_bpf_disabled # 1 or 2 => root or CAP_BPF required

bpftrace --info 2>/dev/null | head -20  # feature probe: what this kernel actually supports

BTF and CO-RE are the ones people are not expecting. BTF (CONFIG_DEBUG_INFO_BTF) is kernel type information shipped inside the kernel itself, and CO-RE (“compile once, run everywhere”) uses it so a prebuilt tool binary works across kernel versions. Without BTF, bcc tools compile a program against kernel headers at runtime — so the host needs the linux-headers-$(uname -r) package installed, needs clang, and pays several seconds of compile time on every invocation. On a host where you cannot install a compiler, no BTF means no bcc tools at all.

Privilege. Debian and Ubuntu ship kernel.unprivileged_bpf_disabled=1 by default, so every tool here needs root, or CAP_BPF plus CAP_PERFMON on newer kernels. Do not plan a monitoring agent around unprivileged BPF; it is off on the distributions you will actually meet.

Distribution reality, since the version number alone does not tell you:

The 3.10 row is why the “<4.x” rule of thumb misleads: RHEL backports enough that its version number tells you almost nothing either way. Probe with bpftrace --info; do not infer from uname -r alone.

Knowledge check