Running eBPF safely - kernel requirements, permissions and lockdown

eBPF tooling fails in a distinctive way: not with “not supported” but with an error about a file, a capability or a symbol, several layers away from the thing that is actually missing. During an incident that is an expensive place to start learning.

This lesson is the pre-flight check. Run it on a representative host from each of your platforms, once, and record the answer - so that when you need bpftrace at 03:00 you already know whether it will work.

Kernel configuration

grep -E '^CONFIG_(BPF_SYSCALL|BPF_JIT|BPF_EVENTS|DEBUG_INFO_BTF|KPROBES|UPROBES|FTRACE_SYSCALLS)=' \
  "/boot/config-$(uname -r)"

CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_KPROBES=y
CONFIG_UPROBES=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BPF_EVENTS=y

What each one buys you:

If /boot/config-$(uname -r) does not exist, the same data may be available from the kernel itself:

sudo modprobe configs
zgrep -E '^CONFIG_BPF_SYSCALL=' /proc/config.gz

BTF is the one that matters most

BTF (BPF Type Format) is the kernel’s own description of its data structures, compiled into the running kernel. It is what lets a tool read task_struct fields correctly without kernel headers, and it is the basis of CO-RE - compile once, run everywhere:

ls -l /sys/kernel/btf/vmlinux

-r--r--r-- 1 root root 7061954 Aug 11 06:01 /sys/kernel/btf/vmlinux

That file present means modern bpftrace and CO-RE-based bcc tools will work with no kernel headers installed. Its absence is the single most common reason eBPF tooling does not run on an older or custom kernel, and it produces errors like:

ERROR: Cannot parse kernel BTF: No such file or directory

or, from a bcc tool falling back to the header-based path:

Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m

The fix on a distribution kernel is to install the matching headers package. On a custom kernel it is to rebuild with CONFIG_DEBUG_INFO_BTF=y, which is a maintainer conversation, not something to attempt during an outage.

Distribution kernels have shipped BTF since roughly Ubuntu 20.10, RHEL 8.2 and Debian 11. bpftrace itself needs kernel 4.9 for basic operation; assume 5.x before expecting anything comfortable.

Ask the kernel what it supports

bpftool feature probe is the authoritative answer, because it probes rather than infers:

sudo bpftool feature probe kernel | head -20

Scanning system configuration...
bpf() syscall for unprivileged users is disabled
JIT compiler is enabled
JIT compiler hardening is disabled
JIT compiler kallsyms exports are disabled
Global memory limit for JIT compiler for unprivileged users is 264241152 bytes
CONFIG_BPF is set to y
CONFIG_BPF_SYSCALL is set to y
CONFIG_HAVE_EBPF_JIT is set to y
CONFIG_BPF_JIT is set to y
CONFIG_BPF_EVENTS is set to y
...
Scanning eBPF program types...
eBPF program_type socket_filter is available
eBPF program_type kprobe is available
eBPF program_type tracepoint is available

Run it as an unprivileged user and you must say so, or the results are misleading:

bpftool feature probe kernel unprivileged

The man page is explicit about why: “Unprivileged users MUST use the unprivileged keyword: This is to avoid misdetection if bpftool is inadvertently run as non-root.” Without it, a non-root run reports things as unavailable that are merely unavailable to you.

bpftrace has its own summary:

sudo bpftrace --info

Permissions

This is where most failures live. Three separate mechanisms gate eBPF, and they produce different errors.

The capabilities

Before kernel 5.8, everything interesting required CAP_SYS_ADMIN - effectively root. Kernel 5.8 split it:

An observability tool typically needs CAP_BPF and CAP_PERFMON together. CAP_BPF alone loads a program that cannot attach to anything.

The sysctls

sysctl kernel.unprivileged_bpf_disabled net.core.bpf_jit_enable \
       kernel.perf_event_paranoid

kernel.unprivileged_bpf_disabled = 2
net.core.bpf_jit_enable = 1
kernel.perf_event_paranoid = 4

kernel.unprivileged_bpf_disabled values:

• 0 - unprivileged bpf() allowed.
• 1 - disabled permanently. This value cannot be changed back without a reboot, by design.
• 2 - disabled, but the setting can still be changed at runtime. This is the Debian and Ubuntu default, corresponding to CONFIG_BPF_UNPRIV_DEFAULT_OFF=y.

kernel.perf_event_paranoid gates perf-event attachment separately, and Debian and Ubuntu ship 4, which blocks unprivileged access entirely. Both must be satisfied.

Granting capabilities to a long-running agent

For a monitoring agent that must not run as root:

sudo setcap cap_bpf,cap_perfmon,cap_dac_read_search+ep /usr/local/bin/my-agent
getcap /usr/local/bin/my-agent

/usr/local/bin/my-agent cap_dac_read_search,cap_perfmon,cap_bpf=ep

Or, more manageably, as unit properties so the grant is declarative and shows up in configuration review:

[Service]
ExecStart=/usr/local/bin/my-agent
AmbientCapabilities=CAP_BPF CAP_PERFMON CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_BPF CAP_PERFMON CAP_DAC_READ_SEARCH
NoNewPrivileges=yes

Note that CAP_BPF and CAP_PERFMON are close to root in practice - a process holding them can read arbitrary kernel memory through tracing. Treat granting them as a privileged change, not a convenience.

Lockdown

Secure Boot on many distributions enables the kernel lockdown LSM, and lockdown blocks exactly the mechanisms tracing depends on:

cat /sys/kernel/security/lockdown

[none] integrity confidentiality

The bracketed value is active. The three modes:

• none - no restriction. eBPF tracing works normally.
• integrity - blocks modifying the running kernel. kprobes and bpf_probe_write_user are restricted.
• confidentiality - blocks reading kernel memory too. bpf_probe_read_kernel is refused, which kills essentially all kernel observability.

mokutil --sb-state

A pre-flight script

#!/bin/bash
# eBPF readiness check. Run as root for a complete answer.
set -uo pipefail

echo "== kernel: $(uname -r)"

echo "== config"
grep -E '^CONFIG_(BPF_SYSCALL|BPF_JIT|BPF_EVENTS|DEBUG_INFO_BTF|KPROBES|UPROBES)=' \
  "/boot/config-$(uname -r)" 2>/dev/null || echo "  no /boot/config file"

echo "== BTF"
if [ -r /sys/kernel/btf/vmlinux ]; then
  echo "  present: $(stat -c %s /sys/kernel/btf/vmlinux) bytes"
else
  echo "  MISSING - CO-RE tools will need kernel headers"
fi

echo "== sysctls"
sysctl -n kernel.unprivileged_bpf_disabled net.core.bpf_jit_enable \
          kernel.perf_event_paranoid 2>/dev/null

echo "== lockdown"
cat /sys/kernel/security/lockdown 2>/dev/null || echo "  lockdown LSM not present"

echo "== can we actually attach?"
if bpftrace --dry-run -e 'tracepoint:sched:sched_process_exec { exit(); }' \
     >/dev/null 2>&1; then
  echo "  yes"
else
  echo "  NO - run as root and re-check"
fi

--dry-run compiles and attaches the probes and then terminates immediately, which makes it a real test of whether attachment works without leaving anything running.

Run this on one host per platform - each distribution, each kernel line, each cloud image, and inside a container if you expect to trace from one - and put the results in the runbook next to the bpftrace one-liners. That table is worth more at 03:00 than any individual command.

Knowledge check