eBPF — an introduction for container operators · Docker & Containers

eBPF lets you load a small sandboxed program into the running kernel, attach it to a hook, and have it observe or act on events — without a kernel module, without a reboot, and without modifying the application being observed.

For containers that is unusually valuable, because a container is a process on a kernel you already control. You do not need to get a tracing agent into the image, negotiate with the application team, or restart anything.

This lesson is an introduction. It is deliberately honest about what it costs to use, because most eBPF material is not.

What an eBPF program is

A user-space loader compiles a program to eBPF bytecode and submits it via the bpf(2) syscall. Before the kernel will run it, the verifier proves statically that the program:

terminates — no unbounded loops;
reads and writes only memory it is permitted to touch, with every pointer dereference preceded by a bounds check the verifier can see;
calls only the helper functions allowed for its program type;
stays within instruction and stack limits.

If the proof succeeds, the program is JIT-compiled to native code and attached to a hook. It then runs in kernel context, at kernel speed, with a static guarantee that it cannot crash the kernel or read arbitrary memory. That guarantee is what makes it acceptable to run on a production host, and it is the entire reason eBPF exists rather than everyone writing kernel modules.

Programs communicate with user space through maps — typed key-value structures the kernel program writes and the loader reads. Attach points include kprobes (any kernel function), tracepoints (stable kernel events), uprobes (user-space functions), perf events, network tc and XDP, and the LSM hooks used by security tooling.

The requirements, stated honestly

This is where eBPF articles stop being useful

Blog posts show a one-liner and a beautiful histogram. They do not mention that the one-liner fails on the host you actually have.

Kernel version. The bpf syscall landed in 3.18, kprobes in 4.1, and the features modern tooling assumes arrived much later. The practical baseline for current bpftrace and BCC tools is kernel 5.8 or later, and RHEL and its derivatives backport heavily, so a RHEL 8 kernel numbered 4.18 may support far more than the number suggests. Check features, not version numbers.

BTF. Modern tools use BTF — kernel type information compiled into the kernel image — to relocate struct field offsets at load time (CO-RE, compile-once run-everywhere). Without it, tools must compile against matching kernel headers on the host, which means shipping a compiler and the exact kernel-devel package to production. Check for /sys/kernel/btf/vmlinux; if it is missing, the kernel was built without CONFIG_DEBUG_INFO_BTF and your tooling options narrow sharply.

Privileges. Loading tracing programs requires root, or CAP_BPF plus CAP_PERFMON on kernels 5.8 and later where those capabilities were split out of CAP_SYS_ADMIN. Network programs additionally want CAP_NET_ADMIN. Many distributions also set kernel.unprivileged_bpf_disabled=2, which blocks unprivileged loading outright.

Lockdown. A host with Secure Boot and kernel lockdown in confidentiality mode blocks kprobes and BPF program types that can read kernel memory. The error is a permission denial that looks like a capability problem and is not.

Read-only / Safecan this host run eBPF tooling?— Five checks. Run them before you plan any eBPF work on a fleet.

echo "kernel:    $(uname -r)"

# BTF: the difference between portable tools and needing kernel headers.
[ -r /sys/kernel/btf/vmlinux ] \
&& echo 'BTF:       present' \
|| echo 'BTF:       MISSING - CO-RE tools will not load'

# Unprivileged loading policy. 2 means disabled and not re-enableable at runtime.
echo "unpriv_bpf_disabled: $(sysctl -n kernel.unprivileged_bpf_disabled 2>/dev/null || echo 'n/a')"

# Kernel lockdown blocks several program types under Secure Boot.
echo "lockdown:  $(cat /sys/kernel/security/lockdown 2>/dev/null || echo 'not enabled')"

# Anything already loaded?
sudo bpftool prog show 2>/dev/null | head -10 || echo 'bpftool not installed'

The structural problem: eBPF does not know what a container is

This is the part that matters most for this course and the part most introductions skip entirely.

The kernel has no concept of a container. eBPF programs see PIDs in the initial PID namespace, so a probe that reports pid 48213 gives you a number that does not exist inside the container and does not appear in docker ps. Correlating an observation to a container is your problem.

The reliable handle is the cgroup ID, because cgroups are exactly the kernel object Docker creates per container.

Read-only / Saferesolve a container to its cgroup path— Read the cgroup from the process rather than constructing it; the layout depends on the daemon cgroup driver.

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(sed -n 's/^0:://p' /proc/"$PID"/cgroup)
CGPATH="/sys/fs/cgroup$CGROUP"

echo "container: $CONTAINER"
echo "host pid:  $PID"
echo "cgroup:    $CGPATH"

bpftrace provides both halves of the correlation: a cgroup builtin holding the cgroup ID of the process that triggered the probe, and a cgroupid() function that resolves a cgroup v2 path to that ID at compile time.

Service impact possibletrace one container only— Every exec inside a single container, and nothing from the rest of the host. Probes add overhead; scope them and keep the run short.

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGPATH="/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/"$PID"/cgroup)"

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_execve
/cgroup == cgroupid(str($1))/
{
printf("%-8d %-16s %s\n", pid, comm, str(args->filename));
}' "$CGPATH"

Read-only / Safescoped exec trace— Only processes in the target container appear. The PIDs are host PIDs.

$ sudo bpftrace -e '...' /sys/fs/cgroup/system.slice/docker-....scope

Attaching 1 probe...
48213    sh               /bin/sh
48219    curl             /usr/bin/curl
48244    sh               /bin/sh
48245    nc               /usr/bin/nc

Illustrative output

A shell and a nc spawning inside a service container is the classic post-exploitation signature, and this is roughly what Falco and Tracee productise: the same probes, plus rule evaluation, plus the container metadata lookup done for you.

Why the cgroup filter belongs in the kernel program

You could pipe an unfiltered trace through grep. Do not, and the reason is about cost rather than tidiness.

An eBPF program attached to sys_enter_execve runs in the context of every execve on the host, in kernel space, before the syscall proceeds. If the filter is in the program, a non-matching event costs one comparison against a 64-bit ID and returns. Nothing is copied, nothing is written to the perf ring buffer, and user space never wakes up.

If the filter is in grep, every event on the host is formatted, written to the ring buffer, read by bpftrace, formatted again, and written to a pipe. On a busy host that is a large amount of work per event and a real, measurable CPU cost — and it scales with total host activity rather than with the container you care about.

This is the general shape of eBPF performance discipline: filter in the kernel, aggregate in the kernel (with maps and histograms), and move as little as possible to user space. It is also why bpftrace aggregations like @hist[comm] = hist(latency) are cheap while printf on a hot path is not.

What people actually use it for

Tool	What it does	Attach points
`bpftrace`	Ad-hoc one-liners and short scripts. The exploratory tool.	kprobes, tracepoints, uprobes
BCC tools (`execsnoop`, `opensnoop`, `biolatency`, `tcplife`)	Prebuilt, focused answers to common questions	mixed
Cilium	Container networking and policy in eBPF instead of iptables	`tc`, XDP, socket
Falco	Runtime security rules over syscall streams	tracepoints, modern BPF driver
Tracee	Runtime security and forensics events	tracepoints, LSM
Pixie, Parca, Pyroscope	Continuous profiling and auto-instrumentation	perf events, uprobes

Three one-liners worth knowing, each answering a question that is genuinely hard otherwise:

Service impact possiblethree useful probes— Run each for a bounded period on a host you are allowed to trace. Aggregation happens in-kernel, so overhead stays low.

# 1. Block I/O latency as a histogram, host-wide. Answers 'is the disk slow?'
sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete
/@start[args->dev, args->sector]/
{
@usecs = hist((nsecs - @start[args->dev, args->sector]) / 1000);
delete(@start[args->dev, args->sector]);
}
interval:s:30 { exit(); }'

# 2. Which files a container opens, by cgroup. Answers 'what config is it reading?'
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/cgroup == cgroupid(str($1))/
{ @[str(args->filename)] = count(); }
interval:s:20 { exit(); }' /sys/fs/cgroup/system.slice/docker-CONTAINER_ID.scope

# 3. Count syscalls by name for one cgroup. The seccomp-profile starting point.
sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
/cgroup == cgroupid(str($1))/
{ @[ksym(*(kaddr("sys_call_table") + args->id * 8))] = count(); }
interval:s:20 { exit(); }' /sys/fs/cgroup/system.slice/docker-CONTAINER_ID.scope

The third one is worth flagging: reading sys_call_table is architecture- and kernel-specific and will not work everywhere. It is included because syscall enumeration is the real starting point for building a seccomp profile, and because it illustrates the general fragility of kprobe-style tracing — you are attaching to kernel internals that carry no stability guarantee.

Run it on the host, not inside a container

The correct place for eBPF tooling is the host. It is host-level instrumentation; it observes the kernel, which is shared.

Containerised eBPF tooling exists, and making it work requires a container with CAP_BPF, CAP_PERFMON, CAP_SYS_ADMIN on older kernels, a mount of /sys/kernel/debug, host PID namespace, and access to BTF or kernel headers. A container with that set is not sandboxed in any meaningful sense — it can load code into the kernel that observes and, for some program types, modifies behaviour for every process on the machine.

That is fine for a deliberately deployed node agent such as Falco or Cilium, which is a host-level component that happens to be packaged as a container and should be reviewed, versioned and access-controlled as such. It is not fine as a convenience for a developer who wants to run bpftrace and finds that adding a few capabilities makes the error go away. If a workload needs these capabilities to do its job, it is not a workload.

When eBPF is the wrong tool

Reach for something simpler when:

A cgroup file already answers it. Throttling, memory usage, per-container I/O and OOM kills are all in /sys/fs/cgroup. No tracing required, no overhead, no privileges beyond reading a file.
The application can be asked. An HTTP request rate is a metrics endpoint, not a uprobe. Instrumented is better than inferred whenever it exists.
You need history. eBPF observes live events. It cannot tell you what happened at 02:00 last night unless something was already running and recording.
The host cannot support it. No BTF, locked-down kernel, or a managed platform that does not grant the capabilities. Find out before you build a plan around it.
The probe cost is not justified. A printf on a high-frequency tracepoint on a loaded host is real overhead. Aggregate in the kernel, scope by cgroup, and bound the run with an interval and exit().

What happens when the verifier rejects the program?

The bpf syscall returns -EACCES or -EINVAL, the program never loads, and the loader prints a verifier log that is long, precise, and initially unreadable — a full instruction-by-instruction trace of the states the verifier explored before it gave up.

Nothing is left attached and nothing is harmed. Verifier rejection is the sandbox working; a rejected program is one the kernel could not prove safe, not one that would definitely have crashed.

The common causes, in the order you should suspect them:

A loop the verifier cannot bound. Older kernels reject any backward jump. Kernel 5.3 and later support bounded loops, but the bound must be provable. Use #pragma unroll or a fixed iteration count.

A pointer dereference without a preceding bounds check. Every access to packet data or a map value needs an explicit comparison the verifier can follow, immediately before the access and in the same branch.

A helper not permitted for that program type. Program types have distinct allowed helper sets. A helper that is fine in a tracepoint program may be rejected in an XDP one.

Instruction or complexity limits. Very large programs, or ones with many branches, exhaust the verifier’s exploration budget. Split into several programs and use tail calls.

Read the last few lines of the verifier log first, not the first. The failure is at the end; everything above it is the successful exploration that preceded it.

Knowledge check

Why this matters

eBPF is the answer to a specific and recurring shape of question: how do I observe what this container is doing without modifying it, without restarting it, and without trusting what it reports about itself? Nothing else answers that. A sidecar needs a deployment, an APM agent needs the application to cooperate, and strace costs an order of magnitude in overhead and requires CAP_SYS_PTRACE on the target.

You do not need to write eBPF programs to benefit. Cilium, Falco, Tracee, Pixie and the continuous profilers are all eBPF underneath, and knowing that tells you what they can and cannot see, why they need the privileges they ask for, and why they care so much about your kernel version.

What separates useful adoption from a stalled project is checking the prerequisites first. Kernel version, BTF, the unprivileged loading sysctl, lockdown — four commands, run once per host image, before anyone builds a plan around a tool that turns out not to load. And when the answer is already in a cgroup file, read the cgroup file. eBPF is the tool for questions nothing simpler can answer, and there are fewer of those than the enthusiasm suggests.