TL;DR — eBPF lets you attach sandboxed programs to kernel hooks — syscalls, network events, tracepoints — without modifying kernel source or loading traditional kernel modules. A built-in verifier guarantees safety, a JIT compiler delivers near-native speed, and maps provide the bridge between kernel and user space. The result is a paradigm shift in how we build observability, networking, and security tooling on Linux.

The Problem eBPF Solves

For decades, extending the Linux kernel meant one of two unpleasant choices: patch the kernel source and rebuild, or load a kernel module (LKM) with all the attendant risks of memory corruption, privilege escalation, and system instability. Neither option suited the needs of modern, dynamic infrastructure. Network packet filtering was the original use case — the classic tcpdump filter ran through the original BPF interpreter, which was slow and limited to 16 registers and a small instruction set.

The landscape changed dramatically with the introduction of eBPF (extended BPF) in kernel 3.18 (2015) and its maturation through kernel 5.x and beyond. Today, eBPF underpins Cilium’s networking layer, Falco’s runtime security monitoring, and the observability pipelines behind companies running thousands of nodes. The technology has become foundational infrastructure in its own right.

Architecture: How eBPF Actually Works

At its core, eBPF is a safe, in-kernel virtual machine. When a user-space process wants to run an eBPF program, it follows a well-defined pipeline:

  1. Load: The program bytecode is passed to the kernel via the bpf() syscall.
  2. Verify: A static verifier analyzes the bytecode for safety — no infinite loops, no out-of-bounds memory access, no kernel pointer leaks.
  3. JIT Compile: If verification passes, the bytecode is compiled to native machine code via a Just-In-Time compiler.
  4. Attach: The program is hooked to a kernel tracepoint, kprobe, perf event, or XDP hook.
  5. Execute: When the hook fires, the eBPF program runs in kernel context with restricted access.
  6. Communicate: Results are passed back to user space through eBPF maps — typed, kernel-managed hash tables, arrays, and ring buffers.

The Verifier: Gatekeeper of Safety

The verifier is the heart of what makes eBPF trustworthy. It performs a depth-first walk of every possible execution path, enforcing a strict set of invariants:

  • Termination: Every branch must eventually reach an exit instruction. Loops are permitted only if the verifier can prove they terminate.
  • Type safety: Every memory access must be validated against the types of pointers carried in registers.
  • Bounds checking: Offsets into maps and stack memory must be proven within bounds at every instruction.
  • Kernel pointer isolation: A pointer received from the kernel cannot be passed to a user-space map without explicit conversion.
# Conceptual illustration of eBPF program loading flow
# (Actual eBPF programs use a restricted instruction set, not Python)

def load_ebpf_program(bytecode):
    # Step 1: Validate instruction count and format
    assert len(bytecode) < 4096, "Program exceeds instruction limit"

    # Step 2: Run static verifier
    if not verifier.analyze(bytecode):
        raise VerifierError("Unsafe memory access or infinite loop detected")

    # Step 3: JIT compile to native code
    native_code = jit_compiler.compile(bytecode)

    # Step 4: Register with kernel
    prog_id = bpf_syscall.BPF_PROG_LOAD(native_code)
    return prog_id

The verifier rejects roughly 30–40% of first-time submissions during development, which is why tools like bpftool and the BCC library provide detailed error messages pointing to the exact instruction that failed validation.

Maps: The Kernel ↔ User-Space Bridge

eBPF maps are typed data structures created in kernel space and accessible from both kernel and user space. They are the primary mechanism for exchanging data between eBPF programs and the outside world. The main map types include:

  • Hash maps (BPF_MAP_TYPE_HASH): Key-value stores with O(1) lookup, ideal for per-connection or per-flow state.
  • Arrays (BPF_MAP_TYPE_ARRAY): Fixed-size indexed stores, used when you need deterministic access by integer key.
  • Ring buffers (BPF_MAP_TYPE_RINGBUF): High-throughput circular buffers introduced in kernel 5.8, replacing the older perf ring buffer for streaming events to user space.
  • Per-CPU hash maps: Eliminate lock contention by giving each CPU core its own hash table instance.
  • LRU hashes: Automatically evict least-recently-used entries when the map reaches capacity — critical for production systems that cannot unboundedly grow memory.
// Example: Defining an eBPF map in C (libbpf style)
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, __u32);
    __type(value, __u64);
} conn_counts SEC(".maps");

Maps are reference-counted and persist independently of the eBPF program that created them. This means you can load a program, update its map from user space, and tear down the program without losing accumulated telemetry.

JIT Compilation: Near-Native Performance

The original BPF interpreter executed each 8-byte instruction through a switch statement in the kernel, incurring significant overhead. eBPF introduced a JIT compiler that translates the virtual instruction set into native x86_64 or ARM64 instructions. The JIT is enabled by default on all major architectures and provides roughly 5–10x speedup over the interpreter.

The JIT compiler is architecture-specific but follows a consistent pattern:

  1. Instruction expansion: Complex eBPF instructions (like BPF_JMP | BPF_CALL) are translated into sequences of native branches and calls.
  2. Register allocation: eBPF’s 10-register model maps efficiently onto the native ABI’s register file.
  3. Prologue/epilogue generation: Stack frame setup and return value handling are inserted automatically.
  4. Hardening: The compiled code runs with SMEP (Supervisor Mode Execution Prevention) and SMAP (Supervisor Mode Access Prevention) on modern kernels, preventing the JIT-compiled code from accessing user-space memory.

The JIT code is stored in kernel memory and is visible through /sys/fs/bpf/ and bpftool prog dump jited. You can verify JIT is active by checking /proc/sys/net/core/bpf_jit_enable, which should read 1 on production systems.

Production Use Cases

Networking: XDP and Cilium

The most performance-critical eBPF application today is in networking. XDP (eXpress Data Path) attaches eBPF programs at the earliest point in the network stack — the driver level — before any kernel networking overhead is incurred. This enables packet processing at line rate on 100Gbps+ interfaces.

Cilium, the CNCF project built on eBPF, replaces traditional iptables rules with eBPF programs hooked at the socket layer. The performance difference is substantial:

ApproachConnections/secRules Update Latency
iptables~50,000O(n) scan of all rules
eBPF (Cilium)~500,000O(1) map lookup

Cilium’s datapath compiles CiliumNetworkPolicy CRDs into eBPF bytecode at runtime, eliminating the need for userspace proxies in many service-mesh configurations.

Observability: bpftrace and BCC

Writing a kernel probe traditionally required printk, ftrace, or the overhead of SystemTap. eBPF tools like bpftrace and the BCC (BPF Compiler Collection) library make kernel instrumentation as simple as a one-liner:

# Trace every open() syscall and print the process name and filename
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

Under the hood, bpftrace compiles this script into an eBPF program attached to the sys_enter_openat tracepoint, with a perf ring buffer map streaming results back to the terminal. The entire workflow — write, compile, attach, observe — takes seconds, with zero kernel module loading.

BCC goes further by providing Python and Lua bindings that let you write the user-space orchestration in a familiar language while the performance-critical logic runs as eBPF:

from bcc import BPF

# Load an eBPF program from source
b = BPF(src_file="tcpv4.py")

# Attach to a kernel function
b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect")

# Read output from a perf buffer
b["events"].open_perf_buffer(print_event)
while True:
    b.perf_buffer_poll()

Security: Falco and Tracee

Runtime security tools leverage eBPF to monitor system calls, file access, and network activity with minimal overhead. Falco uses eBPF probes to detect anomalous behavior — a container spawning a shell, a process writing to /etc/shadow, an unexpected network connection.

Aqua Security’s Tracee takes a different approach, using eBPF to capture full system call arguments, return values, and stack traces, then feeding them to a user-space analysis engine. The eBPF component runs with near-zero CPU overhead because it leverages the kernel’s existing tracepoint infrastructure rather than interposing on every syscall through LD_PRELOAD tricks.

Patterns in Production

When integrating eBPF into a production system, several patterns have emerged from battle-tested deployments:

  1. Separate compilation from attachment: Compile eBPF programs during CI/CD, validate them in a staging environment, and deploy the compiled .o files alongside your application. This avoids runtime compilation failures on nodes with different kernel versions.

  2. Use libbpf, not BCC, for production: BCC is excellent for prototyping and debugging, but libbpf (the upstream library used by Cilium and Falco) has lower overhead, no Python dependency, and better kernel version compatibility through the CO-RE (Compile Once – Run Everywhere) feature.

  3. Design maps for eviction: Production eBPF programs must handle memory pressure. Use LRU or LRU-per-CPU maps rather than plain hash maps to prevent unbounded growth. Set max_entries conservatively and monitor map utilization through bpftool map show.

  4. Pin maps for state persistence: Use bpftool pin to persist maps to /sys/fs/bpf/. This allows you to restart the user-space daemon without losing accumulated state — a critical pattern for systems that track connection counts, rate limits, or distributed caches.

  5. Test the verifier early: The verifier is strict and unforgiving. Integrate bpftool prog load into your CI pipeline to catch rejection errors before they reach production. Kernel version differences can change verifier behavior, so test against your minimum supported kernel.

Limitations and Honest Tradeoffs

eBPF is powerful, but it is not a universal solution. Consider these constraints:

  • Instruction limit: Programs are capped at 4096 instructions (raised from 1024 in earlier kernels). Complex logic must be decomposed or pushed to user space.
  • No loops (without proof): The verifier rejects unbounded loops. You can use unrolled loops with a fixed iteration count, but dynamic iteration requires creative patterns like tail-call chains.
  • Kernel version dependency: eBPF program types and helper functions vary across kernel versions. CO-RE mitigates this by allowing the compiler to generate relocation metadata, but it adds complexity.
  • Debugging is hard: When an eBPF program crashes, it typically kills the entire process that loaded it. Tools like bpftool prog tracelog and bpftrace’s --unsafe mode help, but the debugging experience still lags behind user-space development.

Key Takeaways

  • eBPF provides a safe, verified, JIT-compiled execution environment inside the kernel — eliminating the need for kernel modules while delivering near-native performance.
  • The verifier is the cornerstone of eBPF’s safety model, statically proving program correctness before any code runs in kernel context.
  • eBPF maps are the primary data exchange mechanism between kernel and user space, with typed structures like hash maps, ring buffers, and LRU caches.
  • Production-grade eBPF development favors libbpf and CO-RE over BCC for lower overhead and better kernel compatibility.
  • XDP and Cilium have made eBPF the standard for high-performance Linux networking, replacing iptables in many large-scale deployments.
  • Map eviction strategies, pinning, and verifier testing are essential production concerns that are often overlooked in tutorials.

Further Reading