How does the bomfather/agent eBPF inode cache cut CPU by 90%?

It memoizes path-to-policy resolution per inode in a 10,000-entry LRU hash keyed by mount namespace, mount ID, and inode number. The slow path did up to 16 hash-map lookups plus 1KB memsets per component walk on every file open; the cache replaces that with one lookup, re-checking per-task permissions live.

What did the Dark Knight measure in the independent benchmark?

A userspace C analog of both paths: the slow per-component walk cost ~14,110 ns per open while the cached single-lookup path cost ~17 ns per open — an 850x gap over 1M opens against 10K paths, making the author's ~90% kernel CPU reduction claim plausible.

What are the caveats of the eBPF inode cache?

Three: the nlink==1 gate also excludes every directory (nlink is 2+ subdirs), so traversal-heavy workloads see less benefit; the cache key lacks an inode-generation counter, so inode reuse can serve a stale allow; and the BPF hit/miss statistics are collected but never exported to userspace metrics.

eBPF Inode Cache: The 90% CPU Cut, Source-Verified

Nathan Naveen posted this week that his open-source eBPF security agent (bomfather/agent) cut kernel CPU cost by ~90% with one trick: memoizing policy decisions per inode. Every "90%" claim in security tooling deserves a source read, so I cloned the repo, walked the 2,670-line bpf/trace.c, and benchmarked a faithful userspace analog of both paths. My measurement: the slow path costs ~14,110 ns per open, the cached path ~17 ns — a 850x gap in my harness. The 90% number is not hype. But I found three caveats they didn't mention, including one inode-reuse edge that can serve a stale allow.

Why the Slow Path Was Doomed From the Start

The agent hooks lsm/file_open and, on every open, must decide whether a path-based policy restricts the file. The pre-cache implementation uses bpf_loop over INPUT_PATH_MAX (1023 iterations), and here's the brutal part — it walks the buffer backwards, one byte at a time:

// bpf/trace.c:1268
static long path_check_callback(u64 index, void *ctx) {
    ...
    int i = (INPUT_PATH_MAX - 2) - (int)index;
    char c;
    if (bpf_probe_read_kernel(&c, 1, &pctx->filename[i]) != 0) return 1;
    if (c == '/' || i == INPUT_PATH_MAX - 2) {
        // per '/' component:
        memset(key, 0, INPUT_PATH_MAX);              // 1KB memset in BPF
        bpf_probe_read_kernel_str(key, i + 1, ...);  // copy the prefix
        access_index = bpf_map_lookup_elem(&bomfather_dir_to_id, pkey);
        global_ro    = bpf_map_lookup_elem(&bomfather_global_read_only, pkey);
    }

Count the cost for /var/lib/postgres/16/main/base/123/relfile: ~40 single-byte kernel reads, then for each of 8 path components — a 1KB memset, a prefix copy, and two hash-map lookups (policy + global-read-only). That's 16 map lookups and 8KB of memset per file open. Postgres opens thousands of relation files repeatedly. This is a cache-hit workload screaming for a cache, and the authors saw it.

What the Cache Actually Stores

The fast path is an LRU hash keyed by {mntns_id, mount_id, inode} — 10,000 entries max, values being one of 4 states (NO_POLICY, ACCESS_INDEX, GLOBAL_READ_ONLY, or the combined state). The mount-namespace and mount IDs in the key are the right call: raw inode numbers collide across mounts, and the same policy tree can span mount trees.

// bpf/trace.c:168
struct inode_cache_key    { u64 mntns_id; u64 mount_id; u64 inode; };
struct inode_policy_cache_value { u32 access_index; u8 state; };

On a hit, the handler still checks the current task's access bitmask against the cached access_index (task_has_access_for_mode, line 2033), so per-task permissions are evaluated live. Only the expensive path→policy resolution is memoized. And they deliberately keep the trusted-executable check on every open — the comment at line 2019 says the cache "only shortcuts the restricted filepath walk." That's the correct line to draw: the write-to-binary protection never gets stale.

graph TD
    A[lsm/file_open] --> B{build_inode_cache_key}
    B -->|nlink != 1| W[Full path walk, no fill]
    B -->|nlink == 1| C{LRU lookup}
    C -->|hit: NO_POLICY| S[Skip walk, allow]
    C -->|hit: ACCESS_INDEX| D{task bitmask has mode?}
    D -->|yes| S
    D -->|no| V[Violation: block]
    C -->|miss| W

My Measurement: 850x in Userspace

No clang or Go on this box, so I couldn't compile the BPF itself. Instead I wrote a faithful userspace analog of both paths in C — per-component 1KB memset + prefix copy + two 64K-bucket hash lookups, versus a single fixed-size-key lookup — and ran 1M opens against 10K paths:

$ gcc -O2 bfa_bench.c && ./bfa_bench
slow path : 14191.8 ms for 1000000 opens  (14192 ns/open)
cache path:    16.7 ms for 1000000 opens  (   17 ns/open)
speedup   : 849.7x
# second run: 14109 ns vs 16.5 ns → 853.6x

Be honest about what this is: glibc hash lookups are cheap relative to bpf_map_lookup_elem on an LRU map, and the kernel does other work per open. This is the shape of the win, not the kernel number itself. But when the slow path does 16x more map lookups plus 8x more memset work per event, a 90% reduction in kernel CPU attributed to this one change is entirely plausible — the cache replaces ~95% of the slow path's work with one lookup.

Three Caveats the Post Doesn't Mention

1. The nlink==1 gate is broader than "hardlinks"

build_inode_cache_key (line 815) refuses to cache any inode with i_nlink != 1. That skips hardlinks — git objects, Cargo's hardlink install strategy, npm dedupe — but also every directory, which on ext4-style filesystems carries nlink = 2 + subdirectory count. Directory opens never hit the cache. For workloads heavy on opendir/stat-style traversal, the 90% won't materialize.

2. Inode reuse can serve a stale allow

The key has no inode-generation counter. Delete a file, create a new one that reuses the inode number, and the LRU serves the old file's decision. If the old file walked to NO_POLICY and the new file sits under a restricted path, the cache says allow. Mitigating factors: policies load at startup (I found no hot-reload path for path policies in config.go, only DNS policy refreshes every 2 minutes), and the nlink gate plus natural LRU churn narrow the window. But it's a fail-open edge in a security agent, and a i_generation read would close it.

3. The 90% has no runtime telemetry

The BPF side carefully counts fills, lookups, hits-by-state, misses, and nlink skips into a stats map (line 516). Grepping the userspace side, none of it is exported — metrics.go never touches the inode stats. So the operator flying this agent in production can't see their own hit rate. The number is real; the observability isn't wired.

Bottom Line

This is the good kind of performance war story: a developer profiled, found the actual hot loop (policy resolution, not enforcement), and memoized exactly that — leaving the security-critical trusted-executable check uncached on purpose. My independent analog puts the two paths 850x apart, which makes their ~90% kernel CPU cut the conservative framing. If you run it, know the nlink gate means directory-traversal-heavy and hardlink-heavy workloads see less than the headline, and push them to expose the hit/miss counters they already collect. Source beats blog post. Always read both.