OKF Agent Memory: I Audited the <300µs Claim

Yesterday a repo called OKF Agent Memory hit Hacker News (54 points when I cloned it, 119 stars, zero days old) with a benchmark table that reads like a victory lap: sub-300µs search in Go versus "Python / Vector DB Runtimes (Mem0, Letta)" at 150–800ms, plus an 80% token reduction from progressive disclosure. Git-native memory in plain Markdown, zero dependencies, embedded MCP server — that's exactly the kind of tool I want to exist. So I did what you're supposed to do with launch-day benchmarks: cloned it, read the source instead of the README, and measured.

The short version: the tool is real and decent, the <300µs claim is true only at toy scale — at 5,000 concepts the same search measures 354ms, which is inside the "Python vector DB" band the README mocks. And the same query returns a different ranking on every run. Receipts below.

The claims, measured on launch day

The pitch: agent memory as OKF v0.2 Markdown bundles (a real Google spec — I verified it exists upstream), searched with in-memory "BM25", validated as a link graph, served to agents over MCP. Setup on my box (Linux, Go 1.24):

git clone https://github.com/okf-memory/okf-agent-memory   # 124 KB, pure Go
go build ./... && go vet ./... && go test ./pkg/okf/...    # ok 0.146s
make build                                                 # bin/okf, 3.8 MB

Everything builds clean with zero dependencies — the go.mod has no requires, which is rarer than it should be and genuinely nice for an agent-loop tool. All tests pass. The MCP server works: I drove a JSON-RPC initialize + tools/list handshake over stdio and got back okf_search, okf_show, okf_validate, okf_create with proper input schemas. The memory footprint claim holds too: 11.9 MB RSS measured, under the advertised 15 MB. Cold start for a full okf validate process: 4.4ms p50 over 30 runs — a hair over the "<4ms" claim, within noise of honest.

The <300µs search claim collapses past 500 concepts

I built an in-process benchmark against their own library (not the CLI — no process overhead), ran 500 iterations per query, and then generated synthetic OKF bundles at 500 / 2,000 / 5,000 concepts to see where the curve goes:

graph LR
  Q[query] --> S[scan EVERY concept]
  S --> T[re-tokenize title+desc+body
on every search]
  S --> D[df via substring scan
over all docs]
  T --> R[sort by score]
  D --> R
  R --> O[no index. O concepts x doc size, per query]
            

The source explains it. pkg/okf/search.go builds no inverted index. Every search re-tokenizes the title, description, and full body of every concept, and document frequency is computed with strings.Contains — a substring scan over all documents, per term, per query. That's O(N × doc size) per call, forever. At 8 concepts nobody notices. At 2,000 concepts the p50 lands at 141ms — and here's the uncomfortable part: the README's own comparison table prices "Python / Vector DB Runtimes" at 150–800ms. Their tool crosses into the range it ridicules somewhere around 1,500–2,000 concepts, and hits 354ms at 5,000. A real vector DB with a prebuilt ANN index doesn't do that; latency stays roughly flat as the corpus grows. This thing gets slower every time your agent learns something.

While I was in there: the "BM25" label is generous. It's weighted TF-IDF (title 4.0, tags 3.5, description 2.5, id 2.0, body 1.0 capped at 5) multiplied by BM25's IDF formula. No k1, no document-length normalization, no saturation. Call it what it is — it's fine, but the benchmark table's authority rests on a word that isn't accurate.

Fifteen runs, fifteen rankings

Worse than the latency curve is this. Bundle.Concepts is a Go map, iterated in random order, then ranked with sort.Slice — which is unstable. When scores tie, ordering is whatever the map iterator coughed up. I ran one identical query against a 500-concept bundle fifteen times:

15 runs of: okf search "cache invalidation retry"
15 distinct top-3 orderings. Same query. Same index. Same second.

Think about what that means operationally. This is a memory system. Its whole value proposition is that your agent's recall of its own past is stable, auditable, reviewable in git diff. Instead, the same question asked twice can surface a different "most relevant" memory — different enough that position 2 and 3 reshuffle on every single run. For a tool whose pitch is determinism over black-box vector stores, that's the one bug I'd block a 1.0 on. The fix is two lines: store concepts in a slice, sort by (score, concept_id).

What actually holds up

Credit where it's due, because most of the design survives contact:

• The −80% token claim is honest arithmetic. Their monolith fixture is 11,504 chars; the retrieved concept is 2,044 — that's −82%, and their measured 3,034 → 603 tokens is −80.1%. But understand what it measures: loading one needed document instead of ten. The win comes from progressive disclosure as a practice, not from the tool. You can approximate it with grep and a well-organized docs folder.
• Retrieval quality is real. On their own benchmark premise ("implement enterprise encryption payload"), security/encryption-policy ranks #1 with score 30.18 vs 5.73 for the runner-up. The field weights do their job.
• The git-native format is the right idea. Provenance (sources), trust tiers (generated vs verified), staleness (stale_after) — the OKF v0.2 spec encodes exactly the metadata agent memory needs, in plain text a human can review.
• Zero-dependency Go, passing tests, working MCP server, 11.9 MB RSS. The engineering hygiene claims are all true.

The validation side is also genuinely fast — 1.3ms to check 5,000 concepts strict with drift detection, because the graph pass is indexed properly. Which proves the team knows how to build the fast path; the search path just never got one.

Bottom line

OKF Agent Memory is a good idea shipping with a benchmark table it can't cash. The <300µs headline is a property of 8-concept toy bundles, not of the tool; at a realistic few-thousand-concept corpus it's slower than the vector-DB strawman in its own README, and the nondeterministic ranking is a correctness bug, not a nitpick. None of this is fatal — an inverted index at load time and a stable sort are a weekend of work, and I'd rather audit a team that over-measured than one that didn't measure at all. Until then: use it for small, curated bundles where 8–50 concepts is the point. For project-scale memory, wait for the index. Launch-day benchmarks aren't evidence. They're a hypothesis with good marketing.