Retrieval Surface-Form Bias: 0.0% Hit@1 on Structure
A retrieval system that returns a query's exact lexical twin instead of the structurally correct document isn't retrieving — it's matching spelling. That's the thesis of "Retrieved but not ranked," a new MIT preprint that pulls surface form and meaning apart on purpose and measures what production embeddings do when they conflict. The headline number is brutal: on disguised competition math, strict Hit@1 is 0.0% for both embedders tested, with a bootstrap 95% CI of [0.0, 0.0]. The correct item sits in the top 10 nearly every time — the system can find structure, it just cannot rank it.
The Problem: Embeddings Rank Words, Not Structure
Vector retrieval assumes semantic neighborhoods. But "semantic" is doing a lot of work: when two items share structure but not vocabulary — the same math problem rewritten, the same task solved with different objects — embeddings have to ignore surface form and track the underlying equivalence. Rashid and Kellis test this in two unrelated domains under one protocol: competition mathematics (MathNet-Retrieve: 500 queries over a 117,088-item corpus) and embodied-agent trajectories (ALFWorld-derived: 118 queries over 336 trajectories). In math, query problems are rewritten in escalating "disguise tiers" so the structurally identical problem is lexically distant. In trajectories, the gold trajectory must solve the same task using different objects and receptacles. Same test, two flavors of structure.
Method: Disguise Tiers, Then Two Reranker Interventions
The protocol is clean: embed every query, retrieve with off-the-shelf production embedders, and compare the winner against gold at each disguise tier. Then two interventions. First, a lexical reranker control (BM25-style) re-ranks the retrieved set by surface overlap. Second, an LLM reranker judges which candidate structurally matches the query. A paired downstream experiment (210 queries, graders at 96–99% agreement) feeds the top hit to a solver, with oracle and adversarially bad retrieval as endpoints.
Results: 0.0% Hit@1, and What Rerankers Actually Recover
The failure is structural, not statistical. In math at the heaviest disguise tier, both production embedders score 0.0% strict Hit@1 while gold is almost always in the top 10. In 95.2–99.8% of misses, the winning item is more lexically similar to the query than the correct answer — the retriever isn't confused, it's confidently wrong about the wrong dimension.
Trajectories are worse. When gold must involve a different object, retrieval lands at or near hypergeometric chance; once gold must differ in both object and receptacle, all three embedders score below chance. The retriever anchors on literal tokens ("apple" vs "cupboard"), not task structure.
The interventions get interesting here:
- The lexical reranker's sign is a diagnostic. It hurts math but helps trajectories, closing 26–36% of the gap (CIs excluding zero). A lexical reranker that helps means your benchmark's surface variation is incidental; one that hurts means adversarial. Free litmus test.
- LLM rerankers recover 5–63% of the math gap and 43–76% of the trajectory gap, replicating across all three judges (21/21 cells positive) — but effect sizes vary wildly by judge.
- Part of the recovery is memorization. Math gains concentrate on well-known competitions (+19.8 points, CI [+6.7, +33.2], one of six cells). The reranker "recognizes" famous problems, not structure.
- The downstream test flips the script. A solver fed oracle retrieval is indistinguishable from one fed adversarially bad retrieval (McNemar p = 0.678). Its 69.5% zero-shot accuracy is 97–100% on finished answers — a truncation proxy. Retrieval quality didn't matter because the benchmark wasn't testing retrieval at all.
Limitations: Synthetic Disguise and a Proxy Solver
The disguise protocol is constructed, not organic — real-world surface variation is messier and usually correlated with meaning. "Structure" here is narrow: math equivalence classes and object/receptacle slots, not the full lattice of code or theorem structure. Only three production embedders are tested. The downstream arm uses a single solver, and the memorization confound means LLM-reranker gains on public data overstate structural understanding. The trajectory corpus is small (336 trajectories), and hypergeometric chance is a harsh bar — most production queries have lexical signal to lean on.
Why Builders Should Care
Three takeaways for anyone shipping RAG over math, code, or agent traces. One: don't trust cosine similarity for structure-sensitive retrieval — a 0.0% Hit@1 tier means your retriever passes the buck to a reranker that may just be memorizing. Two: run the lexical-reranker sign test on your own eval; it tells you in minutes whether your benchmark's rewrites are adversarial or incidental, and whether your retrieval pipeline is even being exercised. Three: check whether your downstream task actually uses what you retrieve — a 69.5% accuracy that barely moves under oracle-vs-adversarial retrieval is an eval that has nothing to say about retrieval quality. If you're building agent memory or code search, treat these numbers as the floor: structure is the next retrieval frontier precisely because it's the one embeddings can't see.