Code Embedding Retrieval: Exec@1 = 0.331 vs Buggy Clones

Every coding agent quietly depends on a promise: that embedding-based code retrieval surfaces the right code for the job. The reality, newly measured by a benchmark called ExecRetrieval, is that embeddings can't reliably tell correct code from a buggy near-clone of it. The top hosted system retrieves the right answer 100% of the time in its top-10 — and only 33.1% of the time at rank 1, because the wrong-but-look-alike code keeps winning.

The Problem: Retrieval Scores Similarity, Not Correctness

Embedding retrieval ranks code by vector similarity: how close a candidate's meaning is to the query's. It has no notion of whether a candidate executes correctly. That was fine when benchmarks rewarded finding topically relevant code. But agents retrieve code to use it — and when a candidate is a near-clone of the right solution with one mutated line, it embeds almost identically to the correct version.

Previous code-retrieval benchmarks never tested this. Their search pools didn't contain execution-verified buggy variants planted as counterfactuals, so a retriever could look perfect while being unable to functionally discriminate correct from incorrect code. ExecRetrieval is built to close exactly that gap.

The Method: Execution-Verified Counterfactuals

ExecRetrieval (from Aaryan Kapoor and Md Abdullah Al Hafiz Khan) contains 939 Python tasks, each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors — each generated by a mechanical mutation making a single targeted edit. The buggy variants pass some tests and fail others: they're genuinely broken, but near-identical to the canonical in text and embedding space.

The authors then evaluate 23 dense embedding configurations plus BM25 under provider-native invocation, with paired McNemar tests and query-level bootstrap intervals. Because the wrong answers are deliberate near-clones, a retriever's rank ordering gets tested for functional discrimination instead of topical or identity overlap.

graph LR
    A["939 Python tasks"] --> B["Canonical implementation
(execution-verified)"]
    A --> C["Up to 4 buggy near-clones
(single-edit mutations, execution-verified)"]
    B --> D["Search pool"]
    C --> D
    Q["Natural-language query"] --> E["23 embedding configs + BM25"]
    D --> E
    E --> F["Ranked list"]
    F --> G["exec@1 = 0.331
exec@10 = 1.00"]
            

The Results: exec@1 Collapses When Buggy Clones Compete

The headline numbers are brutal for anyone deploying code retrieval today:

• exec@10 = 1.00 but exec@1 = 0.331 for the top hosted system — the right code is always in the top 10, yet usually not first.
• Rank-1 misses are paired buggy variants 91.5–99.4% of the time across the four leading systems. When the agent grabs the top hit, it's grabbing broken code.
• The canonical scores below at least one of its four buggy distractors in 67–78% of queries on the leading systems.
• A naive syntactic baseline recalls at most 14% where the embedding systems recall ~100% at exec@10 — so the failure isn't about recall, it's specifically about ordering.

Limitations

The benchmark is Python-only, and the buggy variants are mechanically generated single-edit mutations — real-world bugs are messier, multi-edit, and entangled with context. The authors also acknowledge that execution-verified distractors are the hard case: in production, a retriever's low-ranked hits may simply be irrelevant rather than subtly broken, which flatters these numbers. And the top-k results still contain the correct code, so downstream reranking or execution verification can rescue the pipeline — the benchmark measures the retriever alone, not the full agent.

Why Builders Should Care

If you ship a coding agent, RAG-for-code pipeline, or autocomplete that pulls from a code index, this is a direct measurement of your failure mode. Retrieval alone cannot be trusted to order code by correctness when near-clone bugs exist — and near-clones are exactly the code that pollutes real repositories: copy-pasted with a typo, a reordered argument, an off-by-one.

Three practical moves: (1) treat exec@1 as a fantasy metric and gate retrieved code through an execution or lint-based verifier before an agent acts on it; (2) use reranking with execution signals (unit tests, type checks) rather than pure embedding distance; (3) prefer agents that retrieve-then-verify over retrieve-then-trust. Wrong code retrieved confidently is worse than no code — it fails later, and fails silently.