RealSWE: Realistic Prompts Drop Coding Agent Success 6.4%

Coding agents are graded on a benchmark that doesn't describe how people actually ask for code. SWE-bench tasks are built from curated GitHub issues — long, structured, information-rich bug reports. Real user requests are short, messy, and vague. For months the field has been optimizing agent pipelines against one distribution of inputs while shipping them into another, with no number on how far apart the two are. RealSWE, from a team at NAVER, quantifies the gap — and it's not small: realistic inputs cut agent resolution rates by 6.4 points on average, and they can reshuffle the agent leaderboard.

The Problem: SWE-Bench Evaluates the Wrong Inputs

SWE-bench Verified and Pro are the de facto yardsticks for coding agents. Their problems come from real GitHub issues, but the selection process skews toward the long-tailed, well-specified reports — the kind that include reproduction steps, environment details, and expected behavior. Real users in chat interfaces don't write like that. A user says "this API call is broken" or "make it faster," and the agent is expected to decode intent from a fraction of the context a curated issue provides. The benchmark measures skill at reading structured bug reports; production measures skill at reading humans.

The Method: 381 Task Families with Controlled Input Variation

The paper's design is the point. Rather than collecting a small pile of real-world failures, the authors define a six-category information taxonomy — Problem Statement, Desired Behavior, Motivation, Reproduction Steps, Environment Information, and extra context — plus four dimensions of linguistic style. They annotate real prompts from SWE-chat and compare against SWE-bench Verified and Pro problem statements. The finding: requests carrying only a problem statement, alone or with limited context, make up 88% of real prompts but just 7% of benchmark problems. And 87% of real prompts are casually written while 94% of benchmark problems are formal.

Then comes the controlled part: sys, 381 multi-variant task families derived from SWE-bench. Variants in each family share the same underlying task and gold patch, differing only in information composition and linguistic style. Seven contemporary LLMs are evaluated across the grid, which lets the authors isolate what actually drives performance — without confounds like task difficulty.

The Numbers: A 6.4-Point Drop — and What Actually Moves It

• Realistic inputs reduce resolution rates by 6.4 percentage points on average, and can change model rankings outright.
• Desired Behavior and Motivation matter. Including them significantly improves performance — and most real prompts omit them.
• Environment Information and Reproduction Steps are dead weight. They add tokens without measurable benefit — the benchmark's most verbose categories buy nothing.
• Linguistic style has only small, model-dependent effects. Casual wording costs far less than missing information.

The actionable takeaway is clean: the gap between benchmarks and reality is mostly an information gap, not a style gap. Tell the agent what you want and why — the parts humans naturally skip — and the success rate climbs.

Limitations — Read These First

• The task distribution is still SWE-bench. The families are re-wrapped versions of one benchmark's tasks with known gold patches — the difficulty mix inherits SWE-bench's biases even if the inputs don't.
• The fix shifts work onto users. "State desired behavior explicitly" improves success, but it's advice for prompt crafters, not an agent-side solution. Making agents elicit missing context themselves is the harder problem that remains.
• Seven models, one eval window. Rankings can change with the next generation of agents; the 6.4-point average may be model-dependent — the paper itself shows the drop varies across models.
• Taxonomy annotation is manual. The six categories are human-applied, so borderline prompts introduce judgment calls that are hard to fully audit.

Why Someone Building Things Should Care

If you ship a coding agent, SWE-bench numbers are your pitch deck. RealSWE is the correction factor: expect ~6.4 points of your benchmark score to evaporate in the real world, and expect leaderboard order to shuffle — what ranks first on curated issues may not rank first on chat prompts. If you build agent prompts, the results are a direct checklist: surface desired behavior and motivation in your scaffolding, in the agent's clarification questions, or in your product's input form. If you read agent evals, treat "beats SWE-bench" as "beats SWE-bench on SWE-bench-shaped problems" — and demand the realistic-input variant. The entire community is training on the wrong distribution, and now we have the number.