Vibe coding is having its reckoning. Seventeen months after Karpathy named it, the first comprehensive review of the evidence landed on arXiv today — and the picture it paints is not what the hype would have you believe.
The headline findings from "Vibe Coding: Practice, Performance, Productivity, and Risk" by Michels et al.:
- Peer-reviewed field experiments: +26% more tasks per week
- Independent randomized trials: 19% slowdown
- Team-level telemetry: code-review time up +441%
All three are real. All three describe the same phenomenon.
The paper assembles evidence across a cross-disciplinary corpus spanning software engineering, HCI, labor economics, security, governance, and education. It's not a single experiment — it's a meta-analysis of the entire first wave of empirical work on AI-assisted coding. The contradictory results aren't noise. They're the signal.
The Contradiction, Explained
The authors identify six patterns behind the dispersion:
- Effect-shrinkage under broader measurement — narrow tasks show big gains; full-sprint tracking shows smaller or null effects
- Self-report diverges from independent measurement — developers feel 2x faster; stopwatches disagree
- Output volume conflated with productivity — writing more code faster isn't the same as shipping working software
- Bold claims walked back over longer horizons — early results look best; the advantage shrinks with study duration
- Task granularity masks capability gaps — code generation is strong; fault detection is weak; documentation is hard to audit
- Novice gains vs expert regression — juniors benefit most; seniors see smaller (sometimes negative) returns
The headline productivity contradiction (+26% vs -19%) resolves once you control for measurement method, scope, and time horizon. The gains are real on new code and shrink or reverse on mature codebases.
That last point is the paper's falsifiable conjecture: vibe coding's advantage is a function of codebase age. Greenfield projects benefit; brownfield projects suffer.
What the Benchmarks Actually Say
The model landscape at time of writing: early benchmarks are saturated, but task-level capability is uneven. Code generation works. Fault detection doesn't. Documentation quality is hard to audit but suspected poor.
The paper documents:
- Security failures in deployed applications built with AI assistance
- Code-quality degradation visible in large-scale telemetry
- Unsettled copyright exposure (no resolution from courts or regulation)
- Evidence of skill atrophy — developers who rely on AI assistance show weaker debugging skills when working without it
Where the Gains Come From
The paper's framework: vibe coding describes a workflow where the developer describes intent in natural language and validates by running rather than reading the generated code. The cognitive load shifts from construction to verification.
This works well when:
- The task is well-understood and the solution space is constrained
- The developer can evaluate correctness by observing behavior (tests, screenshots, logs)
- The cost of "wrong" is bounded
It fails when:
- Correctness depends on deep system understanding
- The cost of subtle bugs is high (security, safety, financial)
- The codebase has accumulated constraints not visible in surface behavior
Why This Matters
The paper's closing claim is the most important: that the productivity debate is asking the wrong question. Instead of "does vibe coding make developers faster?", the real question is "under what conditions, and for whom?"
The answer so far: yes for greenfield, no (or negative) for brownfield. Yes for junior, mixed for senior. Yes for generation, no for debugging. Yes for output volume, no for code quality.
The field needs more granular measurement, not more polemics.
- Vibe Coding: Practice, Performance, Productivity, and Risk - A State-of-the-Art Review — Michels, Ghazaleh, Lazzari, Kassem, Klein, 20 Aug 2026
Also notable from today's arXiv drop:
Who Delegates to AI? Evidence from 53,000 Agent Configurations — Lee, Cheon, Kim. The first empirical map of who actually adopts AI agents, using 53K skill specs from the Manus marketplace. Key finding: delegation peaks at the bachelor's level and the middle of the wage distribution — not at the top.