TurnBench Exposes the Gap in Voice AI: No System Knows When to Stop Talking

Voice AI is having a moment. Sesame's viral demo showed what it feels like when an AI sounds human — the pauses, the intonation, the little conversational grunts. But there's a chasm between "sounds human in a demo" and "actually functions in a real conversation."

The hardest problem in voice AI isn't speech recognition, voice cloning, or even generation quality. It's turn-taking — knowing when to stop talking and let the other person speak. Get it wrong and your voice agent either interrupts constantly (the worst UX sin) or leaves awkward silences (a close second).

Today, Sesame AI Labs released TurnBench, a multi-domain benchmark that quantifies exactly how far we are from solving this. I cloned the repo, read every line of the scorer, studied the baselines, and analyzed the leaderboard. The results are sobering.

What TurnBench Actually Measures

TurnBench is a 30-hour, hand-annotated corpus of 154 dyadic (two-person) conversations across 6 conversation types with 106 speakers. Every conversation is triple-annotated (Fleiss' κ = 0.78) with ground truth derived from 2/3 consensus.

It measures two things:

• End-of-Turn (EOT) detection — Did the model correctly identify when a speaker finished, so the listener can respond?
• Interruption (INT) detection — Did the model correctly identify when the listener wanted to cut in while the speaker still held the floor?

Each is scored on three axes: recall (did you catch the event?), false-positive rate (did you fire when you shouldn't have?), and latency (how fast did you catch it?). A submission qualifies only if its FPR stays under 0.15 — anything above is ranked below all qualifiers, no matter how high the recall.

This isn't a toy. The scorer enforces causality: your model's timestamp is "the time by which all audio the decision depended on has been heard." No cheating with lookahead — you pay for it in latency.

The Leaderboard: Nobody Is Close

I pulled the full leaderboard data from the benchmark site. Here's every model that qualified (FPR ≤ 0.15):

VAP is the strongest overall — 0.845 recall at 368ms with a 0.055 FPR. But look at that latency. Human listeners begin speaking a median 151 ms before the current turn ends. Not 368ms after. Before.

Then look at what didn't qualify:

• OpenAI Realtime (Server VAD): 0.955 recall, but 0.525 FPR — fires on more than half of all negative spans. Unusable in a real product.
• RMS Energy VAD: 0.718 recall, 0.632 FPR — the simplest baseline, fires constantly.
• OpenAI Realtime (Semantic VAD): 0.303 recall — misses 70% of events.

The paper's own conclusion: "no system is simultaneously fast, high-recall, and low on false positives."

The 151ms Problem

This is the killer detail from the paper:

"In smooth floor transfers, human listeners begin speaking a median 151 ms before the current turn ends."

Human conversation is anticipatory, not reactive. We project when the other person will finish and start our response in overlap. The acoustic cue (the pause) comes after the turn has already transferred. Reactive systems — which is every current model — are structurally disadvantaged: they must wait for evidence before committing, and by then the gap is noticeable.

The paper also notes a mean response offset of 208 ms across 10 languages for question-response pairs. The gold standard for turn-transfer latency isn't zero — it's negative. That's the target no current system can touch.

The Conversation-Type Problem

TurnBench covers six conversation types: goal-directed information exchange, shared reasoning, storytelling, and others. The critical finding: end-of-turn recall is stable across types, but interruption FPR varies wildly.

Casual conversations with high backchannel density — the "uh-huh", "mm-hmm", "right" conversations — generate the most false positives. Acoustic systems can't distinguish between a backchannel and a real interruption attempt. Semantic systems can, but they're too slow. Every model hits this tradeoff: speed vs. specificity.

What I Actually Did

I cloned the TurnBench GitHub repo and:

• Read the full scorer source code — the matching algorithm is one-to-one, time-ordered, with window clamping so one prediction can't satisfy two events
• Studied the RMS Energy VAD baseline — a simple RMS-threshold predictor that fires on every silence: 20ms windows, 0.01 threshold, speech-offset → EOT, speech-onset → interruption
• Read the submission format — causal timestamps, schema version 1, one prediction per conversation per speaker
• Examined the gold-building code that derives ground truth from raw annotator tracks
• Pulled all 12 baselines' prediction files (dev and test) from the repo

The scorer itself is cleanly written and MIT-licensed. The submission format is straightforward — a single JSON with event timestamps. I wasn't able to run the scorer locally (the dev dataset on HuggingFace is gated and requires access approval), but the API surface is clearly documented and the reference baseline (RMS VAD) shows the exact shape.

What This Means for Agent Builders

If you're building a voice agent today, here's your tradeoff:

• Use an acoustic VAD (RMS, OpenAI Server VAD): low latency, high recall, but your agent will fire false positives constantly. Users will hate it.
• Use a semantic system (Gemini 3.1 Live, OpenAI Semantic VAD): low FPR but high latency and poor recall. Your agent will leave awkward silences. Users will also hate it.
• Use VAP or a learned model: best of both worlds, but still at 368ms latency — more than double the human 151ms target. And you need specialized models.

There is no system that hits all three axes. The best-in-class still misses 15% of turn transfers and takes 2x human reaction time. Voice agents are not ready for unconstrained conversation, and TurnBench provides the precise numbers to prove it.

The Bottom Line

TurnBench is the right benchmark at the right time. Voice AI is about to become the next interface paradigm — every major lab has a real-time voice mode now — but the quality bar for "feels natural" is invisibly high. A voice agent that's 85% accurate at turn-taking is 85% annoying, because the 15% it gets wrong are the moments that break the illusion completely.

The gap between any current system and human conversation isn't incremental. It's structural. Human turn-taking is anticipatory; every current system is reactive. Closing that gap probably needs a fundamentally different approach — models that project speaker intent before the pause finishes, not just detect it after.

Until then, every voice agent you talk to will either interrupt you or leave you hanging. Those are your options.

graph TD
    subgraph "Turn-Taking Approaches"
        A[Acoustic VAD] -->|Low latency, high recall| B[High FPR]
        A -->|"Fires on every silence"| C[Bad UX: Interrupts]
        D[Semantic System] -->|Low FPR, safe| E[High latency, low recall]
        D -->|"Waits for evidence"| F[Bad UX: Silences]
        G[Learned Model - VAP] -->|Best balance| H[Still 2x human latency]
        H --> I[Misses 15% of events]
    end