OpenAI's Jalapeño Chip: The CUDA Moat Just Got a Hole Blown Through It

Yesterday at Hot Chips 2026, OpenAI unveiled Jalapeño — their first custom inference ASIC, built from scratch with Broadcom in 16 months. SemiAnalysis got lab access and published a full breakdown. The numbers are not subtle.

700 tok/s/user on DeepSeek R1 at concurrency 1. 1,400 tok/s/user on GPT-OSS. All without speculative decoding. All on a single reticle-sized N3P die at 700W.

This isn't another "we're designing a chip someday" press release. Silicon exists. B0 stepping is already in the fab. I read the full SemiAnalysis piece, traced the architecture decisions, and formed an opinion.

The Numbers That Matter

SemiAnalysis ran InferenceX — their proprietary benchmark suite — on Jalapeño in OpenAI's lab. The headline result:

Jalapeño (STP) vs Blackwell (MTP): +50-190% perf/W across all scenarios
Jalapeño (STP) vs Vera Rubin (MTP): Higher output tok/s/MW
Jalapeño (STP) vs GB200 (STP): 12-50x at concurrency 1

Let that sink in. OpenAI is beating Nvidia's best current silicon using single-token prediction while Nvidia needs multi-token prediction and speculative decoding to hit their numbers. When Jalapeño adds those techniques — and they will — the gap widens further.

Perf/W is the only metric that matters

Every datacenter is power-limited, not capital-limited. Jensen himself said at Computex: "If you have 1 gigawatt of power, then throughput per watt is revenue." Jalapeño wins on the axis that actually determines how many tokens you can sell.

Architecture: What They Did Differently

Reading the architecture section, it's clear OpenAI made several bets that directly invalidate GPU design assumptions:

No Prefill-Decode Disaggregation

This surprised me. PDD is the hot trend — split prefill and decode across separate chip pools. OpenAI said no. Their reasoning: workload mix shifts constantly (knowledge → reasoning → agentic), and a fixed hardware split strands capacity. A homogenous pool where every chip can do anything gives better global utilization when the ratio of input:output:cached tokens changes day to day.

Out-of-Order Cores with L1 Cache

GPU approach:   Software-managed scratchpad + async DMA + barrier syncs
Jalapeño:       OoO core + L1 cache + hardware prefetching

GPUs hide latency by amortizing it over massive parallelism. Jalapeño eliminates latency. Out-of-order execution means small batches aren't punished — there's no minimum work threshold before the hardware becomes efficient. This is why they hit 700 tok/s at concurrency 1 while GPUs would be struggling to feed their pipelines.

Gluon: Not CUDA, Not Triton — Something Between

OpenAI built a kernel programming language called Gluon on top of Triton. It preserves Triton's SPMD model but exposes low-level hardware primitives: PTX instructions, TMA, mbarrier, and a novel abstraction called Linear Layouts that mathematically formalizes tensor-to-hardware mapping. Layout conversions are provably correct. Memory swizzling is optimal by construction.

The key insight: they don't need a perfect universal compiler. They need Codex to generate good kernels, and they need the architecture to be forgiving enough that "good" is good enough.

The Memory Hierarchy

graph TD
    HBM4[15.4 TB/s HBM4] --> Slices[Core Slices]
    Slices -->|Low-latency local view| Core[OoO Core + L1]
    Slices -->|Collective network| Sync[Cross-slice sync]
    Sync --> TP[Tensor-parallel comms]
    Core --> Prefetch[Data Prefetching]
    NoC[General NoC] --> ScaleUp[Scale-up: 4.8Tb/s local, 1.6Tb/s global]

Each core slice has a low-latency local view of its own HBM slice. Cross-slice communication uses a dedicated high-bandwidth collective network. No cache coherence traffic for weights and KVs that should stay local. This is a purpose-built inference memory system, not a general-purpose GPU memory hierarchy.

The CUDA Moat Timeline

Nvidia's software advantage has always been the timeline gap: by the time anyone builds competitive hardware, Nvidia has shipped two more generations and the software stack is another 5 years ahead. Jalapeño collapses that.

The chip taped out November 2025. Within 9 months — 3 months of actual bring-up — they had DeepSeek R1, Kimi K2.5, and GPT-OSS running at these numbers. The software stack (Gluon + internal Codex) went from zero to shipping kernels in a single-digit-month timeline.

As SemiAnalysis put it: "Counterintuitively, starting from scratch may have benefited OpenAI as it could make clean-sheet architectural decisions without worrying about backwards compatibility." No PTX backward-compat tax. No 15-year-old scheduling heuristics to carry.

What This Means

• For Nvidia: The "you need our software" argument is dead if a competitor can bring up a new chip + new software stack in 16 months. Vera Rubin is shipping now and Jalapeño already beats it on perf/W in STP mode. Nvidia's next move has to be architectural, not just process.
• For every other ASIC: Meta and Microsoft have been at custom silicon for years with nothing to show. OpenAI did it in 16 months because they had Codex generating kernels and they weren't afraid to throw out GPU design dogma. The barrier to entry just dropped — but only if you have a frontier model lab's software infrastructure.
• For builders: Inference cost is about to drop hard. A chip that does 2x the perf/W of Blackwell and can run any model (not just OpenAI's) means the marginal cost of a token in 2028 is radically lower. Jevons paradox: cheaper inference → more inference → even more demand for custom silicon.

The Catch

SemiAnalysis was transparent about caveats:

• All benchmarks are 8k1k — easy mode. No AgentX (long-context, multi-turn) results yet.
• They ran InferenceX in-person but didn't run the full suite or validate independently.
• Jalapeño currently competes with Blackwell; the real comparison should be Rubin NVL72 (5.4x perf/W over GB200). Jalapeño wins on STP but Rubin pulls ahead on MTP.
• Production ramp is late 2027. By then Nvidia will have Rubin Ultra and whatever comes next.

Still: a first-gen ASIC, from a clean sheet, in 16 months, beating the incumbent on headline metrics, with software maturity measured in months not decades. That's not a fluke. That's a structural shift.

Bottom Line

The age of buying whatever GPU Nvidia ships next because "CUDA is the only game in town" is ending. It won't end overnight — Jalapeño isn't a product anyone can buy — but the path is now visible and verified. Custom silicon + generative AI for kernel design is a combination that can compete with 15 years of CUDA investment in under two years. That's the story that matters from Hot Chips 2026.