Google dropped the DiffusionGemma technical report today — and it's the most consequential LLM architecture paper since the transformer itself.
DiffusionGemma is an open-weight language model that replaces autoregressive decoding with discrete diffusion. Instead of generating one token at a time (the bottleneck that speculative decoding and Medusa try to patch around), it iteratively refines blocks of 256 tokens in parallel. The result: ~20 tokens per forward pass vs. AR's hard limit of 1. On a single H100, that translates to roughly 1,500 output tokens per second — faster than AR models even with state-of-the-art speculative decoding bolted on.
What makes this real
This isn't a from-scratch diffusion experiment. Google fine-tuned Gemma 4 (3.8B active / 25.2B total MoE) using a two-stage pipeline: supervised fine-tuning teaches bidirectional denoising, then RL + sampler distillation optimizes for generation quality and inference efficiency. Total training spend: less than 10% of the original model's token budget.
More importantly, DiffusionGemma keeps the things that made Gemma 4 useful — thinking mode (chain-of-thought), multimodal inputs, and long contexts. And because it retains the ability to fall back to AR generation, Google is signaling a path to hybrid diffusion-AR decoding: crank out bulk tokens via diffusion, then refine the critical ones autoregressively.
Why this matters
The inference industry has spent two years optimizing AR: speculative decoding, draft models, KV-cache compression, flash attention. All of that is still optimization around a fundamentally sequential bottleneck. DiffusionGemma says the bottleneck itself is optional.
For anyone running agents at scale: 1,500 tok/s on a single H100 changes the cost equation. You can run real-time reasoning loops, iterate generations, or serve more users per GPU without throwing hardware at the problem. And because it's open-weight, this isn't a Google Cloud exclusive — within weeks, the llama.cpp / vLLM community will have a port running on consumer hardware.
This is the first time a diffusion LM at scale has been both open-weight and competitive on a real evaluation suite. The autoregressive decade just got its expiration date.