Every local AI project has the same demo: llama.cpp on a MacBook getting 3 tok/s on a 70B model. Impressive as an engineering flex. Useless as a daily driver.
FreeToken changes the equation. It's an edge-native MoE serving engine from UC Berkeley that runs 290B+ parameter frontier models on consumer GPUs — RTX 30, 40, and 50 series — at interactive speeds. Not Qwen2.5-7B. DeepSeek-V4-Flash. Qwen3.6-35B-A3B. GLM-5.2. The models that actually compete with frontier APIs.
The trick is that FreeToken treats CPU, GPU, host memory, and PCIe bandwidth as a unified inference platform. Its bandwidth-adaptive CPU-GPU co-execution policy (they call it the q* policy) dynamically decides which experts run where based on real-time bandwidth conditions. Full-layer double-buffered prefill streaming. Global LRU expert caching. No VRAM swapping when you hit the limit — because the limit isn't fixed.
Three things make this worth paying attention to:
- Desktop app exists. Download from flashml.ai, point it at a model, it just works. No terminal gymnastics.
- Coding agents work out of the box. The CLI supports
ft launch claude / codex / dsh / hermes / openclaw / opencode— it writes the provider config and starts the agent. This is built for the agentic workflow, not just chatting. - The paper is real. arXiv:2608.16157 with authors from UC Berkeley (Stoica, Zaharia, Han, Keutzer). Not a weekend project.
The local AI argument has always been about privacy and cost. The counterargument has always been capability — "sure, but can it run good models." FreeToken doesn't close that gap. It collapses it. A Qwen3.6-35B-A3B running on a 5090 at usable speeds is not a toy. It's a production-capable local inference node that costs zero per token to operate. The ceiling for local models just moved up by an order of magnitude, and the floor for what "consumer hardware inference" means just got redefined.