DeepSeek v4.1 Flash on 2x DGX Spark: 31.6 tok/s, 600k Ctx
Three days ago the frontier-local story was someone getting 23 seconds per token running DeepSeek v4.1 Flash on a 2020 M1 Mac Mini. Today, a new repo — MiaAI-Lab's EXL3 build, created September 13, already at ~88 stars — serves the same model at 31.6 tok/s decode on two DGX Sparks. That is a ~700x swing in four days, and it puts a full frontier MoE (with 600k context and vision) inside roughly $2,000 of desk hardware.
What shipped
The checkpoint is EXL3 2.9 bpw (mul1 codebook, ~196 GiB across 39 shards) served by a vLLM overlay with tensor-parallel 2 over ConnectX-7. Two details separate this from a random quant dump. First, the K map is per tensor — routed experts at K=3, attention at 5, lm_head at 6, indexer at 8 — with a warning not to naively floor the 2.9 average. Second, the Engram n-gram tables (~190 GiB) are never quantized and stay file-backed from the original FP8 shards, because quantizing them produces fluent garbage.
Why the numbers matter
The measured table is the receipts: 1,041 tok/s prefill at 10k, still 802 tok/s at a 455k prompt (567 s to first token), 19–24 tok/s steady decode at 100k–455k context, and 42.5 tok/s aggregate on two streams. KV costs ~3.4 KiB/token against a 2.5 GiB pool covering 774k tokens. Batch serving is the interesting inversion: turn the built-in DSpark speculation off at 4 streams and aggregate jumps from 42.8 to 53.7 tok/s — speculation is a single-stream tax, not a free lunch.
The fine print
The memory floor is a long prefill, not boot: a 3 GiB KV pool passes the smoke test and dies at ~470k of a 600k prompt. The repo also documents a real GB10 failure mode — ~113 GiB of driver allocations are invisible to the RSS-based OOM killer, which killed desktop daemons instead on September 11. One unverified caveat: these are the maintainer's own numbers, no third-party replication yet. But the recipe stamps, packed-matrix counts, and per-layer K map read like someone who actually ran it, repeatedly.
The verdict: the "frontier model on a desk" line moved from party trick to serveable in under a week. Watch for an EXL3 2.9 bpw single-Spark squeeze and independent throughput audits next.