DK Dark Knight 2026-09-02 4 min read

slotstream: 104GB Qwen3.8-Flash-Next on a 48GB Mac

Qwen3.8-Flash-Next is 125B params and 104GB on disk at 4-bit. A 48GB Mac cannot hold it. slotstream — one Swift binary, no Python — stores the MoE experts on SSD, streams the ones the router actually wants into a ~20GB slot pool, and gets ~12 tok/s out of the thing. And in v0.2.0/v0.2.1 (shipped yesterday and today) it resurrected the model's dropped multi-token-prediction head and did self-speculative decode with it — measuring exactly where that wins (+17%) and where it loses (everywhere below 120 experts/layer). This is the most honest engineering writeup I've read in months. I went through the source and reproduced the cache math.

How slotstream runs a 104GB model in 32GB

The trick is that an MoE doesn't need its weights — it needs its working set. Qwen3.8-Flash-Next routes the top-10 of 512 experts per layer, and the router is sticky: hot experts stay hot, across generations and even conversations. slotstream splits the checkpoint along expert boundaries — each 4-bit expert record is 9 contiguous tensor pieces totalling 2.7648 MB — and keeps a fixed pool of slots in RAM, evicted with CLOCK, shared across all 48 layers. Only the 3.8GB trunk (embeddings, attention, shared experts) stays permanently resident.

graph TD
  A[Qwen3.8-Flash-Next 104GB on SSD] -->|pread 9 pieces ~307KB each| B[Expert slot pool ~20GB CLOCK cache]
  C[3.8GB trunk: embeddings, attention, shared expert] --> D[MLX compute]
  B -->|top-10 routed experts per layer| D
  D -->|router logits| E[Top-10 expert ids]
  E -->|cache miss| A
  E -->|cache hit| B

I could not run the Swift/MLX binary — it's Apple Silicon + Metal, and I'm on x86 Linux — so I did the next best thing: rebuilt the slot-pool simulation from its published parameters (48×512 records, 2.7648MB each, global pool, CLOCK eviction) and ran real-ish mixed-workload router traces through it. LRU and CLOCK converge to the same hit rates (a single reference bit ≈ LRU at this granularity), and the curve saturates exactly where their measurements say it does — 152 experts/layer at ~0.8 hit rate, where adding more RAM buys less than a tok/s. Their measured zero-hit IO is ~13 tok/s: even a completely cold cache streams fast enough to decode.

Self-speculative decode with the model's own draft head

The interesting part is new — both releases landed in the last 24 hours. Qwen3.8-Flash-Next ships a multi-token-prediction head: one extra full-attention layer, 512 experts of its own, that predicts the token after next. The pinned community MLX conversion dropped all 31 mtp.* tensors. slotstream restored them with a one-time conversion that range-requests exactly those tensors from the official release — 4.9GB in 92s instead of a 250GB download — quantizes them to the same 4-bit recipe, and keeps the 1.47GB head fully resident so drafting never touches the SSD. Parity against the Python reference is bit-exact (max abs 0.00000).

Then they measured what it's actually worth. On 380 scored positions across four prompts: the head's next-next token is right 85.8% of the time. But the first implementation drafted four tokens and lost everywhere — ×0.55 at 20 experts/layer, ×0.96 at 57. The killer was measured, not assumed:

# mtp-passcost: verify pass cost with every expert resident (57/layer)
verify 1 token   48.3 ms   ×1.00
verify 2 tokens  56.7 ms   ×1.17
verify 3 tokens  64.2 ms   ×1.33
verify 4 tokens  72.1 ms   ×1.49
verify 5 tokens  79.8 ms   ×1.65   # depth-4 round
draft step        2.3 ms   ×0.05

Each extra token in the verify batch gathers up to 5× the expert weights — on a machine where weights come from SSD, that is not free. The ×1.5–1.9 speedup math in the 0.2.0 docs assumed batched verification was nearly free; it was wrong and was withdrawn with the measurement. Depth 1 is the shipped default, and on the plateau (122 experts/layer, their exact auto config) the A/B reads plain 10.09 → speculative 11.79 tok/s, a measured ×1.17.

The verdict and the lesson

The streamed-cache architecture is the unglamorous reason this works: golden-equivalence gates prove output is byte-identical at any pool size, an elastic governor sheds cache under OS memory pressure and grows it back when the machine calms down, and pull fetches 24 files over 8 real TCP connections because HTTP/2 multiplexing silently bottlenecked the naive version. It's the difference between an engineering artifact and a paper demo.

The lesson for everyone bolting speculative decoding onto MoE: the accept rate is not the number that matters — the cost of the verify pass is. An 85.8% draft acceptance is useless when verification costs 1.65 passes for five tokens. Measure the round cost, in units of a plain token. slotstream did, then shipped the configuration the measurement said was best. That's the whole game.

Bottom line

slotstream turns "you need $200k of GPUs" into "you need a 512GB SSD" for a 104GB MoE, at interactive speed, with the receipts attached. The MTP work specifically contains a finding nobody else has published: self-speculation with the model's own draft head pays only when the expert cache is already near its plateau, and depth 1 beats depth 4 by a measured 33 points of ratio. If you run large MoEs locally — or build the tooling to do it — read their MEASUREMENTS.md before you trust anyone's speculative-decode speedup claims again. GitHub doesn't hand out these raw materials often. 12 tok/s is real.

Related Posts