SSD-Streamed MoE: Running 104GB Qwen3.8 on a 48GB Mac

Qwen3.8-Flash-Next is a 125B-parameter mixture-of-experts model. At 4-bit that's 104 GB of disk. My Mac has 48 GB of RAM. The math doesn't work unless you stream the weights from SSD — and every naive approach (mmap, MLX's own loader) dies immediately because the model's 512 experts per layer materialize all at once.

Slotstream, a Swift tool posted to HN today (2026-09-01), solves this with a specific engineering trick: pread-based expert reads into a fixed CLOCK-evicted slot pool shared across all 48 layers, plus a 32 GB n-gram table on the same pread path. It runs at ~12 tok/s on a 48 GB M5 Pro. I cloned the repo, read the engine source, and checked every claim against the repo's own measurements doc. The numbers are honest — sometimes brutally so.

The real story isn't "104 GB model on a 48 GB Mac." It's that Apple's MLX framework can't materialize a partial tensor from a memory-mapped file. Every expert gather evaluates all 512, loading ~100 GB and pushing the machine into swap. Slotstream's fix — pread + explicit slot cache — is a pattern that will matter for every large MoE model going forward.

Why mmap Fails for 104GB MoE Models

This is the key insight. Qwen3.8-Flash-Next has 48 layers, each with 512 routed experts plus a shared expert. Only 10 experts are active per token, but MLX's mx.gather_qmm materializes the entire tensor it reads from. A memory-mapped file doesn't help: mmap pages in on access, but the gather operation touches every row of the selected tensor, which for 512 experts means loading all ~100 GB before it can pick 10.

The README is explicit: the stock mlx_lm.load() route took this 48 GB machine into 48 GB of swap without producing a single token. This isn't a memory pressure issue — it's a framework limitation. MLX is designed for models that fit in max_recommended_working_set_size. The moment a model's tensor dimensions exceed that, the API has no escape hatch.

The Fix: pread + CLOCK Expert Cache

Slotstream's approach: don't mmap. Read exactly what you need with pread, one expert record at a time, into a fixed pool of cache slots. An expert record is nine pieces (gate/up/down projection × weight/scales/biases), each ~307 KB. The tool issues 9 parallel preads per expert at a queue depth of 12 (swept: 12 and 32 tie at ~4.5 GB/s; 64+ is worse, so the sweet spot is aggressively tuned).

graph TD
    SSD[103.8 GB on SSD
68 GB routed experts
32 GB n-gram table
3.8 GB resident trunk] -->|pread 9×307KB
QD=12, <4.5 GB/s| Stage[Aligned staging buffers
posix_memalign 16KB]

    Stage -->|MLXArray rawPointer
+ dealloc callback| Pool[Slot pool
CLOCK eviction
shared across 48 layers
bit-exact weights]

    Pool -->|gather_qmm
from pool tensor| Compute[MLX compute
dequantize + matmul
10 active experts/token]

The pool holds bit-exact quantized bytes — the same U32 and BF16 values that are on disk. This is why cache size changes speed, never output. Greedy decoding is byte-identical between a 4 GB cache and a 24 GB cache, and that equivalence is a standing test in the repo.

The n-gram/PLE store (32 GB, 320 million rows — 10× the plan's initial estimate) uses the same pread pattern. Each table row is a compact 100 bytes (weight + scales + biases interleaved), and the 16 scattered reads per token are page-granular cached (128 rows per 16 KB page).

Measured Numbers (and the Honest Corrections)

The measurements doc (MEASUREMENTS.md) is the most refreshing part of this project. It includes a retracted first measurement (a naive benchmark that called mx.eval() after every single expert write, reporting 1.54 GB/s — until the author realized batching before one eval is what a real engine does). It also publicly re-anchors the decode estimate after finding it was 25–45% optimistic.

Here are the real numbers, measured on a single M5 Pro 48 GB Mac:

The 20 tok/s figure at 181 experts/layer could not be re-verified. Forcing that config drove the machine to 158 MB free and 13 GB of swap, producing a wide 12.5–18.6 tok/s band. The estimator now interpolates verified points and holds flat above 120 experts/layer — the right failure direction for a planner.

Prefill is the slow axis: the entire prompt is processed before the first token appears. At 8,000 tokens that's ~95 seconds. But the prefix cache (32K tokens across 4 conversations) means follow-up turns pay only the new prefill, keeping time-to-first-token flat at ~6 s even at turn 8.

Limits and Trade-offs

• macOS only. Apple Silicon, macOS 14+, and ~110 GB of free disk. 512 GB Mac is the realistic minimum.
• One model. v0 runs exactly qwen3.8-flash-next:4bit. The engine is built around this geometry — expert count, layer count, n-gram row width, everything hardcoded.
• Long prompts are expensive. 8K tokens = ~95 s prefill. The cost is split three ways: reading (34s), scattering (10s), compute (50s). Read-ahead doesn't eliminate the serialization because the three phases are fully sequential.
• No tool calling, structured output, or image inputs. The API returns explicit 400s for unsupported features. What you get is a clean chat completion server with Ollama-compatible and OpenAI-compatible endpoints.
• Smaller Mac tiers are estimates. The 8 GB, 16 GB, and 24 GB rows are from slotstream doctor --sim-ram, not measured on hardware. The author says "smaller Macs also have slower SSDs," so real-world numbers will be lower.

Bottom Line

Slotstream is a well-engineered solution to a specific problem: running a MoE model whose total weight is larger than available RAM, when the ML framework can't materialize partial tensors. The pread + CLOCK cache pattern is the right one, and the byte-identical output guarantee across cache sizes is a strong engineering property that most inference stacks don't offer.

What I respect most: the measurements doc is honest. It includes the wrong first measurement, retracted. It re-anchors the decode estimate downward when the real number didn't match. It admits the 20 tok/s figure couldn't be reproduced. That's rare, and it makes every number in the README trustworthy. The tool itself is v0 — one model, one platform, a single stated problem — and it solves that problem cleanly. For anyone who needs to run a 104 GB MoE model on a consumer Mac, this is the best path forward.