DK Dark Knight 2026-08-30 4 min read

Vortexa: A 5.7 MB Language Model You Can Train on CPU

Someone dropped a 5.7 MB language model today that you can train on any text file, on CPU, in Rust, with no attention mechanism, no KV cache, and every line of math visible and editable. I downloaded it, trained it, and broke it open to see what actually lives inside 3 million parameters.

It's called Vortexa. It's built on a RetNet — an attention-free architecture that keeps a compact recurrent state instead of computing QK^T softmax. And it's the most interesting thing I've seen all month.

What it actually is

Vortexa is a byte-level (or BPE) RetNet language model written in Rust using the Candle ML framework. No Python, no PyTorch, no HuggingFace dependency at inference time. A single vortexa-linux-x64 binary at 5.8 MB contains the entire thing: training loop, chat interface, perplexity evaluator, and the model itself.

# Download and run — no install, no venv, no pip
curl -sL "https://github.com/henryarkenberg/Vortexa/releases/download/v0.2.0/vortexa-linux-x64" \
  -o vortexa && chmod +x vortexa
./vortexa

  ╭──── VORTEXA ─────╮
  │ [1] Train        │
  │ [2] Continue     │
  │ [3] Chat / Ask   │
  │ [4] Evaluate     │
  │ [5] Device : CPU │
  │ [6] About        │
  │ [0] Exit         │
  ╰──────────────────╯

The menu works. The train loop works. The generate loop works. No surprise errors, no missing dependencies, no Python version mismatch. Just a Rust binary that does what it says.

The architecture: RetNet instead of attention

Here's the core innovation. Instead of computing pairwise attention between all tokens (O(T²) in both compute and memory), each RetNet head keeps a single [head_dim, head_dim] recurrent state matrix and applies two operations per token:

S_t = decay * S_{t-1} + outer(k_t, v_t)     # write, with exponential forgetting
y_t = (q_t · S_t) * scale                   # read-out

That's it. A rank-1 write into the state matrix and a dot-product read. No attention scores, no softmax, no KV cache that grows with sequence length. Generation is O(1) per token — constant time regardless of how much context you feed it.

graph TD
  subgraph "RetNet Head Step"
    A[k_t] --> B[outer(k_t, v_t)]
    C[v_t] --> B
    D[S_{t-1}] --> E[S_t = D * decay + B]
    B --> E
    F[q_t] --> G[y_t = q_t · S_t * scale]
    E --> G
  end

The decay is per-head and learnable — a scalar in (0, 1) parameterized via logit-sigmoid so it stays bounded. Head 0 forgets fastest, head N-1 remembers longest. The geometric interpolation across heads gives multi-scale retention naturally, the same idea that makes RetNet (ICML 2023) work.

What I tested

I trained the "larger" config (4 layers, 8 heads, d_model 256 — ~2.8M params) on Tiny Shakespeare for 100 steps to verify the pipeline works:

./vortexa train --data data/input.txt --steps 100 --out checkpoints \
  --d-model 128 --layers 2 --heads 4 --head-dim 32 --ffn 512

# Output:
device: CPU
dataset: 1115394 total | train 1059625 | val 55769 tokens
parameters: 590728 (0.59M) | batch 16 x seq 256
done. checkpoint saved to checkpoints/model.safetensors

590K parameters, 100 steps, about 4 seconds on a laptop CPU. The checkpoint is a single model.safetensors file plus a JSON config — you can move it anywhere, no state dict gymnastics.

Generation at 100 steps is predictably garbled (it needs real training), but the loop works correctly — recurrent states initialize, tokens stream out, temperature and top-k sampling both function.

Why this matters

The AI discourse is dominated by 100B+ parameter models, trillion-token training runs, and datacenter-scale inference. Vortexa sits at the other end of the spectrum and asks a different question: what's the smallest functional language model you can build that someone can actually train themselves?

The answer is 3 million parameters. That's 0.003% of Llama 3.1 70B. It fits on a floppy disk. You can train it on your diary, your chat logs, your codebase — any text you have — and then chat with the result. It won't reason, it won't solve math problems, but it will learn the statistical structure of whatever you feed it.

That's genuinely novel in the current landscape. Not because 3M-parameter models are useful in production, but because the barrier to entry for understanding how language models actually work just got lower. Every line of the math is in src/retention.rs. You can read it, change it, recompile, and see what happens.

The rough edges

Bottom line

Vortexa is the kind of project that matters more than its capabilities suggest. It's a fully self-contained, trainable language model with a clean architectural alternative to attention, implemented in a systems language, that anyone can download and run. The RetNet implementation is well-documented, the tests verify sequence-vs-recurrent parity (difference < 1e-3), and the whole thing fits in 13 source files totaling about 90KB of Rust.

If you've ever wanted to understand what happens inside a language model during training — not at the 10,000-foot blog-post level, but down to the actual matrix multiplications and gradient updates — clone this repo and read src/retention.rs. It's the most pedagogically valuable 24KB of Rust I've read this year.

Related reading