OpenArch Teardown: I Ran All 13 Models, Only 1 Works

OpenArch hit the Show HN front page today: hand-written PyTorch implementations of 20+ modern LLM architectures, "one readable file per architecture," built on Sebastian Raschka's LLM architecture gallery. Great idea. Terrible execution. I cloned it, built a smoke-test harness, and ran every text model twice — once at tiny dims for forward passes, once on the meta device at the published configs to verify headline parameter counts. 12 of 13 models fail. Only Llama 4 Maverick produces a single token.

The method: smoke test + meta-device audit

Two phases, both automated:

# Phase 1: tiny-dim forward pass (embed_dim=64, 2-4 blocks, seq_len=16)
# Phase 2: instantiate at published config.json dims on torch.device('meta')
#          and sum parameter numel — counts a "671B" model without 2.5TB of RAM

The meta-device trick is the useful part here: a repo that claims to implement DeepSeek R1 or GLM-4.5 should at least have parameter counts that match reality. You don't need a GPU cluster to check that.

The scoreboard: 12 failures, 5 before construction

Five models can't even be instantiated:

• Llama 3, DeepSeek R1, Mistral 3: nn.ModuleList(*[...]) unpacks the list into positional args — TypeError on every construction. This is a Python-level bug, not a modeling subtlety.
• Llama 2, GPT-2: KeyError: "attribute 'mask' already exists" — the same bug, two files.
• Qwen3-4B: the model constructor accepts qk_norm but never forwards it to TransformerBlock.

The rest fail at forward:

[FAIL] kimmi-K2:      mat1 and mat2 shapes cannot be multiplied (16x64 and 32x128)
[FAIL] gpt-oss-20B:   _get_causal_mask() missing 1 required positional argument: 'device'
[FAIL] glm4.5-355B:   tensor a (16) vs tensor b (64) at dim 3
[FAIL] qwen3-30B-A3B: tensor a (16) vs tensor b (4) at dim 2  — even with qk_norm off
[FAIL] gemma3-27B:    TransformerBlock() got multiple values for 'attn_type'
[OK]   llama4-400B:   6.67M params (tiny cfg), fwd 31.7 ms, out (1, 16, 1000)

The Gemma 3 failure is my favorite, because it's two bugs stacked: Gemma3Model passes **cfg into TransformerBlock and passes attn_type explicitly — so attn_type arrives twice. And the cfg keys don't match the block's signature anyway.

One bug, three models: QK-norm on the wrong axis

GLM-4.5 and Qwen3 both implement QK-norm as RMSNorm(embed_dim) applied to a tensor already reshaped to (batch, heads, seq, head_dim). So the norm's weight — sized for the model dimension (64 in my test) — meets a last axis of size head_dim (16). Instant shape crash. QK-norm normalizes per head, along head_dim. Three independent models, identical wrong pattern, copied forward. That's the tell of an LLM-assisted codebase: the same mistake replicated with total confidence.

Qwen3-30B-A3B is worse — it crashes even with qk_norm=False, and its attention has a literal duplicated line:

v = self.w_v(x)
v = self.w_v(x)   # twice is not twice as good

And the dense Qwen3 model's forward never iterates its own blocks: x = self.transformer_blocks(x, mask) calls the nn.ModuleList object itself, not the layers inside it.

The parameter-count audit: 3 pass, 2 are 7.7× off

On the meta device at published configs, the story splits:

• gpt-oss-20B: 20.91B vs 20.9B target. Dead on.
• kimmi-K2: 1,037B vs ~1T target. Correct.
• llama4-400B: 397.76B vs ~400B. Correct.
• glm4.5-355B: 2,749B — 7.7× the real 355B. The expert FFN dims are inflated.
• qwen3-30B-A3B: 233.56B — 7.7× the real 30.5B. The real Qwen3-30B-A3B uses moe_intermediate_size=768; this repo uses the dense 6,144.

Both misses are 7.7×. Same class of error, two models.

The hygiene receipts

• grok2.5-270B/model.py is 0 bytes — a config.json pointing at nothing.
• gpt-oss-20B/mdoel.py — the file name is misspelled.
• gpt-oss-20B/config.json has a trailing comma. It's not valid JSON.
• llama3-8B/model.py opens with from multiprocessing.managers import convert_to_error — a hallucinated import that's never used and wouldn't exist.
• Kimi K2's "dense first layer" applies a raw MLP that overwrites the block output instead of replacing the MoE inside it.

Verdict: a gallery, not a library

To be fair to the author: the README says the goal is "clarity and learning," not production. And the per-model table of norm/attention/MoE choices has genuine reference value — the docs are better than the code. But "implemented to the best of my knowledge" and a ✅-for-"usable for forward passes" is a falsifiable claim, and 12 of 13 falsify it. The ✅ column is aspirational labeling. If a reader picks Llama 3 as their learning reference, the first thing they learn is a Python error.

The fix is cheap and would transform the repo: one CI job that instantiates every model at tiny dims and asserts a forward pass, plus a meta-device param-count check against config.json. That's maybe 100 lines of pytest and it converts a claims board into a test suite. Until then, read Raschka's gallery it's based on — the prose there is load-bearing in a way this code isn't.

Bottom line: OpenArch is a beautifully organized index of models that mostly don't run. Of 13 text implementations, 5 die at construction, 7 die at forward, 1 (Llama 4) actually works — and the parameter counts prove the working ones were checked, while the broken ones were labeled anyway. For learning architectures, use it as a reading companion, never as code. And if you're shipping "from-scratch" implementations of anything: a 30-second smoke test would have caught every single one of these.