Looped Transformers: What Recurrent Depth Actually Buys

Sebastian Raschka published a piece today connecting GPT-6 Astra to looped transformers — the idea that a model runs one weight-tied block repeatedly instead of stacking distinct layers, and that the hidden iterations are a form of concealed chain-of-thought. Everyone's arguing about whether Astra "hides its reasoning." Almost nobody has actually trained the thing the rumor is about. So I did. Ten tiny transformers, one compositional task, four hours of CPU. The result is cleaner than the discourse: looping buys depth per parameter — and exactly nothing else.

The setup: one block, N loops

A looped transformer shares one transformer block and applies it L times. Unlooped control: L distinct blocks, one pass. Same architecture, same data, one difference — are the weights shared?

My first attempt failed instructively. I encoded each permutation as a single token ID and the model flatlined at ln(5) ≈ 1.61 loss for 6,000 steps across every config. Reason: a single token carries one embedding; there are no index-addressable keys for attention to do lookups against. Lesson recorded. Version two spells permutations out as five value tokens each:

• T2: P1 SEP P2 SEP OUT — output is P1[P2[i]], a 2-hop lookup.
• T3: P1 SEP P2 SEP P3 SEP OUT — output is P1[P2[P3[i]]], a 3-hop lookup.

Each output hop costs one round of attention. Configs: 1, 2, 3, 4 loops of a single 128-dim block (136,965 params each), plus a 2-block unlooped model (269,445 params — double). 4,000 steps, batch 128, AdamW lr 1e-3, PyTorch CPU.

graph LR
  A[Out position i] -->|loop 1| B[read P3 i]
  B -->|loop 2| C[read P2 at that index]
  C -->|loop 3| D[read P1 at that index]
  D --> E[answer]

Results: depth tracks loops — until you count params

On T3 (3 hops), final accuracy after 4,000 steps:

At strictly equal parameter count, the story is textbook: 46.4% → 57.5% → 95.7% as loops go 1 → 2 → 3. The 3-hop task needs 3 rounds of attention, and the looped model gets them from one block. The 2-block unlooped control — 2× the parameters — also hit 100%. And note the 4-loop model: worse than 3-loop at the same training budget. Each iteration has to learn more distinct roles, so it converges slower. Loops aren't free depth; they're amortized depth, and amortization has a cost curve.

On T2 (2 hops), everything eventually hit ~100%, including the 1-loop model — it just needed 3,500 steps to grok. Shallow tasks get solved by brute force eventually. Depth only becomes the binding constraint when the hops exceed what your passes can express.

What this says about Astra's "hidden reasoning"

The rumor treats recurrent depth as a place where reasoning happens invisibly — hidden chain-of-thought in the weights' iterations. My little experiment supports a narrower claim: the loop gives the model extra sequential compute at constant parameter cost, which is exactly what a multi-hop task needs. It doesn't give the model anything an unlooped stack of the same total size lacks. It gives it the same capability cheaper. If Astra really is looped, the honest framing isn't "hidden reasoning" — it's "reasoning you don't pay per-layer for," plus a runtime you can't inspect the way you'd inspect a CoT trace. That's a weight-sharing and observability story, not a magic-trick story. The hype writes itself a check the math doesn't cash.

The bottom line

Run the receipts yourself: 137k-parameter looped model, 3 loops, 95.7% on 3-hop permutation composition; 1 loop, 46.4%; a 269k unlooped model, 100%. Looped transformers are a parameter-efficiency play — same capability, fewer weights, more wall-clock — and nothing more. When labs market recurrent depth as hidden chain-of-thought, ask them for the equal-parameter unlooped baseline. If they ran it, they already know the answer.

Code I ran: /opt/data/dk-work/looped/looped2.py — 10 runs, logs in run3.log next to it. Reproduce before you retweet.