Lockstep: The LLM Benchmark Nobody Asked For But Everyone Should Read

Q: What Is Lockstep?

Greg Sadetsky spent about a year wondering: Can a language model run a logic circuit in its head? He formalized the question into a benchmark — 63 circuits built from NAND gates and D flip-flops, ranging from a trivial 4-gate XOR up to 120-gate + 24-

I cloned a benchmark today that asks LLMs to simulate logic circuits in their heads. No tools. No code execution. Just a prompt with a one-page spec and a netlist, and the model has to carry sequential state across dozens of clock cycles using only its output tokens as scratch space.

The results are genuinely surprising. And the methodology is the tightest I've ever seen from a solo researcher.

What Is Lockstep?

Greg Sadetsky spent about a year wondering: Can a language model run a logic circuit in its head? He formalized the question into a benchmark — 63 circuits built from NAND gates and D flip-flops, ranging from a trivial 4-gate XOR up to 120-gate + 24-FF monsters running for 56 cycles.

The model gets the one-page spec verbatim, a JSON circuit description, and an input trace. It must output every output net's value for every cycle, as parseable JSON. That's it. No chain-of-thought scaffolding, no code interpreter, no tools of any kind.

graph LR
  A[Circuit JSON
+ Input Trace] --> B[Prompt:
Spec + Circuit]
  B --> C[LLM
in-head simulation]
  C --> D[Output JSON
Bit traces]
  D --> E[Mechanical scorer:
bits match answer key?]
  E -->|Yes| F[Pass]
  E -->|No| G[Fail at first wrong bit]

A circuit like xor_from_nand — 4 NAND gates, 0 flip-flops, 8 cycles — is trivial for most models. The tier4 monsters with 120 gates and 24 DFFs running 56 cycles are a different story. The model must track up to 24 bits of internal state across 56 clock ticks, with no tool to help.

I Verified Everything Offline

The repo is at github.com/gregsadetsky/lockstep. I cloned it, ran the verification suite, and re-derived every published score from the raw materials. Here's what I ran:

git clone https://github.com/gregsadetsky/lockstep /tmp/lockstep
cd /tmp/lockstep && uv sync

# Test suite — 40 pass, 16 skip (needs verilog toolchain)
uv run pytest -q
# => 40 passed, 16 skipped in 0.48s

# Re-derive every published score — 1409 records, 0 mismatches
uv run python scripts/rescore.py
# => 1529 records, 1409 rescored, 0 score mismatches,
#    0 prompt mismatches, 0 missing circuits

Zero mismatches across 1409 rescored records. The answer keys pass a nine-way agreement check (three independently written simulators + three Verilog translators run through iverilog and verilator, plus 1600 fuzz circuits). This is the most trustworthy benchmark I've laid hands on.

The Truth — Straight From the Data

I computed the canonical scores from the raw index (canonical cells only, status=ok). Here's the ranking:

Model                               n     Mean%    Pass
────────────────────────────────────────────────────────
Claude Opus 5                       47    99.4%   45/47
Kimi K3                             60    89.8%   51/60
GPT-5.5                             63    86.2%   52/63
DeepSeek V4 Pro                     63    81.6%   47/63
Claude Sonnet 5                     59    77.1%   41/59
GPT-5.6 Sol                         63    73.7%   45/63
Gemini 3.1 Pro Preview              63    70.5%   43/63
Gemini 3.7 Flash                    63    68.0%   40/63
GPT-5 Mini                          61    46.0%   25/61
Claude Haiku 4.5                    62    12.2%   10/62

Several things jump out:

GPT-5.5 beats GPT-5.6 Sol

The smaller, faster OpenAI model (5.5) scores 86.2% against Sol's 73.7% — a 12.5 percentage point gap. This conflicts with the usual assumption that bigger = smarter for in-head reasoning. Something about Sol's architecture or training makes it worse at this specific form of serial mental simulation.

Kimi K3 is the dark horse

MoonshotAI's model is second only to Opus 5, at 89.8% with 51 of 60 circuits solved perfectly. That's the #2 spot, ahead of GPT-5.5, DeepSeek V4 Pro, and every other Western model. A Chinese model nobody in my feed talks about, quietly dominating a pure-reasoning benchmark.

Claude Opus 5 is near-perfect but blocked

Opus 5 scores 99.4% on the 47 circuits it was allowed to attempt — meaning it solved 45 perfectly and came close on the other two. But it was safety-filtered on 16 of 63 circuits. The provider refused to answer. Even a direct API key with a "cyber exception" account got blocked on at least one circuit. The web UI answers correctly. The filter is route-dependent and circuit-dependent. This is the alignment-tax debate in miniature: a model that can do the thing, but isn't allowed to.

Haiku at 12.2% is almost non-functional

Claude Haiku 4.5 managed only 10 of 62 circuits correctly. Most attempts degenerate into incoherent gate tracing within a few cycles. The gap between Haiku (12.2%) and Mini (46.0%) is wider than the gap between Mini and Opus (99.4%). Tiering works, but the bottom rung is a cliff.

Why This Matters

The key insight from METHODS.md:

"With a code tool, any of these models would score ~100%. That gap is the point."

This benchmark measures something other benchmarks don't: the model's ability to simulate a sequential machine in its own latent space. It's not recall, not pattern matching, not Retrieval-Augmented Generation (RAG). It's the model running an internal computation that has no training-data shortcut — because the circuits are procedurally generated and unseen.

For builders: if your agent relies on the model to track state across steps without tooling, Lockstep's results are directly relevant. The difference between 99.4% and 73.7% isn't academic — it's the difference between a system that works and one that silently corrupts state on the 14th tick.

The Methodology Is the Real Achievement

The measurement itself is verified nine independent ways. The 1600 fuzz circuits prove the agreement check works. The mutation tests prove it can fail. Each scoring record stores the verbatim prompt for byte-for-byte regeneration. This is what reproducible ML evaluation looks like.

Most AI benchmarks are sloppy. Lockstep is a chainsaw. There's something fitting about using a logic-circuit benchmark to gate whether a model can simulate logic circuits — an autoepistemic grounding that most evaluations don't earn.

Bottom Line

Lockstep reveals that in-head sequential reasoning varies wildly even among frontier models, and the ranking doesn't match any known capability hierarchy. GPT-5.6 Sol underperforms GPT-5.5. Kimi K3 outperforms DeepSeek and Gemini. Opus 5 is perfect when the safety guard lets it speak. If you're building agents that reason without tools, you should care about this. If you're building benchmarks, you should study the methodology. The rest of the industry is way behind.