Can AI Design Circuit Boards? EEBench Has Numbers

Wednesday, OpenAI showed GPT-6 Astra drawing a PCB in KiCad on the front page of a flagship launch. Thursday, a small team funded by the makers of atopile published the tool that tells you whether those circuits would actually work. The gap between the two is the whole story of AI benchmarks right now: one camp demos, the other grades. EEBench V1 is 13 circuit tasks, real manufacturer parts, SPICE simulation, deterministic grading — and its leaderboard has a receipt nobody in the demo business wants framed: the best model scores 61.6%, and OpenAI's two tested models sit at the bottom.

The Demo vs the Measurement

OpenAI's GPT-6 Astra launch put a model clicking around KiCad in front of millions of eyeballs. It looked great. It also proved nothing — a GUI demo shows a model can operate software, not that the design it produces survives contact with physics. The EEBench team's counter-move is elegant: don't test computer use at all. Their harness uses atopile, where a circuit is declarative code. The agent edits components, connections and electrical constraints directly, builds, simulates, and inspects what failed — no coordinates, no menus, no screenshots.

The grading is fully deterministic: build the submitted design, construct the circuit graph and bill of materials, run SPICE simulations and design checks, and bind each measurement to a spec limit. Cost efficiency only counts once the circuit works. It's SWE-bench's core insight — a compiler and tests — except the tests measure voltages and tolerance corners.

graph LR
  A[Agent edits .ato source] --> B[Build circuit graph + BOM]
  B --> C[SPICE simulations]
  C --> D[Tolerance corners]
  D --> E{Spec limits met?}
  E -->|Yes| F[Score = technical + cost]
  E -->|No| G[Fail with named voltage/corner]
  G --> A
            

The 22 µF Receipt

One public task is a residential energy meter. When its 5 V supply disappears, the circuit must hold the processor's rail above the 3.0 V brownout threshold for 20 ms so the accumulated reading can be saved. The right base move is obvious: add a capacitor. Every model gets that far. The receipt is what happens next.

A submitted design used a 22 µF nominal capacitor. The grader found 11.4 µF of effective capacitance at 4.7 V bias — a ceramic part delivering roughly half its label — against a 545 µF requirement. The source built cleanly. The circuit still failed: the rail dropped below 3 V after 0.85 ms, 23.5x short of the window.

I didn't take that on faith. I rebuilt the discharge math myself:

# t_holdup = C_eff * (V0 - Vmin) / I
# Deriving load current from EEBench's measured failure:
# 11.4uF * (4.55V - 3.0V) / 0.85ms = 20.8 mA

capacitor                        hold-up  verdict
22 uF nominal, derated (real)     0.85 ms  FAIL
22 uF nominal, as-specced         1.64 ms  FAIL
required 545 uF (nominal)        40.64 ms  PASS

My model reproduces their sim exactly — 0.85 ms from the same 20.8 mA load the failing design implies. And it surfaces the uglier fact underneath the derating story: even at nominal value, 22 µF fails this spec by 12x. The 545 µF requirement isn't padding; it's tolerance corners. "Knows electronics" has to mean knowing a 0805 X5R gives up half its capacitance at operating bias, not reciting the label. Any benchmark that graded nominal values would have passed designs that die in the field. That's the difference between a demo and a grader.

The Scoreboard Labs Don't Push

The September 1 V1 results, 13 tasks:

• Claude Opus 5 — 61.6%
• Grok 4.6 — 57.1% (xAI's own model-card run: 60.0% at xhigh reasoning effort)
• Claude Fable 5.1 — 56.4%
• Claude Fable 5 — 54.3%
• Claude Opus 4.8 Max — 51.4%
• GPT-5.5 — 42.3%
• GPT-5.6 Sol — 39.4%
• GPT-6 Astra — not tested yet

Three things worth noticing. One: xAI put EEBench in the Grok 4.6 model card under "engineering acceleration," next to parametric CAD and 3D modeling — a frontier lab citing a third-party physics benchmark is new, and the atopile team isn't selling scores. Two: OpenAI's tested models trail the leaders by ~20 points, and the model OpenAI chose to demo on a circuit board has no published number. Three: Grok 4.7 is reportedly weeks away, trained on SpaceX engineering data — this scoreboard is about to get stress-tested whether labs like it or not.

What I Ran: atopile From Source

I cloned atopile master and tried to build a trivial board — divider, LED, hold-up cap. Install friction, honestly logged: master demands Python 3.14 (PyPI's 0.2.69 release is a different, older architecture — my first attempt died on its syntax), compiles a Zig core on first run (ziglang==0.15.1 pinned; 0.16.0 fails to build), needs more-itertools held back to the 10.8.0 lockfile pin because 11.x removed zip_equal, and its own version check rejects the dev-install version string unless you set SETUPTOOLS_SCM_PRETEND_VERSION=0.14.0.

# what it took
uv venv --python 3.14 && uv pip install -e atopile \
  && uv pip install "ziglang==0.15.1" "more-itertools==10.8.0"
SETUPTOOLS_SCM_PRETEND_VERSION=0.14.0 uv pip install -e atopile
ato build

Then the interesting part — the build stages, verbatim from my run:

✓ Initializing build context [5.1s]
✓ Modify type graph
✓ Instantiate app
✓ Verify instance graph
✓ Verify electrical design
✓ Loading PCB
✗ Picking parts — Fetching component data failed:
    connection error: Name or service not known

Everything deterministic — parse, type graph, instance graph, electrical verification — runs locally and passed my toy circuit on the first try. Part picking phones home to components.atopileapi.com, which my sandbox couldn't resolve. So the grader's core is genuinely local and reproducible, but the picker is a cloud dependency; if EEBench's tasks grade BOM cost against real orderable parts, their scores lean on that service being honest and up. Worth watching, not disqualifying.

Bottom Line

Electronics is the next frontier for agent benchmarks, and it's arriving the right way: deterministic, physical, hostile to nominal-value thinking. The demo told you Astra can click; EEBench tells you the field's best is at 61.6% on problems where the penalty for not knowing DC-bias derating is a dead meter. When Astra's number lands, compare it to that 61.6% — not to the KiCad video.