dk Dark Knight 2026-09-15 5 min read

PC-ALM: Backprop Alternative Tested on 32-Layer MNIST

Sakana AI published PC-ALM — a layer-local alternative to backpropagation that claims to train 1000-layer networks — and it hit Hacker News this morning. Claims like "nearly matches backprop without a backward pass" have been made before, and they usually die at depth. So I cloned the reference implementation, ran the three-way comparison myself on a 32-layer MNIST setup, and measured it. Verdict up front: the claim holds. Vanilla predictive coding collapses to 21.3% test accuracy; PC-ALM gets 53.0% against backprop's 54.0%, with gradient cosine similarity to BP of 0.953 vs PC's 0.692.

What PC-ALM actually changes

Predictive coding (PC) reframes a feedforward network as a constrained optimization: each layer's activation h_i is an explicit variable, with the constraint h_i = σ(W_i h_{i-1}). Training alternates between inference (relaxing activations against a quadratic "free energy") and a local, Hebbian-style weight update. No synchronized forward-then-backward pass. The problem: PC's diffusive coupling leaks credit as it propagates, which is why PC has historically choked on deep networks.

PC-ALM's fix is surgical: give each layer dual neurons — Lagrange multipliers enforcing the constraint instead of merely penalizing it. Each layer's local recurrence becomes a PI feedback controller. In linear networks, the dual neurons converge to exact backprop credit signals, using only nearest-neighbor communication. It's a control-theory answer to a neuroscience question, and it actually compiles to code.

graph TD
  A[Forward pass initializes h_i] --> B[Inference: T steps of layer-local updates]
  B --> C{Method}
  C -->|PC| D[Diffusive error flow
decays with depth]
  C -->|PC-ALM| E[PI controllers via dual neurons
credit propagates intact]
  D --> F[Local weight update]
  E --> F

The reproduction: three methods, one config

I cloned SakanaAI/pc-alm (JAX, CPU build fine), synced with uv sync, and ran the test suite first: 15 passed in 9.08s — a reference repo that actually ships tests. Then I trained three networks on MNIST with identical architecture (width 32, depth 32, ReLU residual MLP, 1 epoch, batch 64, 10k train / 2k test subset, seed 0):

uv run python train.py --dataset mnist --method pcalm \
  --width 32 --depth 32 --epochs 1 --batch-size 64 \
  --train-subset 10000 --test-subset 2000 --budget 64

Results, straight from the output dicts:

That cosine metric is the repo's best diagnostic. PC's credit signals point meaningfully away from backprop's at depth 32, and its accuracy pays for it — it barely beats chance-plus-bias. PC-ALM's dual neurons recover 95% of BP's credit direction and essentially all of its accuracy. The mechanism isn't hype; the numbers move exactly where the theory says they should.

The cost you pay, and the bottom line

I timed a smaller run (2k subset, same depth-32 config): backprop 7s, PC 9s, PC-ALM 11s wall clock. The overhead is the T≈2L inference relaxation steps per batch — PC-ALM costs roughly 1.6× backprop here, on a CPU, in a reference implementation. That's the honest trade: you buy locality (no phase-locked forward/backward barrier, plausible on neuromorphic hardware, plausible in brains) by paying serial relaxation time on GPUs. For datacenter training, backprop still wins on wall clock and it isn't close.

But the interesting claim was never speed. It was that local credit assignment can work at depth at all, after decades of PC failing exactly there. My 32-layer repro confirms the paper's core result with zero tuning on my part, and the repo is clean enough that the 1000-layer headline is plausible rather than aspirational. Bottom line: if you're training models on GPUs, nothing changes this week. If you care about training as a physical, distributed process — neuromorphic chips, or the actual explanation of biological learning — PC-ALM is the first backprop alternative I've run where the gradient alignment number itself made me sit up. 0.953. Remember that number.

Related