DK Dark Knight 2026-09-02 7 min read

The Emergent Symbolic Structure of Neural Networks

Yesterday's top arXiv paper isn't another benchmark. It's the receipt for a 35-year-old bet in cognitive science. In 1990, Paul Smolensky published a formalism — Tensor Product Representations (TPRs) — claiming vectors could literally hold symbols: bind a filler (the letter M) to a role (second-from-the-end) by multiplying their vectors, sum the bindings, done. The vector-based AIs that conquered the 2020s looked like the final refutation. Now Smolensky — with R. Thomas McCoy, Paul Soulos, and Tal Linzen — has published the settlement (arXiv 2608.29530): the internal representations of trained networks can be replaced by a single interpretable symbolic equation, and the networks don't notice. I read the full paper and reproduced the core experiment on a toy network. The result holds — with serious caveats the authors print in their own paper.

How Do You Prove Vectors Hide Symbols? Swap Them In

The method is called DISCOVER (DISsecting COmpositionality in VEctor Representations), from the same lineage, scaled here from RNNs to LLMs. The trick is the test, not the fit. They train a small model that is explicitly a linearly-transformed TPR — a sum of role⊗filler tensor products pushed through a linear map — to reproduce the target network's encodings. Then comes the part that separates this from every decoder-probe ever published: they take the TPR's symbolic encodings, plug them into the target network's own decoder, and check whether the network still produces the correct output. Approximation accuracy means: percentage of test inputs where the network, running on the symbolic equation instead of its own vectors, outputs the complete correct answer.

graph TD
  subgraph TARGET["Target network (black box)"]
    T["Transformer / MLP encoder"] --> E["encodings E"]
  end
  subgraph DISCOVER["DISCOVER (symbolic hypothesis)"]
    R["role embeddings"] --> TP["tensor products Σ r⊗f"]
    F["filler embeddings"] --> TP
    TP --> W["linear map W + b"]
    W --> S["E_TPR — closed-form symbolic equation"]
  end
  E -->|"trained to match (MSE)"| S
  S -->|"fed into target's own decoder"| D["target decoder"]
  D --> Q{"behavior preserved?"}

This is compression in the mathematical sense: every vector the model produces gets replaced by a formula you can read. Probes tell you what information is in vectors; DISCOVER reconstructs the structure of the vectors themselves. The whole symbolic hypothesis lives or dies on the substitution test.

What Are the Numbers Behind Emergent Symbolic Structure?

The substitution test passes, sometimes absurdly well:

That last failure is worth sitting with: the single most symbolic domain (arithmetic) is the one where the symbolic approximation generalizes worst. The map is not the territory.

Where Does the Symbolic Account Break?

Read the paper's Section 3.5 before you tweet the headline. Four limits, self-printed:

The honorable move is their interpretation: limitivism. Networks approach symbol systems in the limit but realize them approximately, noisily — and that noise is probably the source of their power at fuzzy natural language, precisely where pure symbols fail. This kills eliminativism (you can't delete symbols from the theory of what these models do) without falling into naive implementationalism. Fodor and Pylyshyn's ghost is still arguing, but the ground moved.

Did the Structure Show Up in My Own Reproduction?

I built the smallest honest version of the pipeline in numpy: a 64-unit MLP encoding 12-letter lists (length ≤ 6) and decoding their reversals — 96.65% test accuracy. Then I fit a linearly-transformed TPR to its hidden activations with ridge regression and fed the symbolic encodings into the target's own decoder:

# DISCOVER-style fit: phi = role-filler basis, A = target activations
W_tpr = np.linalg.solve(Phi.T @ Phi + 1e-3*np.eye(d), Phi.T @ A)
A_tpr = phi_test @ W_tpr          # closed-form symbolic encodings
out = softmax(A_tpr @ W2 + b2)    # target's own decoder

Same ordering as the paper's Figure 3.2, reproduced with 50 lines of numpy. Constituent surgery — swap the letter at position 1 via the TPR's role-filler arithmetic — flipped outputs correctly in 315/357 cases (0.882). My first training run diverged into a frozen degenerate attractor (a 6×-too-hot cross-entropy gradient, runaway output weights, tanh saturation pinning the loss at 25.37 for 150 epochs) before Adam fixed it — the fit only works if the target is actually good at its task, which is itself a silent precondition worth remembering.

Bottom Line

Symbols don't have to be bolted onto neural networks — they emerge inside them, as an information-structure discovered by gradient descent, in models from 64-hidden-unit MLPs to 27-billion-parameter LLMs. That's the real result: emergence, not implementation. For anyone doing mechanistic interpretability, the unit of analysis just changed — the causally relevant atoms are multiplicative role⊗filler bindings, not neurons or directions, which retroactively explains a lot of why sparse autoencoders work. For everyone else: this is proof of concept that closed-form symbolic equations can stand in for network internals with ~90% behavioral fidelity — a lever for editing model behavior with surgical precision, distributed structure and all. Just don't read it as "LLMs are symbolic." Read it as "symbols are what vectors do when they get good."

Related Posts