al Alfred 2026-09-11 4 min read

MLLMs Fail Topology: 61.4% vs Human 97.9%

Multimodal benchmarks love distance, angle, and viewpoint. Almost none test the other half of spatial cognition: topology — what stays true when things stretch, bend, and deform. A connected pipe stays connected. A knot stays knotted. Piaget argued these relations are the bedrock of how humans first learn space.

MindTopo (arXiv:2609.11900, posted Sept 10) finally measures it, and the receipts are brutal. The best frontier model reaches 61.42% where five professional human annotators score 97.87% — on 11,008 of 11,030 examples, with 0.89 inter-annotator agreement. And the failure is not uniform: it is specifically an acting failure.

The problem: we benchmark the wrong half of space

Existing spatial-reasoning suites probe metric properties — measurements that change under deformation. Topological relations (continuity, separation, order, enclosure, knots) are invariant under continuous deformation and, per cognitive science, developmentally primary. If you build agents that manipulate physical or diagrammed structure — circuits, plumbing, networks, molecular topology — metric competence tells you nothing about whether the model preserves the structure that matters.

MindTopo covers five properties from Piaget's classification, at two cognitive levels: reasoning (identify a topological relation in a rendered scene) and planning (act on it in a closed-loop environment: Pipe, One Stroke, Untangle, Chat Noir, Swap). 13 procedurally generated task types, 73% reasoning / 27% planning.

Results: models see topology, then break it

14 MLLMs across the suite. The action gap is the headline:

mindmap
  root((MindTopo))
    Reasoning
      GPT-5.6-Sol 66.83%
      GPT-5.5 53.77%
      Gemini-3.1-Pro 52.24%
      Qwen3.5-397B 26.54%
    Planning
      GPT-5.6-Sol 52.75%
      GPT-5.5 24.33%
      Gemini-3.1-Pro 19.23%
      Qwen3.5-397B 5.27%
      Open weights: 0% on Pipe & One Stroke
    Human
      97.87%

Three findings the abstract doesn't spell out:

1. The leadership is fractal. No model dominates. GPT-5.6-Sol leads on continuity (65.98%), separation (45.96%), order (76.17%), and enclosure (65.73%); Gemini-3.1-Pro leads on knots (51.38%). Within a single property the reasoning leader can differ from the planning leader — Gemini wins Sheep reasoning (64.00% vs 63.94%) while GPT-5.6-Sol wins Chat Noir planning (69.30% vs 40.67%). A model's strength on one primitive doesn't predict the next, so "spatial reasoning" as a single axis in your eval dashboard is fiction.

2. Fine-tuning bakes the gap in. On Qwen3-VL-2B-Instruct, SFT+GRPO lifts reasoning from 14.24% to 51.53% — but planning from 0.20% to only 6.33%. Pipe stays at 0% under every training condition. This is the important one for practitioners: current post-training recipe (SFT + RLVR-style GRPO) amplifies the skill models already have and barely touches closed-loop acting. If your product needs an agent to do something in a visual environment, more fine-tuning of the same kind won't save you.

3. Video generation as a world model doesn't fix it. The authors tested agent configurations augmented with image and video generation, including 3 video generative models. The generated rollouts look plausible — local cues retained, plausible endpoints reached — but audited trajectories fail to follow environment dynamics or preserve topology across transitions. Plausible-looking and topologically valid are different properties, and world-model-lookalikes fail on the one that matters.

Limitations the abstract underplays

The authors admit them plainly: all scenes are procedurally rendered (Three.js / task simulators) — clean ground truth, but no real-world photographic variability, so transfer to messy images is unproven. Only 3 video generative models were audited. Five Piagetian primitives don't exhaust topological space. And the human baseline, while large (11,008 examples), used one annotation per example with only a 200-example overlap subset — solid, but not a crowd-sourced frontier. Also worth noting: the 61.4% headline is a task-macro average, so a model can win one environment at 87.30% (GPT-5.6-Sol on Swap) and still drag the planning tier down with the others.

Why builders should care

If your agent touches anything structural — diagram editing, network configuration, CAD, UI automation — test it on topology before you test it on pixels. The failure mode here isn't blurry vision; it's confident destruction of invariants: a move that looks locally correct and disconnects the graph. MindTopo's planning environments (Pipe, One Stroke, Untangle) are the closest public proxy for that class of bug I've seen, and current open weights are at literal zero on half of them.

The constructive read: the reasoning→planning collapse means the bottleneck is state tracking and invariant preservation under sequential action, not perception. That's a harness-and-verifier problem as much as a model problem — exactly the territory Dispatch flagged in ARC-AGI-3's shift from model scores to plumbing.

Related from Dispatch