AL Alfred 2026-09-02 5 min read

LLMs Compute MD5 Across 196 Tool Calls: State Tracking at 5.5B

Most agent benchmarks are messy. They mix instruction following, tool use, planning, and world knowledge into a single score — and when an agent fails, nobody can say why. A new paper from Dheeraj Mohandas Pai and Lu Xian takes a different approach: make the task so clean that any failure is pure bookkeeping.

The Problem: Benchmarks That Don't Debug Themselves

Long-horizon agentic tasks suffer from a fundamental measurement problem. When each step depends on the last, per-step accuracy that looks fine in isolation (say 98%) decays catastrophically over 100+ steps — the end-to-end failure probability approaches 1. Existing benchmarks report success rates but confound state-tracking difficulty with instruction interpretation. They offer no control group that isolates whether an LLM can simply remember a number across many tool calls, let alone transform it correctly.

The paper identifies three confounds in current evaluation: (1) benchmarks mix task complexity with state management, (2) they provide no ground-truth per-step trace, and (3) they're vulnerable to hallucinated final answers that bypass the actual computation. The result: we don't know whether long-horizon failures are failures of reasoning, failures of memory, or failures of the model serving infrastructure.

The Method: MD5 as a Pure State-Tracking Probe

The authors implement MD5 from scratch following RFC 1321 — a sequence of 196 dependent tool calls over 64 rounds, with the model carrying four 32-bit words (a, b, c, d) in its context from one call to the next. Every call is aligned to a ground-truth trace, and the final digest is checked bit-for-bit. There is no ambiguity: either the hash matches or it doesn't.

In the strongest setting, the authors replace every primitive tool (bitwise operations, modular addition, table lookups) with a second LLM — so a driver and a worker compute the entire hash with no exact-arithmetic oracle in the loop. The worker receives the current state and a single MD5 step instruction, and must output the correct updated state. This eliminates any concern about the model exploiting a calculator tool.

graph LR
    A[Driver LLM] -->|"state (a,b,c,d)"| B[Worker LLM]
    B -->|"new state"| A
    A --> C[Memory: stores reasoning]
    C --> A
    subgraph "64 rounds × 3-4 steps each"
        B --> D[Bitwise ops]
        B --> E[Modular add]
        B --> F[Table lookup]
    end

The Results: Most Runs Succeed

gpt-oss-120b, a mixture-of-experts model with only ~5.5B active parameters per token, at temperature 0 with a short fixed prompt, carries the full state across all 196 calls and returns the correct digest on a majority of completed runs. This is not a 70B+ dense model — it's a sparse MoE where most parameters are inactive at any given token.

Two ingredients determine success, and neither changes the weights:

  1. Keeping the model's own reasoning in context each turn. Without this, performance collapses — the model needs its own scratchpad to maintain coherence across steps.
  2. Voting over a thinking-enabled worker to remove modular-arithmetic slips. The worker occasionally makes addition errors; running multiple copies and taking the majority eliminates these without any retraining.

The authors localize residual failures by origin, separating three categories: state-carrying errors (the model loses or corrupts a, b, c, d between calls), arithmetic errors (the computation itself is wrong), and serving errors (infrastructure issues like timeouts or dropped requests). This triage is itself a contribution — it tells builders exactly where to invest in robustness.

Limitations

The paper tests only one model (gpt-oss-120b) at temperature 0, so we don't know how this generalizes across architectures, scales, or sampling strategies. MD5 is a fixed, well-defined computation — real agent tasks involve ambiguous state, partial observability, and open-ended goals that this benchmark explicitly abstracts away. The 196-call sequence is long but still bounded; truly open-ended agents would need to maintain state across thousands or millions of steps. Additionally, the voting strategy works because the computation is deterministic — for creative or generative tasks, majority voting may not converge on the correct answer.

Why This Matters

This paper gives the field something it's been missing: a microbenchmark for agentic state tracking that strips away every confounding factor. If your agent fails the MD5 test, you know the problem is in how it carries state across tool calls — not in how it interprets instructions, not in how it plans, not in tool availability. That diagnostic power is valuable for anyone building long-horizon agents.

The finding that a 5.5B-parameter model can maintain exact state across 196 sequential operations with high reliability suggests that the bottleneck in current agent systems may not be where we think it is. Context management — keeping the model's own reasoning visible to itself — matters more than raw parameter count for this kind of task.

Related Posts