AL Alfred 2026-09-05 4 min read

Speculative Macro Commit Slashes Agent Latency: 44.9% on AppWorld

Every tool-calling LLM agent has the same bottleneck: wall-clock time spent waiting. Not just on model inference — on the serial turn-by-turn grind of call → execute → observe → think → call again. Each tool invocation, environment transition, and observation observation adds a delay that compounds over a long trajectory. A new paper from Liu, Kundu, and Beerel (accepted at MLSP2026) introduces Speculative Macro Commit (SMC), a runtime mechanism that cuts agent wall time by up to 44.9% by having a fast drafter model pre-execute multi-step action chains before the main actor model has even decided what to do next.

The Problem: Agent Latency Is More Than Inference

Standard agents execute a fixed loop: the LLM generates a tool call, the environment runs it, the observation comes back, and only then does the model decide the next action. Each turn is a full round-trip through inference + execution + observation. Prior work on speculative decoding sped up token generation within a single LLM call, but that addresses only one component of the latency budget. The rest — serial action-observation turns — was left untouched. The Speculative Actions (SA) baseline extended the idea to single-step tool speculation, but it only skips one inference step at a time. The real gains lie in predicting entire multi-step action sequences and committing them as a batch.

How SMC Works

SMC runs a two-tier agent system. A large authoritative model (Qwen3.5-27B in INT4) produces the official trajectory as usual. In the background, a fast drafter model (Qwen3.5-4B) continuously predicts and executes future action chains on an isolated environment snapshot — a sandboxed copy that doesn't affect the real world.

The key insight is a macro library. SMC mines training traces for recurring multi-action skeletons — sequences like search for a contact, retrieve phone number, initiate call — and stores them as reusable macros. At runtime, the drafter's predicted chains are matched against this library. When the actor model issues a tool call that matches the first action of a drafted macro, SMC commits the remaining pre-executed steps and their observations directly into the official trajectory, skipping both inference and execution time for those actions.

mindmap
  root((SMC Architecture))
    Actor Model
      Qwen3.5-27B INT4
      Official trajectory
      Produces next tool call
    Drafter Model
      Qwen3.5-4B
      Predicts action chains
      Executes on snapshot
    Macro Library
      Mined from training traces
      Multi-action skeletons
      Match at runtime
    Commit Decision
      Actor call matches draft
      Commit remaining steps
      Skip inference + execution

The approach extends beyond single-step speculation because the macro library captures structure that repeats across episodes — sequences the drafter would never generate correctly from scratch but can execute reliably once the first action is confirmed.

Results: 44.9% Wall-Time Reduction on AppWorld

The authors benchmarked SMC against sequential execution and the Speculative Actions (SA) baseline on two agent benchmarks:

The larger gain on AppWorld reflects the benchmark's longer, more structured action sequences — exactly the setting where multi-step macro speculation provides the most leverage. The small accuracy drop on AppWorld (not reported as a specific percentage but noted as "small reduction") suggests that not all committed draft actions are optimal, but the latency savings still justify the tradeoff.

Limitations and Open Questions

SMC has several limitations worth noting. First, it requires training traces to mine the macro library — cold-start scenarios with no prior data won't benefit. Second, the isolated environment snapshot adds memory overhead proportional to the number of parallel draft chains. Third, the macro matching is heuristic; a false-positive commit (committing a drafted action that diverges from what the actor would have chosen) degrades accuracy. The authors report a small accuracy drop on AppWorld, confirming this risk.

Additionally, the approach was tested with only one model pair (Qwen3.5-27B/4B). It's unclear how the speed-accuracy tradeoff shifts with different actor-drafter size gaps or different model families. The paper also doesn't explore what happens when the environment state diverges between the snapshot and reality — a concern for non-deterministic tools or shared resources.

Why This Matters

For anyone building production agent systems, latency is a feature, not a metric. Users abandon agents that take too long. Tool-calling agents are already being deployed for customer support, code review, data analysis, and automation — environments where sub-second response deltas translate directly to user satisfaction and task throughput.

SMC demonstrates that the latency budget for agents isn't fixed. By treating tool execution as a speculatively parallelizable pipeline rather than a serial chain, we can reclaim nearly half the wall time on realistic benchmarks. The approach is practical — it requires no model retraining, no architecture changes, and runs on commodity hardware. The macro library concept is also general: any domain with recurring multi-step action patterns (API orchestration, UI automation, database queries) could benefit.

The code is open-source on GitHub. If you're building agent infrastructure, this is worth a read and a fork.

Related Reading