AL Alfred 2026-08-27 5 min read

LocalLSTC: Giving Local GUI Agents Working Memory

What problem does this solve?

GUI agents — models that control a desktop by looking at screenshots and issuing clicks, keystrokes, and scrolls — work well with frontier API models like GPT-5. They fall apart locally. Not gradually — it's a cliff.

Four state-of-the-art GUI frameworks show the same pattern: replace GPT-5 with Qwen3.5-9B and OSWorld success rate (SR-100) collapses from 60.9% to 37.7%. That 23-point gap isn't because the local model can't understand screenshots or plan actions. It's because it can't reconstruct what it was doing from raw interaction history.

Every existing framework hands the model a growing transcript of past observations and actions. The model must figure out the active subgoal, accumulated evidence, and whether the last action succeeded — all from raw text. Frontier models handle this implicit reconstruction. Local models don't. The paper quantifies the result: 91.6% of failed local trajectories contain at least one identifiable control failure.

What's the method?

LocalLSTC externalizes persistent control state — information that crosses step boundaries. The architecture has three pieces:

Critically, LocalLSTC is training-free — no fine-tuning, no adapters, no architectural changes. It's a prompt-and-state-management layer any existing GUI agent framework can wrap.

flowchart TD
    A[Observation] --> B[Short-Term Execution]
    C[Long-Term Control State] --> D[Long-to-Short Planning]
    D --> B
    B --> E[Action]
    E --> F[Outcome]
    F --> G[Short-to-Long Control]
    G --> C
    C --> H{Progress?}
    H -->|Continue| D
    H -->|Recover| I[New subgoal]
    H -->|Terminate| J[Done]

What are the results?

The headline numbers are unambiguous. With Qwen3.6-27B, LocalLSTC achieves:

The gap between LocalLSTC and the prior local state-of-the-art is larger than the gap between the prior local SOTA and the frontier. That's not incremental improvement — that's a regime change.

Ablations confirm both sides of the architecture contribute. Removing L2S or S2L degrades performance, and the effect is larger on harder benchmarks. The architecture also shows increasing returns with longer horizons: the higher the step count, the more value the persistent state provides over raw trajectory conditioning.

What are the limitations?

The paper evaluates with a single backbone family (Qwen3.x). How well LocalLSTC generalizes to other local models — especially smaller 7B variants — isn't tested. The state representation (subgoal + evidence + feedback) is a reasonable default but wasn't optimized. And the architecture is framework-specific in current implementation; a general-purpose API for injecting persistent control state would need separate work.

Most importantly, 64.7% on OSWorld still trails GPT-5's ~60-80% range. LocalLSTC closes the gap but doesn't eliminate it.

Why should someone building things care?

If you're deploying GUI agents on customer hardware — enterprise machines, edge devices, air-gapped environments — you can't rely on API models. Local deployment removes latency, cost, and data-exposure concerns, but introduces a new constraint: the model has to remember what it's doing without a frontier brain to reconstruct context from raw history.

LocalLSTC's key insight is that the bottleneck isn't reasoning — it's memory architecture. The same 27B model goes from near-random to top-tier local performance just by holding an explicit control state. That lesson extends beyond GUI agents: any long-horizon agent on a weaker backbone should care about cross-step state management.

It's also a quantitative reminder that "benchmark = model + system." When someone tells you a local model "can't do" a task, ask whether they gave it the right control structure first.

Related reading