Here's the dirty secret of computer-use agents: they produce mountains of raw trace data — screenshots, mouse movements, keypresses — and almost nobody knows what to do with it. The traces sit in a database, taking up space, containing the entire history of what the agent did and why, and yielding exactly zero structured knowledge.
Jiang, Wang, Chen, and Yang from Stanford dropped "Inducing Task Models from Computer-Use Traces" on arXiv yesterday. It solves this exact problem. They built a system called TMI (Task Model Induction) that takes a raw, unconstrained computer-use trace — the kind your agent generates every time it works — and extracts structured, symbolic task models from it. Not step summaries. Actual models: hierarchical goal decomposition trees paired with control-flow procedures.
The Problem With Raw Traces
Think about what a computer-use trace actually looks like. It's a timestamped stream: screenshot at T+0, mouse to (x,y), click, type "%s", screenshot at T+1, scroll down, click "Submit", screenshot at T+3, realize the form rejected the input, scroll back up, fix field, click "Submit" again. No labels. No task boundaries. And real work interleaves — you're writing a document, then you check email because a notification popped up, then you go back to the document.
Existing methods assume you already know what task the trace belongs to. They produce "step 1: open browser, step 2: navigate to site" — verbose, flat, and useless for understanding how the agent actually reasons about goals and sub-goals.
TMI starts from the opposite end: it discovers the latent tasks first, then induces structured models for each one.
graph TB
subgraph "Input: Raw Trace"
RT["Screenshot stream
Mouse/keyboard events
Timestamps"]
end
subgraph "Stage 1: Task Discovery"
TD["Disentangle concurrent activities"]
TD -->|Task A| SA["Segmented trace A"]
TD -->|Task B| SB["Segmented trace B"]
end
subgraph "Stage 2: Model Induction"
SA --> HM["Hierarchical Objective Model
Goal decomposition tree"]
SA --> PM["Procedure Model
Control flow: loops, branches, sequences"]
SB --> HM2["Hierarchical Objective Model"]
SB --> PM2["Procedure Model"]
end
subgraph "Output: Reusable Skill"
HM --> SK["Structured skill
Organizations can audit & reuse"]
PM --> SK
end
The Two Models in One
TMI doesn't produce a single flat model. It produces two, paired together for each discovered task:
The Hierarchical Objective Model captures recursive goal decomposition. The agent had a high-level goal ("file an expense report"), which breaks into sub-goals ("collect receipts", "fill form", "attach PDF", "submit"), which break further ("click 'New Report'", "type vendor name", "select category"). This is the why structure: what the agent was trying to achieve at every level.
The Procedure Model captures the control flow that organized execution. Loops ("repeat for each receipt"), branches ("if amount > $100, require manager approval"), sequences with error handling ("submit, check for error message, if present fix and re-submit"). This is the how: the actual execution structure, not just a list of steps.
The Numbers
The results are strong across both intrinsic and extrinsic evaluation:
- Task discovery at 0.974 agreement with ground-truth groupings on controlled human and agent trajectories. The system correctly disentangles interleaved tasks — it knows when you switched from writing a document to checking email and back.
- 74.9% of execution steps reconstructed by the induced models, far exceeding the strongest workflow induction baseline. This means the models are faithful: they capture most of what actually happened.
- 30.0% improvement on held-out task accuracy when skills derived from TMI's task models are used by downstream agents. This is the practical payoff: induced skills aren't just descriptive — they're useful.
Limitations
The evaluation is on controlled trajectories and a limited set of tasks. Real-world traces from production agents operating on arbitrary websites will be noisier, longer, and harder to segment. The paper also doesn't address how well TMI handles traces where the agent made catastrophic errors and had to restart entirely — those recovery patterns might look like separate tasks rather than loops within one task. And the current system depends on screenshots for visual context; text-only traces (console logs, terminal sessions, API calls) would need a different approach.
Why Builders Should Care
If you're building or operating computer-use agents, this paper answers a question you've probably been avoiding: what do we actually do with all these traces?
Auditability. Right now, if your agent did something wrong, you replay the trace and squint at screenshots. TMI gives you a structured model you can inspect: "the agent's goal was X, it decomposed into Y and Z, and it executed via procedure P." You can see where the goal decomposition went wrong or where the procedure had a bug.
Skill reuse. Your agent spent 4 hours learning how to interact with a vendor's portal. That knowledge currently lives in the agent's prompt or an ad-hoc skill file — or it doesn't persist at all. TMI extracts a reusable, transferable skill from the trace. The 30% accuracy improvement on held-out tasks is proof this works.
Organizational knowledge. A team of agents doing work produces trace data that nobody reads. TMI converts that data into auditable, versionable, reviewable models. Companies can build libraries of "how we do things here" — not from documentation that's always out of date, but from actual execution traces.
This is the kind of infrastructure work that doesn't make headlines but makes everything else possible. If you're running agents in production, TMI's approach is worth knowing about — and probably worth building toward.
- Inducing Task Models from Computer-Use Traces — Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang, Aug 2026