AgentJIT Teardown: 2,945x Claim, Silent Wrong Answers

AgentJIT hit Show HN this morning promising to "compile flaky, 30-second multi-step AI agent workflows into 5-millisecond deterministic code" — 1000x+ speedups, zero tokens on the hot path. I cloned it, installed it, ran the official benchmark, and then did what the benchmark doesn't: fed the compiled pipeline inputs it wasn't traced on. It answered wrong silently, 100% hit rate, zero bailouts. And the thing it's supposedly speeding up — the LLM — doesn't exist anywhere in the code.

What AgentJIT Actually Is

The pitch: decorate an agent with @jit, run it once, and AgentJIT traces the trajectory, compiles it to straight-line Python, and replays it forever — no LLM calls, no token spend, ~0.016ms per call. It's 1,150 lines of Python, Apache-2.0, tests pass (9/9 in 0.09s). The mechanics work exactly as advertised:

# my run, first call = trace + compile, subsequent = replay
python examples/01_quickstart.py
# Result: {'item': 'Mechanical Keyboard', 'total': 118.99}
# compiled_hit_rate: 100.0   total_tokens_saved: 7500

Read the generated code and the mechanism is obvious — it's capture-replay with type guards, not a compiler:

# generated by AgentJIT, verbatim
step_1_out = _tools['get_price'](name=item)
step_2_out = _tools['discount'](price=step_1_out['price'])
return {'name': item, 'price': step_2_out}

There is no model, no prompt, no API key, no completion call in the entire src/ tree. The tracer has an is_llm_call: bool = False flag that is never set to True anywhere. The "agent" is any ordinary Python function; the LLM story is aspirational scaffolding — an integration helper that generates OpenAI-compatible tool schemas for you to wire up. The claimed savings are real only if you bring your own model.

The 2,945x Benchmark Is Against a Fake Baseline

I ran the official suite. It reports exactly what the README promises:

Mean Latency   |  45.98 ms  |  0.0156 ms
SPEEDUP FACTOR: 2945.3x FASTER
TOKEN SAVINGS:  100.0%

Look at how the baseline is built:

# benchmarks/benchmark_speedup.py — the "uncompiled agent"
def uncompiled_agent(query, level):
    time.sleep(0.015)   # simulating 15ms LLM reasoning
    doc = query_vector_db(query=query)
    time.sleep(0.015)
    ...

The 2,945x is time.sleep(0.015) divided by real compiled latency. Sleep measured against no-sleep. The badge says 1000x+, the benchmark dutifully delivers 2945.3x, and neither number describes any real system. To AgentJIT's credit, the README is upfront that these are simulated LLM overheads. But a benchmark whose baseline is synthetic should never ship with a "2945.3x FASTER" verdict banner. The honest number for the compiled replay is 0.0156ms mean — that part I verified.

The Bug: Baked-In Branches, Silent Wrong Answers

Guards check only argument types. Anything the trace decided — a branch taken on tool output, a lookup key, an error path — is compiled into the pipeline permanently. So I built an agent with a data-dependent conditional: discount items over $100, surcharge everything else. I traced on a $299.99 item, then called with cheaper items:

# compiled from a trace on the EXPENSIVE branch
agent("Expensive Item")  # -> 269.99  correct (299.99 * 0.9)
agent("Cheap Item")      # -> 8.99    WRONG (should be 10.99: 9.99 * 1.1)
agent("Mid Item")        # -> 49.50   WRONG (should be 60.50: 55.0 * 1.1)
# stats: {'bailouts': 0, 'compiled_hit_rate': 100.0}

Two of three answers wrong. Zero bailouts. Telemetry reports 100% success — the failure mode is indistinguishable from a healthy pipeline. In the marketing example this is a pricing agent; swap the domain for claims, dosage, or authorization logic and you have silent wrong decisions at scale, with dashboards that glow green. The second problem is worse: there's no re-trace policy. A guard failure falls back to self.func — the original body — which in a real deployment means re-running whatever slow path you were trying to escape, once, with the very same baked trajectory. It never re-compiles.


graph TD
    A[Call @jit agent] --> B{Compiled?}
    B -- no --> C[Trace one run
compile after warmup_runs=1]
    C --> D[Replay: guards only
check arg types]
    B -- yes --> D
    D --> E{Guard passes?}
    E -- no --> F[Fallback: rerun raw body
never re-traces]
    E -- yes --> G[Replay traced trajectory
baked-in branch may be wrong]
    G --> H[Stats: 100% hit rate
0 bailouts]

Verdict: A Deterministic Cache Wearing an Agent Costume

Strip the branding and AgentJIT is a competent capture-replay memoizer for pure Python tool pipelines: one warm call to learn the call sequence, then sub-millisecond replays with type guards. For workflows that are genuinely trajectory-stable — fixed call order, no data-dependent branching, side-effect-free replay — that's a real 2,000x-plus over rerunning them. But "JIT compiler for AI agent trajectories" describes an aspiration the 1,150 lines don't implement: there is no LLM in the loop, no policy for divergent trajectories, no verification that replay outputs match what a fresh run would produce. In my branching test, it scored 1 of 3 correct and reported 100% success doing it.

Bottom line: if your "agent" is a deterministic Python function, you don't need a JIT for it — cache it. If it actually routes through a model with branching decisions, AgentJIT will confidently replay yesterday's trajectory and tell you everything is fine. The dangerous product isn't the slow one; it's the fast one that's wrong and knows it isn't.

Related posts