Does agent memory survive a model upgrade?

The store persists but accuracy doesn't. In a controlled study swapping Llama-3.1-8B and Qwen2.5-7B behind a fixed memory store, model-written notes shifted accuracy by +9.91 points in one migration direction and -13.28 in the other, while a fixed-schema knowledge graph drifted only +0.0004.

Why does free-text agent memory break when the model changes?

Free-text notes couple the reader to the writer's idiom. The study's decomposition attributes 80% of the NOTES accuracy deficit to information lost during initial construction — the writer omitted evidence spans (Llama retained only 64.5% versus Qwen's 85.6%) — not to the new model's reading. Rewriting note style changed nothing.

Is it safe to mix old and new embedding vectors in one index?

No. Upgrading bge-large-en v1.0 to v1.5 — both 1024-dimensional — a 50/50 mixed index captured only +4.96 points of the +11.90 available from full re-embedding, forfeiting nearly 60% of the upgrade gain. Because dimensions match, no compatibility error is thrown; retrieval degrades silently.

Which memory format is most portable across models?

A fixed-schema knowledge graph: accuracy changed by just +0.0004 (per-reader -0.11/+0.02 pp) after a writer swap, versus swings of up to 13.28 points for free-text notes. The schema pays interpretation cost at write time. Caveat: it can only represent facts the schema covers, and reader capability still differs (0.8456 vs 0.9878).

How do you repair an agent's memory after a model swap?

Keep the raw source history. Store-only rewriting of notes reached the 90% recovery target in 0 of 48 histories in both directions, while raw-history reconstruction recovered 34 of 48 (median cost $0.76) — and only when the repair model could use it: the weaker repairer hit output-token limits in every attempt. RAG re-embedding (48/48, ~$0.013) and schema rebuilds (~$0) are mechanical and near-free.

Agent Memory Doesn't Survive Model Upgrades: -13.28 Points

The memory store survives the swap. The memory doesn't. A new controlled study swapped Llama-3.1-8B for Qwen2.5-7B behind a fixed memory store and watched accuracy move by +9.91 points in one direction and −13.28 in the other — effects that cancel to a harmless-looking average of zero. Free-text notes written by one model actively poison the next; fixed-schema memory drifted 0.0004.

What Breaks When You Swap the Model?

Agent memory is built on an unstated assumption: the store is durable infrastructure, the model is the replaceable part. Teams swap models weekly; the store persists. Goyal and Ray's paper tests what that swap does. A new model may read old notes through its own idiom, mixed embedding versions can corrupt retrieval without throwing an error, and repair may be impossible without the original evidence. Every serious agent inherits these risks on its next model bump.

One History, Four Memory Formats, Two Models

The design is clean. 48 synthetic histories carry randomized answer codes, so chance accuracy is near zero and nothing can leak from pretraining — exact scoring, no LLM judge. The same history is stored four ways: verbatim for long-context reading (LC-RAW), chunked for RAG (bge-large-en, top-8), compressed into model-written notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M each write and read every store, in both directions, with four hypotheses and a five-point action threshold locked in a signed Git tag before data collection.

flowchart TD
    H["One history, 48 variants with randomized answer codes"] --> A["LC-RAW: verbatim transcript"]
    H --> B["RAG: chunks + bge-large-en index"]
    H --> C["NOTES: model-written summary"]
    H --> D["KG-fixed: fixed-schema graph"]
    A --> S["Model swap: Llama-3.1-8B vs Qwen2.5-7B"]
    B --> S
    C --> S
    D --> S
    S --> O1["KG-fixed: -0.11 / +0.02 pp"]
    S --> O2["NOTES: +9.91 / -13.28 pp, average near zero"]
    S --> O3["50/50 mixed index: +4.96 of +11.90 pp, silent"]
    S --> O4["Repair: raw history 34/48, store-only 0/48"]

The Numbers: +9.91, -13.28, and an Average That Lies

KG-fixed is the control that works: Llama scores 0.8456 on its own graph and 0.8445 on Qwen's (−0.11 pp); Qwen holds 0.9878 → 0.9880 (+0.02 pp). NOTES is where the story turns. Qwen reading its own notes: 0.4719; Llama's: 0.3391 — −13.28 points. Llama on its own: 0.3762; Qwen's: 0.4753 — +9.91 points. Averaged, NOTES looks portable. It isn't. Qwen's notes retained 85.6% of required evidence spans in 143 KiB; Llama's kept 64.5% in 159 KiB. Equal byte budgets, unequal memories.

80% of Note Loss Happens at Write Time

The decomposition is the part you can't get from the abstract. Of NOTES' pooled deficit (0.584), store construction causes 80% (0.467); in-store retrieval just 6%, the reader residual 14% — 88.8% construction in the worst direction. RAG inverts: retrieval causes 81% of its 0.450 deficit; construction is 1%. After a swap degrades answers, the reflex is to blame the new reader — but the reader is the smallest slice in both formats. Rewriting notes in the new model's style changed nothing (−0.012); Llama's rewrites dropped exact identifiers in 46% of attempts. Missing content, not unfamiliar phrasing. Meanwhile the embedding upgrade — bge v1.0 → v1.5, both 1024-dimensional — runs without one compatibility error: full re-embedding gains +11.90 points, a 50/50 mixed index captures only +4.96, forfeiting nearly 60% of the upgrade silently. Every fleet doing rolling re-embeds is in that state right now.

What the Study Can't Tell You

One migration pair, both models under 10B — frontier readers with tool access and long contexts are untested, and the LC-RAW controls already show a 20-point reader gap (0.7121 vs 0.9109) that will shift with scale. The histories are synthetic slot-filling: declarative recall, not the preferences and procedural habits agents actually accumulate — which may be even more idiom-sensitive. KG-fixed reads through graph traversal, so it isn't format-matched to NOTES, and a schema can only represent what it covers. Recovery is measured against own-store performance, not ground truth, and no hallucination metric exists — a swap could inflate confident wrong answers while accuracy barely moves. Llama's 0/48 repair is partly an output-token-limit artifact the authors flag themselves.

The Migration Protocol for Agent Memory

Four rules fall straight out of the measurements. Never mix embedding versions in one index — isolate spaces and fully re-embed; the mixed index cost 6.94 points of available gain and failed silently because dimensions matched. If memory must outlive model swaps, prefer fixed schemas: pay interpretation cost at write time, validated, instead of deferring it to every future reader. Retain the raw source history — the only repair path for compressed notes (34/48 histories recovered at the 90% target, median $0.76, versus 0/48 for store-only rewriting); you cannot rewrite what was never stored. And test migration in both directions, measuring the stage before the reader: evidence coverage for notes, recall@k for RAG. The Funes launch and every memory-vendor pitch this month assume the store is portable. This paper is the warning label.

Verdict: receipts, not rhetoric. Pre-registered, contamination-proof, direction-aware — and the average-to-zero store is the one that silently breaks half your users.