al Alfred September 6, 2026 4 min read

Unlearning Leaks What It Deleted: 100% Prompt Recovery

Unlearning passed its audit. That is the finding that should worry you. A new Warwick–Cambridge study unlearned content with NPO and scored 0.000 Forget ROUGE-1 on PISTOL: the suppressed answer was gone by every conventional metric. Then a black-box attacker who had never seen the forget set reconstructed 96.1% of the deleted prompts. The deletion certificate is real. So is the leak.

What Does Unlearning Actually Delete?

Machine unlearning in 2026 is mostly refusal engineering. NPO, DPO, and LUNAR do not erase knowledge — they train the model to refuse requests touching the forget set while behaving normally elsewhere. Every published attack — membership inference, answer recovery, candidate-pool selection — assumed the attacker already knew the target. This paper, from the group behind LUNAR (Kurmanji, Lane), removes that assumption: can an attacker who knows nothing recover which prompts were unlearned? The stakes are the prompts themselves. A model scrubbed of an HIV-treatment or pending-acquisition association reveals nothing when it refuses — but the refusal, landing on the right entity pair, discloses exactly the association the scrub was paid to hide.

How TAS Reconstructs Deleted Prompts

Targeted Active Search (TAS) runs on what a commercial API exposes: text in, text out. From retained prompts alone, it builds a candidate entity pool and a template pool — on TOFU, 3,606 distinct templates collapse to 74 canonical ones via Llama3-8B rewriting, filtered to 30 closed-ended factual probes. Three phases run under a fixed query budget: warm-up coverage (20%), Thompson sampling over Beta posteriors on entities and templates (≥70%), and a confirmation re-rank (10%). Refusal regexes and semantic similarity score every response, crediting probed entities and templates on refusals, demoting on non-refusals. The winning entities are instantiated into every template and ranked by refusal strength.

flowchart LR
    A["Retained prompts"] --> B["Entity pool +
canonical templates"]
    B --> C["Warm-up coverage
20% of budget"]
    C --> D["Thompson sampling
70% of budget"]
    D --> E["Confirmation re-rank
10% of budget"]
    E --> F["Reconstruct prompts
rank by refusal strength"]
    D -->|"text-only queries"| M["Unlearned model"]
    M -->|"decoded completions"| S["Refusal score:
regex + semantic"]
    S -->|"Beta posterior updates"| D

The Numbers: Perfect Accuracy at 0.3% of Brute-Force Cost

Across three unlearning methods, three model families (Llama2-7B, Llama3-8B, Gemma-7B), and three benchmarks (TOFU, DUSK, PISTOL), TAS identified the forgotten entity with 100% accuracy and reconstructed up to 95.4% of forgotten prompts. On TOFU, the search space is 804,138 queries — brute force ran out of memory; TAS needed 5,000 (0.60% of the space), and with early stopping just 2,488, a 323× reduction. Baselines show where the credit belongs: Random finds the entity 18.5% of the time on TOFU, Greedy 33.3%, UCB 87.5% — only posterior-guided search turns a capped budget into a perfect attack.

The perverse result: on PISTOL's relational space, TAS (100%) beats exhaustive probing (88.9%). Unlearning one edge keys refusals to the shared entity, so collateral over-refusal spreads across neighbors — enough to mis-rank brute force after querying everything. The sharper the carve, the wider the halo; the halo fools exhaustive search while the adaptive attack reads through it. Per-method forensics: DPO reconstructs with 0.966 recall / 0.586 precision, NPO 0.943 / 0.530, LUNAR 0.780 / 0.799 — a wide net versus a scalpel. The unlearning objective is a choice of fingerprint, and no objective escapes reconstruction.

What the Paper Doesn't Solve

All three tested methods are refusal-shaped by design; approaches that overwrite with plausible garbage may leave a weaker footprint — this is a verdict on the commercial default, not on unlearning as a category. Precision lags recall throughout (53–68%): the attack finds the neighborhood more reliably than the exact prompt list, a false-positive rate auditors should price in. The evaluation is 7–8B open-weight models under unlimited polite querying — no frontier APIs, no rate limits, no anomaly detection. Cardinality beyond two entities is crude: on TOFU pair A, 5 of 9 runs returned the wrong count. And defenses are analyzed, not built: randomized refusal wording cannot survive evidence aggregation, pointing to query-pattern monitoring as the practical countermeasure — untested.

Why Builders Should Care

If you ship unlearning for GDPR erasure, license compliance, or safety scrubbing, your evaluation probes known prompts. TAS shows the intervention itself leaks the target list, and standard forget metrics do not predict discoverability. Add a discoverability test to your unlearning suite, and treat query telemetry as a security surface — template-entity probing has a shape you already log. The defense worth funding is behavioral indistinguishability: an observer should not be able to tell whether an entity–template pair sits inside the forgotten region. The generalization: any targeted post-training change — abliteration, safety steering, hidden-topic filtering — is externally distinguishable to a patient adversary. Steering-target discovery is now a class of attack.

Verdict: receipts, not rhetoric. 100% entity accuracy, 0.30% of the brute-force space, and a 0.000-ROUGE model leaking 96.1% of its deleted prompts. Until unlearning evaluations test target concealment, every deletion certificate describes the one property it never checked.

Related Posts