Desert Ant Redact Tested: 100% on Cards, 19% on Cities

Desert Ant Labs launched Tuesday with a striking benchmark table: their Redact model, 12MB, catches 88.8% of personal data in text — a 2.3GB GLiNER-PII manages 91.1%, and OpenAI's 3GB moderation filter only 60.2%. Those are their numbers, on their data. I couldn't test on-device — their SDK targets Swift, Kotlin, and JS — but the weights are on Hugging Face as a LiteRT .tflite. So I downloaded Redact, ran it on a plain Linux CPU with LiteRT, and built a 120-sample synthetic PII suite with 874 planted entities. Here's the scoreboard.

How I ran it

The model is a 384-hidden BERT token classifier (89 BEBIOS-tagged labels, 31,475-token vocab) exported to TFLite. The HF redact.tflite file is 24.5MB, not the 12MB in their launch post — that figure is presumably the quantized/ANE build for Apple silicon. On x86 you're running the float32 export.

uv venv venv && uv pip install ai-edge-litert tokenizers numpy
curl -L https://huggingface.co/desert-ant-labs/redact/resolve/main/redact.tflite
# inputs: input_ids [1,256] int32, attention_mask [1,256] int32
# output: [1,256,89] float32 logits — 26.6ms first call, ~24ms median after

Decoding is straightforward: argmax over the 89 tags, stitch B/I/E/S tags into spans, drop ORG (their config disables it by default), then normalize their label taxonomy (ZIP_CODE→ZIP, STREET_NAME+BUILDING_NUMBER→street components, GIVEN_NAME+SURNAME→person). Entity-level matching, not token-level — the strict-label version of this eval scores F1 0.508 purely on taxonomy mismatches, which tells you how much evaluation hygiene matters when a vendor ships 89 granular labels.

The scoreboard

120 synthetic samples, 874 planted PII entities across 10 types (emails, phones, cards, SSNs, IPs, URLs, names, cities, ZIPs, street addresses), realistic prose around them. Label-mapped, overlap-matched:

REDACT(mapped) n=120  P=0.932  R=0.743  F1=0.827  (tp=649 fp=47 fn=225)
latency mean=24.6ms  median=23.9ms  p95=28.9ms  (4 threads, x86 CPU, 256 tokens)

Per-type recall is where the story is:

• EMAIL, PHONE, CREDIT_CARD, SSN: 100% (480/480). Every single one caught.
• IP_ADDRESS: 87% — it misses private-range IPs and once tagged the "10." of an IP as SSN.
• ZIP: 83%, STREET: 59% — building numbers and street names often dropped or split.
• PERSON: 42% — full names over the two models' vocabularies slip through.
• CITY: 19% (23/120), URL: 17% (4/24). Near-blind on both.

So the honest split: hard identifiers are perfect, soft entities are a coin toss or worse.

Soft entities are the gap

Cities are the loudest failure. Hamburg, Lyon, Aarhus — plausible European city names, in exactly the languages Redact ships for — went unflagged 97 times out of 120. Names like Ingrid Garcia and Anna Mueller likewise. That pattern looks like a training-data artifact: the model has likely memorized PII-shaped formats (digits, @-signs, dashes) rather than learned open-vocabulary entity recognition. A regex baseline on the same data scores F1 0.598 with precision 0.779 — Redact beats it decisively overall (0.827), and its precision (0.932) is genuinely excellent, but the categories where regex fails least are the categories where Redact fails most.

Two caveats before you quote my numbers. First, my suite is synthetic and out-of-distribution by design — fictional names, cookie-cutter streets — which flatters regex and stresses learned models. Desert Ant's 88.8% was presumably measured on realistic corpora; my F1 0.827 on hostile synthetic text is not a contradiction, it's a different probe. Second, their config recommends a sigmoid threshold of 0.6 with 64-token striding for long documents; I ran raw argmax on ≤256-char inputs, so boundary entities in longer texts would need their chunking layer. I'm reporting the raw model, not the full SDK pipeline.

Bottom line

Redact delivers most of what the launch claims, with one honest asterisk. If your PII problem is the compliance-shaped one — emails, card numbers, SSNs, phones leaking through text fields — this is a 24.5MB file that catches 100% of them at ~24ms per 256-token window on a CPU with no server round-trip. That's a legitimately useful artifact, and the label taxonomy is careful. But treat "88.8% of personal data" as scoped to identifiable-data detection, not a general NER replacement: it will wave a person's full name and home city straight through. Ship it for the formats, keep a human (or a bigger model) for the humans.