I Tried to Break the Codec That Turns Bytes Into One-Token Words — It Held

There is exactly one kind of data that spends its whole life inside a language model's context window: identifiers. Correlation IDs, nonces, request IDs, session tokens — they get minted, dumped into prompts and logs, and asked back. And until today, everyone encoded them with hex or base64, encodings built for humans and bytes, not for tokenizers. The result: a 16-byte value that costs anywhere from 16 to 27 tokens depending on which 16 bytes it happens to be.

Unigram, posted to Hacker News this morning, is a Rust crate built on a different premise: 256 words, each of which is exactly one token in every major tokenizer family, mapped 1:1 onto bytes. One word is one byte is one token. N bytes cost exactly N tokens — flat, for every value, forever. I cloned it, built it, ran its test suite, then tried to break the claim with my own independent checks. It held.

The Design

The whole thing is a table. 256 lowercase English words, 4 to 10 characters, frozen forever (a test pins the table's digest, because changing one entry changes what every previously issued value decodes to). Byte n maps to ALPHABET[n], the join is a space, and decoding is the inverse lookup. One dependency at runtime: the OS CSPRNG.

use unigram::UnigramId;

let id = UnigramId::from_bytes([0x3d, 0x9a, 0x00, 0xff]);
assert_eq!(id.to_string(), "created office access world");
assert_eq!(UnigramId::<4>::parse("created office access world").unwrap(), id);

That's the entire API surface for the basic case: from_bytes, to_string, parse (canonical), recover (tolerant of what a round-trip through a model does to case and separators), plus a CRC-8 CheckedUnigramId variant for when a mutated value must not pass as valid.

The space join is the subtle part. Tokenizer vocabularies store their canonical word entries space-prefixed, so the space between two words is absorbed into the word that follows it and costs nothing. That's why the property composes: the crate measured all five tokenizer families and every other separator (_, -, ,, newline) costs as much as the payload itself. Space is the only free join.

graph LR
  A["16 random bytes"] --> B["table lookup
1 byte = 1 word"]
  B --> C["'links change points high
random found season events'"]
  C --> D["tokenizer"]
  D --> E["exactly 16 tokens
(any family, any value)"]
  F["hex of same bytes"] --> G["16-27 tokens
depends on the value"]

The Claim, and the Culture Around It

The bold part of the README isn't the average case — it's the flat column. Hex on 4 bytes averages 6 tokens under Claude but spikes to 8; 32 bytes of hex swings from 37 to 49 tokens under the GPT vocabularies. A token budget built on an encoding like that has to be provisioned for the worst case, because you can't know the cost until the value exists. Unigram's cost is known before minting. Always N.

What sold me on this crate isn't the property, though. It's that they shipped the failure mode in the changelog. Version 0.2.0 had 22 alphabet entries that cost two or three tokens bare (at the start of a string), and its verifier tested one convenient payload whose opening word happened to be cheap. Both were fixed. The current verifier (verify-alphabet.py) sweeps all 256 entries through the opening and closing positions of every context a value actually sits in — start of string, JSON, prose, markdown backticks, open parens — and exits non-zero naming every offender if any claim stops holding.

What I Actually Ran

First, the crate itself:

$ cargo build
$ cargo test
# 24 unit tests + 3 doc tests, all pass
# includes: every byte round-trips, no two entries within one edit,
# the alphabet matches its frozen digest, a single-word substitution
# is always caught

Then I stopped trusting their tests and wrote my own. Parsed the alphabet straight out of src/lib.rs (no copying the table — a verifier that checks a stale copy is worse than none), then:

• Bijectivity: all 256 single bytes round-trip; 10 adversarial patterns — all zeros, all 0xFF, incrementing, decrementing, null-terminated, ASCII text, pseudorandom — all round-trip, and word count equals byte count every time.
• Token cost, my measurement: every one of the 256 words is exactly 1 token, both space-prefixed and bare, under all four tiktoken vocabularies I could pull (cl100k_base, o200k_base, p50k_base, r50k_base).
• The flat column, empirically: 24 deterministic payloads per family across sizes 1–32 bytes. Every single one cost exactly N tokens.

[3] Token cost per word (marginal over carrier 'the'):
    cl100k_base : spaced 256/256, bare 256/256  [OK]
    o200k_base  : spaced 256/256, bare 256/256  [OK]
    p50k_base   : spaced 256/256, bare 256/256  [OK]
    r50k_base   : spaced 256/256, bare 256/256  [OK]

[4] Cost comparison (cl100k_base, marginal, 64 samples per size):
     Bytes |   Unigram |   Hex mean/max |    B64 mean/max
       4 B  |    4 flat   |    5.5 /   8    |    4.6 /   7
       8 B  |    8 flat   |    9.9 /  14    |    8.1 /  11
      16 B  |   16 flat   |   18.9 /  24    |   15.9 /  19
      32 B  |   32 flat   |   37.2 /  45    |   30.7 /  35

Then I ran their verifier, which adds the two families I can't reproduce offline: Llama's SentencePiece tokenizer and Claude's (via the offline ctok reconstruction, itself audited against Anthropic's official count_tokens endpoint). Final line, exit code 0:

OK: all 256 entries hold every cost claim, in every family checked.

Where It Breaks

I found the cracks, and to their credit, they're documented. Three of them matter:

• Base64 beats it above 16 bytes under GPT vocabularies. Look at my own table: 32 bytes of base64url averages 30.7 tokens under cl100k_base, below unigram's flat 32. GPT tokenizers have memorized base64 fragments. Unigram wins on the worst case — 35 max vs 32 flat — which is the point for budgeted systems, but the average case is honestly a wash at digest sizes.
• It was built for nonce widths, not digests. The README says it plainly: a 32-byte digest is 224 characters across three wrapped lines. This is for correlation IDs and request IDs, where a human reads them off a screen.
• The claim is pinned to specific tokenizer artifacts. Not "Llama 3" (which tokenizes with tiktoken, not the SentencePiece model checked), not "whatever Anthropic ships next year." The vocabulary versions are pinned by hash in the verifier. When a tokenizer changes, the claim must be re-verified — and the tooling to do that is right there in the repo.

One more honest note: the punctuation surcharge is real but constant. A backtick or open paren immediately before a value adds exactly one token, once, regardless of payload size. That's the punctuation paying for itself, and the sweep proves it never scales.

Bottom Line

Unigram is a 1,171-line Rust file plus a verifier, and it's the most rigorous little crate I've benchmarked in a while. The property is real — I tried to break it from four different angles and couldn't — and the engineering culture is the part worth stealing: they measured the failure they shipped, then built a verifier that sweeps the entire alphabet through every context so it can't happen again. If you ship identifiers into prompts, this is a free 25-40% token cut on the worst case, and the flat cost means you can budget for it before the value exists. At nonce and correlation-id sizes, it's strictly better than hex in every family I could test. For 32-byte digests, use base64 and accept the variance. For everything an agent will ever read back to you — this is the encoding.