Puro-2B: Training a Real LLM from Scratch for $5,090

Pretraining is the last true moat in AI — and the wall isn't talent or data, it's the invoice. Tsinghua's PACMAN group just took a hammer to it. Puro-2B is a from-scratch pretraining recipe that trains a 2B-parameter language model on consumer RTX 5090s — total cost within the price of the card itself: $5,090.

What problem does this solve?

Open-weight models exist. Open recipes — the data, code, and full pipeline needed to reproduce training from zero — effectively don't. The reference points are brutal: training Llama-3.2-3B costs over $1.5M, and reproducing SmolLM3-3B still needs over $700K. At those numbers, pretraining is a corporate privilege. Academic labs can't run controlled experiments on data mixes or optimizers because they can't afford a single from-scratch run. Puro-2B attacks that barrier directly: a complete, repeatable, openly licensed recipe priced for a single research group with one big GPU purchase.

What's the method?

The recipe trains a collection of Puro-2B models from scratch on up to 1.4 trillion tokens in FP8 precision on consumer 5090s, with variants differing in token budget and recipe choices. The efficiency comes from stacking components rather than one magic trick: hardware selection (the 5090's FP8 tensor throughput per dollar), low-precision training done carefully enough to keep loss healthy, a hyperball optimizer, curriculum model averaging across training stages, and a curated data recipe. The clincher for reproducibility: the full pipeline — data, code, and weights — ships under Apache 2.0.

mindmap
  root((Cost to pretrain a ~2-3B model))
    Llama-3.2-3B
      >$1.5M
    SmolLM3-3B
      >$700K
    Puro-2B best model
      <$6.9K compute
    Puro-2B per scaling law
      ~$4.4K

What are the results?

Their best model — compute cost under $6.9K — approaches Qwen2.5-1.5B performance under the authors' evaluation protocol. The more interesting number comes from the second contribution: fitting a Puro Cost Scaling Law across the collection, which relates training spend to average performance. The fitted law suggests roughly $4.4K — less than the titular $5,090 — is sufficient to reach Qwen2-1.5B. That's a 160–340x cost cut against the published reference recipes. The paper also runs an end-to-end case study that open weights alone can't support: how pretraining data curricula shape downstream behavior after post-training.

What are the limitations?

• The performance bar is modest and self-referenced: "approaches Qwen2.5-1.5B" under their eval protocol, against a 2024-era 1.5B model. External standard-benchmark numbers are the missing proof.
• The recipe is tuned to one hardware generation — consumer silicon's FP8 boom. Portability to older GPUs or cloud instances is an open question.
• The scaling law is fitted inside a narrow, 2B-class collection. Don't read $4.4K as a promise for 7B or 70B.
• Cost accounting (owned hardware vs. cloud, energy, the $5090 GPU itself) isn't fully broken out from the abstract-level compute figure.

Why should someone building things care?

Because $5K from-scratch pretraining changes what's worth trying. It turns pretraining from a corporate ritual into an experiment a lab can run and re-run: curriculum studies, optimizer ablations, data-mix audits — all on your own pipeline, not someone else's weights. That full-pipeline access is the quiet revolution here; it's exactly what enables the paper's own curriculum-to-post-training study. For builders shipping domain products, a 2B model trained from scratch under your own data, with Apache 2.0 provenance, is now a real option instead of a finetune of someone's grey-area scrape. And for everyone else, it resets the Overton window on what small-model pretraining should cost — which is the kind of number that makes big labs wince.