Tencent Drops a 770B Open-Weight Bomb: Hy4 Preview, 1M Context, Apache 2.0

On Friday, Tencent released Hy4 preview — a 770-billion-parameter open-weight model with a 1M-token context window, Apache 2.0, deployed day-one into vLLM and SGLang. The discourse mostly shrugged. That's a mistake.

This is the largest open-weight release of the year — 2.6× the parameters of their own Hy3, shipped with full weights, an FP8 build, prebuilt serving images, and three finetune stacks on launch day. I spent the morning inside it: cloned the repo, downloaded the real checkpoint metadata, loaded the tokenizer, and ran the parameter math by hand. Here's what's real, and what the marketing glosses over.

The specs, straight from config.json

I pulled the actual config.json and model.safetensors.index.json off Hugging Face instead of trusting the README. The architecture:

• MoE, 770B total / 49B active per token. 78 layers; layer 0 is a dense FFN, layers 1–77 route through 256 routed experts + 1 shared expert, top-8 activated.
• 1M context (max_position_embeddings = 1048576), vocab 120,832, untied embeddings.
• Gated DeepSeek Sparse Attention with IndexCache — MLA-style KV compression (q_lora_rank 2048, kv_lora_rank 512) plus an indexer (32 heads, top-2048) that reuses sparse attention indices across layers via the IndexCache trick.
• iHC (identity Hyper-Connections) residual path, 4 streams — the GLM-style trick that makes 78 layers trainable without blowing up the residual norm.
• A native MTP layer (10B total, 0.7B active) for speculative decoding — not an afterthought, it has its own loss term (mtp_loss_factor 0.1).
• Weight files: 131 safetensors shards (1.56TB bf16) plus a 130-shard FP8 build (~780GB). Both up on Hugging Face, ModelScope, GitCode, and CNB. Apache 2.0, no strings.

graph TD
  A[prompt tokens] --> B[embedding 120,832]
  B --> C[Layer 0: dense FFN]
  C --> D[Layers 1-77: MoE]
  D --> E[Gated DSA attention
64 heads + IndexCache]
  D --> F[256 routed experts
top-8 + 1 shared]
  E --> G[+]
  F --> G
  G --> H[iHC residual stream x4]
  H --> I[MTP layer: draft next tokens]
  I --> J[speculative decoding]
  J --> K[49B active per token]
            

What I actually did with it

The GitHub repo is 36MB — configs plus finetune code (DeepSpeed, LLaMA-Factory, MS-Swift). Weights live on HF. So I downloaded the real thing:

git clone --depth 1 https://github.com/Tencent-Hunyuan/Hy4-preview.git
# 36M — finetune pipelines: deepspeed_support/, llama_factory_support/, ms_swift_support/

Then loaded the actual tokenizer and rendered the chat template with transformers 5.16.1:

tok = AutoTokenizer.from_pretrained("tencent/Hy4-preview")
# vocab: 120,000 + 832 special tokens
# "Write a Python function that computes the nth Fibonacci..." -> 17 tokens

The interesting part is how reasoning mode works. It's a system directive encoded as special tokens — no <|thinking|> markers, no hidden chain-of-thought preamble. Just:

<｜hy_start:opensource｜>system<｜hy_middle:opensource｜>
<｜reasoning_mode:opensource｜>reasoning_effort:high<｜hy_end:opensource｜>
...
<｜hy_start:opensource｜>assistant<｜hy_middle:opensource｜><think:opensource>

There are exactly two modes: high (default, deep CoT) and no_think. Pass anything else and the template raises. In no_think the model is handed an empty <think:opensource></think:opensource> pair — the markers stay, it just knows not to fill them. Simon Willison's test of the turbine reasoning trace shows the hidden text is compressed, token-efficient English — same pattern as the rest of the frontier.

One dead end worth reporting: the tool-call instructions in the template contain literal {function-name}, {arg-key-1}, {arg-value-1}, {arg-value-2} placeholders. My first instinct was a leaking Jinja variable. It's not — it's a format spec baked into the prompt, and vLLM/SGLang ship a purpose-built hy_v4 tool-call parser for it.

Does the math hold? Yes

770B is a big claim. I summed the parameter counts from config.json by hand — expert towers (256 experts × 2048 × 6144 × 3 projections across 77 MoE layers), MLA attention, indexers, embeddings:

layers 1-77 MoE   755.5 B   (experts 9.70 B/layer | attn 0.05 B | indexer 0.06 B)
embeddings + head   1.4 B
layer 0 dense       0.5 B
TOTAL backbone    757.5 B   + MTP 10 B = 767.5 B  (claim: 770 B)
BF16 on disk      1.51 TB                          (claim: 1.56 TB)

Within 1% of the claim, with the gap explainable by norms and components I didn't itemize. The BF16 disk size matches too. For a release this size, that verification matters — open-weight flagships live or die on their claims, and these hold. Generation-over-generation: Hy3 was 295B/21B active/256K context/598GB. Hy4 is 2.6× params, 2.3× active, 4× context, 2.6× disk.

What it means

Three things, in order of importance:

• The open-weight frontier is now a China three-way race — DeepSeek, Qwen, Tencent — and none of them are slowing down. Tencent's own comparison data (163 internal experts, 203 engineering tasks, blind) puts Hy4 slightly ahead of GLM 5.3 and Kimi K3, with ~47–51% win rates and ~40% loss rates — meaning it's competitive but not dominant. The west has exited this race entirely.
• Sparse attention is the battleground, and 1M context is now table stakes at the top. Gated DSA + IndexCache is a direct descendant of DeepSeek's sparse-attention line. Every flagship is converging on "compress the KV, index it, reuse it" — because nobody wants to pay attention cost linearly in context anymore.
• This is how a serious open release ships. Prebuilt vLLM and SGLang images, MTP speculative decoding wired into both, FP8 out of the gate, three finetune stacks, benchmark figures ("open-source frontier" in the README, though the actual score tables are JPGs — self-reported, unverified). 49B active in FP8 puts the whole thing inside an 8×H200 rack.

The honest caveats: it's a preview — Tencent ships early on purpose and says so, listing over-reasoning and over-verifying its own work as known issues (Hy3 got meaningfully better under the same process). The benchmark appendix is images, not a reproducible eval harness. And the internal blind eval is directional, not gospel.

Bottom line

Hy4 preview is the real thing: a 770B, 1M-context, Apache-2.0 flagship, weights on open platforms, servable on day one, and its architecture is a clean artifact of the industry's current consensus — MoE at extreme width, sparse attention everywhere, speculative decoding baked into training, shipping early and iterating in public. The story isn't the benchmark score, because nobody has independently verified one yet. The story is that the open-weight frontier keeps getting bigger, faster, and more Chinese — and this week, 770 billion parameters cost you nothing to download and a rack of H200s to run. If you're building on open weights, this is the trajectory to plan around.