Tencent Hy4 — A 770B Open-Source MoE Model Hits the Frontier
Tencent dropped Hy4 preview yesterday — a 770B-parameter Mixture-of-Experts model with 49B active per token and a 1M-token context window, licensed under Apache 2.0. It's open-source, it's on HuggingFace, and it's pushing the open-weight frontier hard.
Some architecture highlights:
- 256 routed experts + 1 shared expert per MoE layer, top-8 activated per token
- Gated DeepSeek Sparse Attention with IndexCache for cross-layer sparse index reuse
- iHC (identity Hyper-Connections) expanding inter-layer information flow through residual pathways
- Native MTP layer (10B params, 0.7B active) for speculative decoding
- 78 layers, 64 attention heads, 6,144 hidden size
The MoE arch is clearly inspired by DeepSeek's playbook — Gated DSA is a direct descendant of their sparse attention work — but Tencent layers on their own innovations (iHC, IndexCache) and ships with a 1M context window out of the gate. That context length alone puts Hy4 in rarefied company.
The license is the headline: Apache 2.0. No usage restrictions, no commercial gotchas. That's the same openness that made Llama and DeepSeek viral, and it signals Tencent is serious about ecosystem play rather than walled-garden monetization.
Hy4 preview is available now via vLLM and SGLang, with fine-tuning and quantization paths documented. The model comes in FP8 and full-weight variants on HuggingFace.
Why it matters: The open-source frontier has been a three-horse race between Meta (Llama), DeepSeek, and Mistral. Tencent just bought a ticket. A 770B MoE from a company with this kind of compute and distribution changes the calculus — expect derivative fine-tunes, quantized versions for local inference, and a new benchmark leaderboard contender within weeks.
The arms race isn't slowing down. It's getting another driver.