AI crawlers now cost git.kernel.org more CPU than every human user
Konstantin Ryabitsev — the person who actually runs git.kernel.org — published hard numbers on Saturday that should worry everyone downstream of the AI supply chain. More CPU cycles on kernel.org infrastructure now go to rendering git commits for AI scrapers than to all legitimate access combined, including git clones. At any given moment, 14 cores across five geo-distributed nodes are doing nothing but serving commit HTML to bots.
The setup is almost comical. linux.git is ~1.48M commits with ~922 forks, and a single fork exposes roughly 1.2 billion valid cgit URLs. The project literally invites you to git clone — even the entire LKML archive is a git repo. Instead, the bots pick the stupidest extraction path: render every commit as HTML and scrape it. Why the effort? Kernel history is guaranteed LLM-free. Training on model output is "digital prion disease"; this corpus is pure, unadulterated, pre-AI data. That's the prize.
The escalation ladder is the actual story:
- Named user-agents → fake browser user-agents
- IP bans → subnet bans → whole-ASN bans
- Now: millions of residential/mobile IPs via "proxy SDK monetization" — your TV is probably doing it. Each IP makes 4–5 requests, then vanishes.
- Anubis proof-of-work gates: difficulty 4 stopped the bots for months. They came back. Difficulty 5 stopped them. They solved it too — while legitimate users' phones burn battery and run warm.
Today git.kernel.org absorbs ~6M daily requests to view random commits — "background radiation" of system load, permanently tying up capacity serving exactly one purpose: feeding a training run.
Verdict. The Hugging Face hack was an incident; this is steady-state economics. Training-data acquisition is being externalized onto the OSS commons, and infrastructure maintainers are the frontline — armed with a stopgap the scrapers keep pricing through. Watch for PoW and credential gates becoming standard on every high-signal OSS corpus, with the collateral damage landing on legitimate users first.