What shipped? Cloudflare's new Bot Preference Sync automatically aligns your robots.txt file with your AI bot policies for Search, Agent, and Training.
What changed? Previously, maintaining robots.txt and Cloudflare's AI bot policy meant updating two things in parallel — they'd drift. Now Bot Preference Sync treats your Cloudflare policy as the source of truth. Set AI bot rules once in the dashboard; they get written into your robots.txt automatically. The sync covers three categories: Search (crawlers), Agent (AI assistants accessing your site), and Training (model training scrapers). Rules stay in sync across both layers without maintaining static files.
Why does a builder care? AI bot traffic is exploding and the robots.txt protocol was never designed for the nuance of agent vs. crawler vs. training access. This eliminates a manual sync failure point that every site with AI bot policies has — and makes it trivial to revoke access from specific scraper categories in one place instead of editing a text file and hoping it works.