Overview
- Cloudflare’s Content Signals Policy extends robots.txt with three explicit directives—search, ai-input and ai-train—to tell crawlers which uses are allowed.
- The company says more than 3.8 million domains under its managed robots.txt will adopt the policy automatically, covering roughly 20% of the web it helps run.
- By default, sites in Cloudflare’s managed program will continue to permit traditional search indexing while blocking AI model training.
- CEO Matthew Prince says the signals may carry legal weight and presents Google with a choice to separate crawlers for search and AI answers or risk losing access.
- Cloudflare targets Google’s AI Overviews and contrasts Google’s single-crawler approach with OpenAI’s separate bots, while Google maintains its AI features still send traffic to sites.