Overview
- New Cloudflare domains automatically block known AI crawlers unless site owners explicitly opt in.
- The Pay Per Crawl beta lets publishers set per-request fees for AI scrapers seeking content access.
- Condé Nast, the Associated Press, Reddit and Pinterest have signed on to the initiative.
- Cloudflare uses behavioral analysis, digital fingerprinting and machine learning to distinguish AI crawlers from legitimate bots.
- OpenAI declined to participate, arguing the system adds an extra intermediary layer to content delivery.