Overview

• New Cloudflare domains automatically block known AI crawlers unless site owners explicitly opt in.
• The Pay Per Crawl beta lets publishers set per-request fees for AI scrapers seeking content access.
• Condé Nast, the Associated Press, Reddit and Pinterest have signed on to the initiative.
• Cloudflare uses behavioral analysis, digital fingerprinting and machine learning to distinguish AI crawlers from legitimate bots.
• OpenAI declined to participate, arguing the system adds an extra intermediary layer to content delivery.