Overview

• Starting July 1, Cloudflare switched new domains to automatically block known AI crawlers unless site owners opt in for access.
• Participation in the private beta requires AI companies and publishers to register Cloudflare accounts and negotiate per-crawl fees.
• Cloudflare employs behavioral analysis, fingerprinting, and machine learning to identify and block both publicized and shadow AI scrapers.
• A coalition of major publishers, including Condé Nast, Time, the Associated Press, the Atlantic, Adweek, and Fortune, has backed the permission-based model.
• Cloudflare data shows AI bots scrape content with thousands of requests per referral, underlining the economic imbalance between crawlers and sites.