Particle.news

Download on the App Store

Cloudflare Defaults to Blocking AI Crawlers on New Domains

Publishers can now charge AI firms through a Pay Per Crawl beta following support from major media and social platforms.

cloudflare logo on a phone
Matthew Prince, CEO and co-founder of Cloudflare.
Image

Overview

  • New Cloudflare domains automatically block known AI crawlers unless site owners explicitly opt in.
  • The Pay Per Crawl beta lets publishers set per-request fees for AI scrapers seeking content access.
  • Condé Nast, the Associated Press, Reddit and Pinterest have signed on to the initiative.
  • Cloudflare uses behavioral analysis, digital fingerprinting and machine learning to distinguish AI crawlers from legitimate bots.
  • OpenAI declined to participate, arguing the system adds an extra intermediary layer to content delivery.