Particle.news
Download on the App Store
19 ARTICLES
·
3w ago

Cloudflare Blocks AI Crawlers by Default, Debuts Pay Per Crawl Beta

The change establishes a permission-based model that safeguards content through opt-in access

cloudflare logo on a phone
Matthew Prince, CEO and co-founder of Cloudflare.
A logo of CLOUDFLARE sits outside the company's house on the opening day of the 55th annual meeting of the World Economic Forum (WEF) in Davos, Switzerland, January 20, 2025. REUTERS/Yves Herman/File Photo

Overview

  • Cloudflare will block all known AI web crawlers by default for any new domain joining its network.
  • A private-beta Pay Per Crawl marketplace now enables publishers to set micropayment rates for individual AI crawlers.
  • Major publishers and platforms including The Associated Press, The Atlantic, Fortune, Stack Overflow and Quora have joined the initiative.
  • Cloudflare employs behavioral analysis, fingerprinting and machine learning to detect both recognized and shadow AI bots.
  • The company is collaborating with AI firms to verify crawler identities and declared purposes, allowing site owners to grant or deny access based on use cases.