Overview
- Cloudflare will block all known AI web crawlers by default for any new domain joining its network.
- A private-beta Pay Per Crawl marketplace now enables publishers to set micropayment rates for individual AI crawlers.
- Major publishers and platforms including The Associated Press, The Atlantic, Fortune, Stack Overflow and Quora have joined the initiative.
- Cloudflare employs behavioral analysis, fingerprinting and machine learning to detect both recognized and shadow AI bots.
- The company is collaborating with AI firms to verify crawler identities and declared purposes, allowing site owners to grant or deny access based on use cases.