Overview
- Cloudflare will set new default site settings that allow traditional search crawling but block AI training and agent use on ad-supported pages, with the defaults applying to new sites and existing free customers and an option to change settings in dashboards.
- The company is expanding its Pay Per Crawl experiment into a Pay Per Use program and named Ceramic.ai and You.com as early partners for arrangements that could pay publishers when content appears in AI answers or when an agent buys premium information.
- Cloudflare plans an Attribution Business Insights dashboard to show which AI bots access content, where content is cited, and how much human traffic different AI platforms drive so publishers can measure and optimize for AI-driven answers.
- The push responds to industry data showing agentic AIs fetch far more pages than the referral clicks they return, which has sharply reduced publisher traffic and revenue and has driven multiple firms to pursue licensing or standards for AI access.
- Observers warn the changes could centralize agent identification, measurement, and payments at the infrastructure layer, which may help smaller publishers get paid but also risk increasing dependence on a single intermediary and advantaging well‑funded platforms.