Particle.news

Download on the App Store

Publishers Launch RSL Standard to Charge AI Scrapers, With Fastly as Gatekeeper

The robots.txt-based system pursues collective licensing for AI use of web content, with enforcement and legal status still unsettled.

Overview

  • Reddit, Yahoo, Medium, Quora and People Inc. are among the first backers of Really Simple Licensing, with O’Reilly, wikiHow and Ziff Davis also joining the new effort.
  • RSL lets sites post machine-readable licensing terms in robots.txt and in media, supporting subscription fees, pay-per-crawl charges and pay-per-inference royalties.
  • The RSL Collective, led by RSS co-creator Eckart Walther and former Ask.com chief Doug Leeds, aims to negotiate on publishers’ behalf to streamline compensation.
  • The standard cannot block bots on its own, so the Collective is partnering with CDN provider Fastly to admit only AI crawlers that agree to license terms, while non-Fastly users lack a comparable technical gate for now.
  • Success depends on AI companies honoring the signals and on courts clarifying rights over scraping and training, as ongoing lawsuits and past disregard for robots.txt cloud compliance.