AI Firms Bypass Web Protocols to Scrape Publisher Content

Multiple AI companies, including OpenAI and Anthropic, accused of ignoring robots.txt standards to access copyrighted material.

TollBit reports widespread non-compliance with robots.txt among AI companies.
Forbes accuses Perplexity AI of plagiarizing its investigative stories.
Publishers argue that ignoring robots.txt undermines their ability to monetize content.
Some publishers are pursuing legal action, while others seek licensing deals.
The debate raises ethical and legal questions about AI's use of web content.