Overview
- Developer logs from an eight‑day test this week showed per‑page Markdown ‘twins’ were fetched about thirty times by agents claiming to be AI crawlers while the site’s /llms.txt was fetched only four times and received no clear hits from genuine AI bots.
- Most of the Markdown fetches came from Meta’s crawler and a few from Amazon, Anthropic and OpenAI agents, which shows current crawler activity favors fetching page copies over consulting a single root index.
- Major AI platforms have not broadly committed to parsing llms.txt and Google’s public guidance says Search does not use the file for its AI features, reducing the expectation that llms.txt alone will drive citations.
- Practical advice from the reporting is to treat llms.txt as an easy, low‑risk hedge you can add if convenient but to prioritize server‑rendered, crawlable HTML, clear semantic structure, and fast pages plus per‑page Markdown copies for immediate gain.
- The llms.txt idea dates to late 2024 as a robots.txt‑style index proposed by Jeremy Howard and has some platform backing, yet real‑world uptake is unproven and site owners should measure fetch logs and avoid relying on the index as their primary AI visibility strategy.