llms.txt in 2026: adoption is up, readership still isn't

What the file promises

llms.txt is a plain-Markdown file you drop at a site's root — /llms.txt — listing the pages an AI system should treat as authoritative, stripped of navigation chrome, ads, and JavaScript. Answer.AI's original proposal framed it as a curated map: instead of an AI system scraping and guessing at a rendered page, it reads a short, hand-picked index straight to the content that matters. Two years after that pitch, the file is common. Whether anything reads it is a separate question, and 2026's data answers it more precisely than last year's did.

Adoption is real and still climbing

SE Ranking's May 2026 study of roughly 300,000 domains found llms.txt adoption at 10.13%, up from about 0.4% in April 2025 — something like 25x growth in thirteen months, according to Search Engine Journal's writeup of the study. That's a genuinely fast curve for a convention with no spec body, no browser vendor mandate, and no ranking incentive behind it. It spread the way robots.txt did: word of mouth among people who maintain websites, not enforcement from anywhere.

The same study is the more interesting finding, though: after controlling for site authority, schema density, and content recency, the researchers found no measurable lift in how often a domain gets cited by AI systems, whether or not it published an llms.txt file. They ran the comparison through an XGBoost model and reported that dropping llms.txt as a variable improved the model's accuracy — the file carries no independent signal once you account for the things that already predict citation.

Almost nobody is fetching it

Adoption measures publishing, not reading. For that, the sharper instrument is server logs, and Ahrefs' log-study covering 137,000 domains — cited widely since its release — found that 97% of published llms.txt files received zero requests in May 2026. Of the requests that did land, only about 1.1% came from AI retrieval bots; most of the rest were SEO audit tools checking whether the file existed at all, not systems consuming it. GPTBot shows up occasionally. Google's and Anthropic's crawlers barely register. None of them probe for the file on sites that don't have one — there's no curiosity, just indifference.

Google's own public position matches the data. In a forum exchange covered by Search Engine Land, Search Advocate John Mueller compared llms.txt to the old keywords meta tag — a file websites fill in on the assumption someone's reading it, with no confirmation anyone is. Google's official generative-AI optimization guide, published May 15, 2026 and updated a month later with a section titled "Clarifying guidance on llms.txt files," is blunter still: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and maintaining one "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them."

Where it does get read

The exception is narrower and more useful than blanket AI-search citation: coding tools that a developer points at a doc set on purpose. Cursor ships its own llms.txt at docs.cursor.com and lets you add a project's llms-full.txt as documentation context through its @Docs feature — a deliberate, opt-in fetch, not a crawler decision. That's the pattern across IDE agents and MCP-connected tools generally: llms.txt works when a human or an agent's tool-use loop chooses to fetch a specific URL for a specific task, not when it's left for an autonomous system to discover on its own initiative. Publishing one still saves that agent from parsing your marketing nav to find the API reference.

Google's own tools disagree with Google Search

The clearest sign that this is unsettled, not resolved, is a contradiction inside Google itself. While Search leadership tells site owners not to bother, Chrome's Lighthouse auditing tool shipped an "Agentic Browsing" category that checks for exactly this file. The category was created in Lighthouse v13.2.0 (April 30, 2026), promoted into the default config in v13.3.0 a week later, and its llms.txt validation logic was refined through v13.4.0 (June 9) and v13.4.1 (July 20), which added detailed error messages when a fetch fails. The audit checks four things: an llms.txt file, support for the emerging WebMCP protocol (a Chrome- and Microsoft-backed way for a page to expose callable tools to an agent instead of being scraped), the accessibility tree, and cumulative layout shift.

That's a different bet than search citation: it's about an agent navigating your live page in a browser, not an AI system summarizing your site in an answer. Lighthouse's documentation frames a missing llms.txt as a tax on agent time — "without this file, agents may spend more time crawling the site to understand its high-level structure" — which is a real, if modest, efficiency claim rather than a discoverability one.

What to do with this

The honest read: llms.txt didn't become the citation lever some 2024 pitches implied, and the newest, largest study says so with more precision than last year's. But it didn't die either — it settled into a smaller, real job as a convenience file for tools that fetch documentation on request, with Chrome now treating it as one input to agent-navigation readiness rather than search visibility. It costs about thirty minutes to write. Ship one for the agents that will actually ask for it, and don't expect it to move an AI search result.

The protocol tracker keeps a running status on this and the rest of the agent-interface stack.