Who has an llms.txt? We checked the top 10,000 domains

llms.txt is a Markdown file at the root of a website that tells language models what the site is and where its useful pages are. Jeremy Howard proposed it in September 2024. Two years later it has a second version of the spec, generators for every major docs framework, an audit in Chrome's Lighthouse, and a default file on every Shopify store. It also has a Google Search advocate who compared it to the keywords meta tag, and server logs that say almost nothing fetches it.

Most "llms.txt adoption" numbers come from different lists, different months and different ideas of what counts as a file, so they range from 0.015% to 28%. We wanted one number we could stand behind. On 23 October 2026 we requested /llms.txt and /llms-full.txt from each of the 10,000 most popular domains on the Tranco list, classified every response by hand-written rules, and sent a control request to every apparent adopter to catch servers that answer anything. Then we put the result next to every other study and the published evidence on who reads the file. Our numbers are our own count; everyone else's link to their source.

How many of the top 10,000 have one#

We counted a domain as having an llms.txt when https://<domain>/llms.txt ended on a 200 response, the final URL was still an llms file, and the body was Markdown-like text rather than HTML, JSON, an empty page or a copy of robots.txt. 938 domains passed. A control request for a random file that doesn't exist caught one server that returns the same Markdown for any path, and we removed it.

The denominator matters as much as the count. The Tranco list ranks registrable domains, not websites, and a fifth of its top 10,000 are CDNs, API endpoints and DNS infrastructure that never answered an HTTP request. Another 12.5% answered 401, 403, 429 or a bot challenge to our self-identified crawler; openai.com is one of them. We counted those as unknown, not as "no".

A 200 is not a file#

The easy way to measure adoption is to request the file and count the 200s. That counts 1,998 domains, more than twice the real number. Many sites answer every unknown path with their HTML app shell or a styled "page not found" page that still returns 200. Others redirect /llms.txt to the homepage or to some other page and return 200 there.

This is the main reason published numbers disagree. A tracker that counts 200s will always report more adoption than one that reads the body.

Size barely matters#

You might expect the most popular sites, with the biggest SEO teams, to adopt first. They haven't. Adoption sits between 6.7% and 11.9% in every thousand-rank band, with a mild drop below rank 7,000. Burridge found the same flatness on HTTP Archive data: 6.28% of the top 1,000 against 5.07% of the top million.

The domain's ending says more than its rank. A third of the .ai domains in the top 10,000 serve one, against one in eight .com domains and one of 112 .gov domains. SEOmator's top-100,000 scan also put .ai first and .io second, both well ahead of .com.

Where the AI labs put theirs#

Our main count only looks at the root of each domain, which undercounts the companies that care most. We checked the AI labs and big developer platforms by hand, on their main domain and their docs hosts. The pattern is consistent: the labs publish llms.txt where developers and coding agents read, not on the homepage.

The biggest llms-full.txt files are AI and developer docs too: Cloudflare's developer docs passed our 64 MB read cap, and LangChain's is 15.9 MB. Developer platforms are also the heaviest adopters among the companies we checked: Vercel, Stripe, GitHub, Supabase, Zapier, Next.js, React and Svelte all serve a valid file.

It's growing, but the studies don't add up to a line#

Every adoption study picks its own list, its own month and its own definition of a valid file. That's why they can't be joined into one growth curve. The closest thing to a like-for-like comparison is three crawls of the same population, the Tranco top 10,000, in six months:

The only true time series we found comes from Burridge, who ran the same query on HTTP Archive's crawl each month: 1.04% of a top-10,000 list in July 2025 and 5.61% in June 2026, about 5.4 times higher. His top 10,000 is ranked by Chrome traffic, counts origins rather than registrable domains, and covers the 7,504 sites HTTP Archive crawled; his validator is HTTP Archive's, not ours. So his 5.61% and our 9.4% are different measurements four months apart, not a jump from one to the other.

Every study that measured twice found growth, though. Rankability's own top-1,000 figure went from 0.3% in June 2025 to 9.3% in September 2026, across a change of method it flags itself. The spec's author put it this way in the v2 notes: "Thousands of sites now publish an llms.txt file."

Much of it is automatic#

Most sites with an llms.txt didn't write one. Docs platforms (Mintlify, GitBook), WordPress SEO plugins (Yoast, All in One SEO, Rank Math, Hostinger's) and site builders generate it. The biggest switch was Shopify's. Its developer docs say Shopify "manages an agents.md file by default for every store", and that /llms.txt and /llms-full.txt mirror it. Burridge dates the rollout to late April and early May 2026 and found a file on 78.1% of the Shopify sites in HTTP Archive's crawl.

The developer side shows up in package downloads. We summed monthly npm downloads of eight plugins that generate llms.txt for docs frameworks (Starlight, VitePress, Nuxt, Docusaurus, Rspress and VuePress).

The further down the web you go, the more the files come from defaults. In HTTP Archive's 2025 crawl of all sites, 39.6% of the files carried All in One SEO's comment. In the top 100,000, SEOmator found Shopify behind 18.2% of every llms.txt. In our top 10,000, signatures are rarer.

What's inside the files#

The spec asks for little. The # H1 with the site's name "is the only required section", it says; after it come an optional > summary, free text, and ## sections listing links. 95% of the files we found open with the H1, and 82% also have at least one ## section with a - [name](url) link list, the shape the spec's examples use.

The bigger problem is length. The spec pitches a short index an agent can read in one go. The median file is 7.7 KB, about 1,850 tokens with 37 links, which fits that. But 63 files (6.7%) are longer than a 32,000-token context window, and the largest, truecar.com's, is 4.5 MB with 13,874 links.

The links mostly work. We followed one random link from each of 200 random files: 88% answered with a 2xx, 3% were a hard 404, and the rest were blocked, rate-limited, timed out or a server error. One file pointed at a staging server.

Who reads it#

Publishing a file is one thing; having something read it is another. Ahrefs looked at its own analytics customers in May 2026: 28% of 137,210 domains publish an llms.txt, and 97% of those files received zero requests all month. The roughly 22,000 requests that did arrive came 96% from bots, and most of those bots weren't AI.

A smaller vendor test points the same way. EZY.ai put an llms.txt on 83 business websites in its panel and compared crawler fetches over 12 weeks. OpenAI's, Anthropic's and Perplexity's crawlers fetched robots.txt hundreds or thousands of times and llms.txt nine times or fewer.

The platforms say as much. Google's guide to AI features in Search says an llms.txt "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them", even as Google's own Lighthouse audits for one and its Gemini API docs publish one. None of the crawler documentation pages from OpenAI, Anthropic or Perplexity says their bots read llms.txt. And SE Ranking's study of about 300,000 domains found no effect of llms.txt on how often a domain is cited by LLMs.

What's left is a narrower job. When the SEO consultant Lily Ray pressed him, Mueller called llms.txt a "temporary crutch, perhaps to save some tokens" for AI coding tools, Ahrefs reports. The spec's own v2 pitches it at agents using a site to get a task done. That matches where it's taking hold: developer docs. Mintlify says agents made 66% of measured traffic to the docs it hosts in July 2026, though it doesn't break out llms.txt requests. If you've read our state of agent skills and state of MCP posts, this is the same pattern: the useful readers are agents doing a task, not crawlers building an index.

What this means if you build with AI#

1. Ship one if agents use your site to do work. Docs, an API, a developer product, a store an agent might buy from: an llms.txt is cheap, harmless (Google says so), and the readers that exist are coding agents. For a marketing site or a blog, don't expect it to change anything.

2. Make it valid and short. Start with an # H1, add a one-line > summary, then ## sections with - [name](url): note lists. Link the Markdown version of each page if you have one. Keep it an index of a few thousand tokens; put full text in separate files.

   TextCopy

   # Acme
   
   > Acme is a REST API for invoices. The docs, API reference and SDKs are below.
   
   ## Docs
   
   - [Quickstart](https://acme.dev/docs/quickstart.md): send a first invoice in five minutes
   - [API reference](https://acme.dev/docs/api.md): every endpoint, with examples
   
   ## SDKs
   
   - [Python](https://acme.dev/docs/sdk/python.md)

3. Check what you actually serve. Half the "200 OK" answers in our crawl weren't files. Look at the status, the content type, where you ended up and the first line:

   TerminalCopy

   curl -sL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt
   curl -sL https://example.com/llms.txt | head -3

   You want a 200, text/plain or text/markdown, the same URL, and # on the first line. If a bot challenge or firewall rule blocks unknown clients at that path, agents can't read it either.

4. Check your platform's default before writing your own. Shopify, Yoast, All in One SEO, Rank Math, Mintlify and GitBook can all generate one. Edit theirs rather than fighting it, and read it once: generated files list whatever the generator found, staging links included.

5. Don't expect traffic or citations from it. No AI search crawler documents reading it, Ahrefs found nearly all files unread, and SE Ranking found no link to citations. What those crawlers do fetch is robots.txt and your pages: allow the AI crawlers you want, and serve real HTML (or Markdown) that loads without JavaScript.

6. Measure before you invest more. Your server logs will show who requests /llms.txt. If nothing does, the file has cost you ten minutes; spend the next hour on the pages themselves.

host0's own llms.txt points agents at the skill they use to publish. If your agent made a page, docs or a small app, host0 hosts it at a link you can share, private until you open it up.

Methodology#

We took the Tranco list 56Z8N (generated 22 October 2026 from 30 days of Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Umbrella data; pay-level domains only) and requested https://<domain>/llms.txt and /llms-full.txt from the top 10,000 on 23 October 2026, between 12:03 and 12:47 UTC, from a UK connection. Our crawler identified itself (host0-research (+https://host0.ai)), kept at most 12 domains and one request per host in flight, used 10-second timeouts and followed up to five redirects. When the bare domain failed at DNS, TCP or TLS we tried www.. Domains that timed out were retried once.

• What counts: a final 200 response, a final URL that is still an llms file (redirects to a homepage or another page never count), a body that isn't HTML, JSON, empty, binary or robots.txt, and Markdown-like text. A control request for a random nonexistent .txt on each apparent adopter's host removed one catch-all server.
• Valid means the first line is an # H1, the spec's only hard rule. Full shape adds at least one ## section with a - [name](url) list.
• Denominators: "9.4%" is over all 10,000 domains. "14.2%" is over the 6,602 that gave a 2xx, 3xx or 4xx answer. 1,249 domains that blocked us and 92 that returned a server error are unknown, not "no", which makes 9.4% a floor.
• Root only: files that exist only on a docs subdomain or under a path count as absent in the main numbers. The AI-lab table is a separate check of named hosts.
• Generator signatures come from the first and last lines of each file. They're a floor.
• npm downloads are monthly totals from the npm downloads API for eight generator packages; they count installs, including CI builds.
• Other studies are each cited with their own population, date and rules, and never combined. We re-read every cited figure on its source page on 23 October 2026.

The crawler, classifier, the Tranco list, a per-domain results table and the full data pack are in our repository under docs/research/2026-10-23-llms-txt-adoption.md.

Open questions#

• Subdomains. How many top-10,000 companies have an llms.txt only on a docs. or developers. host? For the AI labs it's the norm, and a crawl of the root alone misses it.
• The blocked tenth. 12.5% of domains refused our crawler. Some of them may publish a file that bots can't reach, which defeats its purpose.
• Who reads it, outside one analytics product. Ahrefs' logs are the best evidence there is. Nobody has published how often Claude Code, Codex or Cursor request llms.txt on their own, as opposed to following a link to it.
• A clean growth rate. No two crawls of the same list used the same rules. We'll rerun this crawl with the same code to get one.