Who blocks AI crawlers? robots.txt across the top 10,000 sites

robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch. It dates from 1994, became an internet standard (RFC 9309) in 2022, and has never been enforceable: a crawler reads it and decides whether to comply. Since OpenAI documented GPTBot in 2023, it has also become the place where websites say what they think of AI companies, one user agent at a time.

Most "who blocks AI" numbers mix different lists, different months and different ideas of what a block is. So we measured it ourselves. On 23 October 2026 we fetched /robots.txt from the 10,000 most popular domains on the Tranco list, the same list we used for our llms.txt crawl, and parsed every file the way Google's open-source parser does, for 23 AI user agents plus Googlebot, Bingbot and Applebot as controls. Then we put the result next to Cloudflare's June 2025 baseline, the news-publisher trackers, and the evidence on whether AI crawlers obey any of it. Our numbers are our own count; everyone else's link to their source.

What the top 10,000 serve at /robots.txt#

A robots.txt is made of groups. Each group starts with one or more User-agent lines and lists Allow and Disallow paths. A crawler follows the group that names it, and falls back to the User-agent: * group if none does. So "blocking GPTBot" can mean a group that names it and disallows /, or a * group that disallows everything for every bot that isn't named.

We counted an agent as fully blocked when the group it follows disallows the whole site, with or without a few allowed exceptions. A missing file, a 404 or an HTML error page counts as "allow everything", which is what the standard says. A domain that gave no answer, a server error or a 403 to our self-identified crawler counts as unknown. Throughout this post, "blocks" means the robots.txt asks that bot to stay out. Whether the bot listens is a separate question, covered below.

The median robots.txt is 637 bytes and names one user agent. 287 files name 50 or more. The most-named agent in the top 10,000 isn't Googlebot: GPTBot appears in 1,110 files, against 563 for Googlebot and 436 for Bingbot.

Which AI crawlers get blocked#

The training crawlers lead: Common Crawl's CCBot, ByteDance's Bytespider, OpenAI's GPTBot and Anthropic's ClaudeBot each sit between 13% and 15%. The search and user-triggered bots sit between 7% and 10%. The controls show what a normal block rate looks like: 2.5% of sites block Googlebot, and most of those are sites closed to every crawler.

Naming a bot isn't the same as blocking it. Of the 1,110 files that mention GPTBot, 154 explicitly allow it, including cloudflare.com, wordpress.org, intel.com, theverge.com and theatlantic.com, and 192 block only some paths.

Files also carry a lot of dead weight. anthropic-ai appears in 644 files and claude-web in 515, more than Anthropic's current Claude-SearchBot (387) or Claude-User (389), and Anthropic's crawler page names neither old token. Another 99 files name agents that match no documented crawler, like ChatGPT (65 files) or OpenAI (43). Under the standard's matching rules, a ChatGPT group doesn't apply to ChatGPT-User.

Training out, search in#

Most AI companies now run separate crawlers for separate jobs, and say so. OpenAI's documentation spells out the split: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot". Anthropic splits ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches a user asked for). Google and Apple offer opt-out tokens, Google-Extended and Applebot-Extended, that sit on top of their search crawlers.

Sites use the split. For every vendor in the chart below, the training token is blocked about twice as often as the search crawler, and far more often for Google and Apple, where blocking the search crawler would mean leaving their search results.

Among the sites that block at least one purpose-labelled AI bot and still allow Googlebot, two policies account for 86%: "no training, but cite me", and "no AI at all". Blocking AI search while allowing training is almost unheard of.

How many agents a site blocks is bimodal. 171 sites block exactly one AI agent, most often GPTBot (42 of them). At the other end, 90 block all 23 while still allowing Googlebot. Two spikes in between come from templates: 51 of the 80 sites that block exactly 13 use Cloudflare's managed robots.txt, which blocks 13 training, data and legacy Anthropic agents and none of the search or user-triggered ones, and 30 block every AI agent except OpenAI's three. 29 of those 30 belong to News Corp or Dotdash Meredith, which brings us to money.

Bigger sites and publishers block more#

The more popular the site, the more likely it blocks. In the top 100, the blockers include facebook.com, instagram.com, amazon.com, tiktok.com, yahoo.com and msn.com, and allowlist-style files at x.com, pinterest.com and baidu.com that name a few search engines and ask everyone else to stay out.

News publishers are the heaviest blockers. We flagged sites whose robots.txt declares a Google News sitemap, a rough proxy that catches most publishers and a few corporate newsrooms. Among those 562 sites, 52.7% block at least one AI agent while allowing Googlebot, against 13.6% of the other 6,365. Thunderbit's May 2026 crawl of the same population, with its own news category, put news at 47%.

Licensing deals reshape who gets blocked#

Since 2023, OpenAI has announced content deals with news groups including the Associated Press, Axel Springer, Le Monde, Prisa, the Financial Times, Dotdash Meredith, News Corp, Vox Media, The Atlantic, TIME, Condé Nast, Hearst, Schibsted, the Guardian and the Washington Post. Most cover showing their journalism in ChatGPT with attribution and links; the AP's 2023 deal licensed part of its archive. Cloudflare counted more than 50 publisher-AI agreements since 2023.

We mapped 84 top-10k domains to the publishers in those announcements, by who owned them at signing (Vox Media sold Polygon in May 2025, so we left it out). Then we compared them with the other 491 news-sitemap sites.

The pattern is sharp in both directions. 100 sites in the top 10,000 block ClaudeBot but not GPTBot, and 66 of them are partner domains. The 168 sites that block GPTBot but not ClaudeBot include none. The Wall Street Journal's file shows the mechanics: GPTBot, OAI-SearchBot and ChatGPT-User sit in an allowlist group with link-preview bots, and everything else falls to a * group that disallows the whole site. Not every partner bothered: Hearst's magazines (elle.com, esquire.com, cosmopolitan.com) block neither crawler.

The news trackers show the same shift over time. Ben Welsh's homepages.news tracker follows the robots.txt of about 1,156 news sites. We read its archived snapshots month by month: the share blocking OpenAI rose fast in 2023, peaked at 53.5% on 1 May 2024, days to weeks before News Corp, Dotdash Meredith, Vox Media and The Atlantic announced their OpenAI deals, and has since slipped to 46.9%.

AI companies take sides in their own files too.

Has blocking grown?#

The obvious comparison is Cloudflare's. On 6 June 2025, Cloudflare Radar found a robots.txt on 3,816 of the top 10,000 domains, and GPTBot "disallowed by 312 domains (250 fully, 62 partially)". That's 8.2% of the files, and 3.1% of all 10,000 domains.

That number can't be set against our 14.2%. Cloudflare counted directives that name GPTBot, over files found; our 14.2% counts any full block, including through *, over domains with a known policy. Counted Cloudflare's way, rules naming GPTBot in a disallow appear in 956 of our 5,778 files.

Two independent 2026 crawls of the Tranco top 10,000 land close to ours. Thunderbit counted 20.3% blocking at least one AI crawler in May, on the 7,248 sites it could analyze (a robots.txt or a clear "none"); we count 19.2%. StackFox's February re-run of 7,636 Tranco top sites found GPTBot blocked in 17% of the 1,912 robots.txt files it found, without saying whether that means a full or partial block; 17.0% of our files fully block it. Across the whole web the numbers are much smaller: the HTTP Archive's 2025 Web Almanac found gptbot named, allowed or blocked, in 4.5% of desktop sites' robots.txt, up from 2.9% in 2024.

robots.txt is a request, not a wall#

Nothing makes a crawler obey robots.txt, and some AI companies say up front that their user-triggered bots won't always follow it. OpenAI's page for ChatGPT-User, the agent that fetches a page when a user asks, now says: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity's documentation says Perplexity-User "generally ignores robots.txt rules", and Meta says its fetcher "may bypass" them. Anthropic's page goes the other way: its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt", a statement made for all three, the user-triggered one included.

So the 8.6% of sites that disallow ChatGPT-User may not be keeping it out. TollBit, which sells publishers a paywall for bots, measures the gap on its own network.

That's why the fight has moved to the network. Cloudflare lets every customer choose which declared AI crawlers to block, says more than 2.5 million websites disallowed AI training through its tools by August 2025, and says its customers send over a billion HTTP 402 "payment required" responses a day. In July 2026 it split its controls into search, agent and training, and in September added a "Disallow AI Training" setting, saying fewer than 1% of its sites block search bots while 17% block training. The economics behind that: in July 2025 Anthropic's crawlers fetched about 38,000 pages for every visitor they referred, OpenAI's about 1,100, by Cloudflare's count of the referrals it can see.

Cloudflare's other bet is putting the preference into robots.txt itself. A Content-Signal line says whether content may be used for search (search), as input to an AI answer (ai-input) or for training (ai-train). In the top 10,000 it's still rare.

The standards process is slower. The IETF's AI Preferences working group has drafts for a vocabulary and for attaching it to robots.txt and HTTP headers, but it has missed its own submission deadlines of August 2025 and August 2026, and enforcement is explicitly outside its charter.

What robots.txt has to do with llms.txt#

The two files point in opposite directions: robots.txt keeps crawlers out, llms.txt invites language models in. Joining our two crawls, sites that publish an llms.txt are about half as likely to block any AI crawler (11.4% against 20.4%), and far less likely to block AI search: 9 of 931 block OAI-SearchBot. 46 sites publish an llms.txt and block GPTBot in the same root, gravatar.com, unesco.org and oreilly.com among them. That's consistent with "use us for answers, not for training". Only 15 top-10k domains serve an /ai.txt, Spawning's alternative opt-out file.

What this means if you build with AI#

1. Decide per purpose, not per company. Disallowing GPTBot tells OpenAI not to use your pages for training; it doesn't keep you out of ChatGPT search, which uses OAI-SearchBot. If you want to be cited in AI answers but not trained on, block the training tokens and leave the search bots alone. That's the most common policy among sites that block AI (43%). One catch: Google says Google-Extended also covers grounding in Gemini apps, not just training.

   TextCopy

   # Training crawlers and opt-out tokens
   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   User-agent: CCBot
   User-agent: Meta-ExternalAgent
   User-agent: Bytespider
   Disallow: /
   
   # Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
   # aren't named, so they follow the * group
   User-agent: *
   Disallow: /admin/

2. Use the names vendors publish today. anthropic-ai and claude-web don't appear on Anthropic's current crawler page; ClaudeBot, Claude-SearchBot and Claude-User do. ChatGPT matches nothing; ChatGPT-User is the agent. Copy names from the vendor pages, not from a 2023 blog post, and check for curly quotes and odd hyphens if you paste.

3. Know that consecutive User-agent lines share one group. Put one blank line between groups, and remember that a bot named in any group ignores the * group completely. A named bot with only Allow: /article/ can reach everything else you'd assumed * was protecting.

4. Don't count on robots.txt for anything sensitive. User-triggered fetchers from OpenAI, Perplexity and Meta may not follow it, and TollBit finds a share of training and search bots ignoring it too. If a page must stay private, put it behind sign-in; if you need bots kept out, use your CDN's bot rules, which can check a crawler's published IP ranges instead of trusting its name.

5. Test what you actually serve. 962 top-10k domains refused our self-identified request for robots.txt itself, so their rules may never reach a polite crawler that isn't on their allowlist. Fetch your own file with a non-browser client and check that it returns 200 and text/plain.

   TerminalCopy

   curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -A 'robots-check' https://example.com/robots.txt

6. For an agent-built site, add rules on purpose. Generators and hosts often ship a robots.txt you never read. If you want an AI tool to find a docs page or a demo, make sure its search and user-triggered agents aren't caught by a blanket block.

host0 serves each app's /robots.txt itself: apps are never indexed unless the owner makes one public on the web, and private apps sit behind sign-in, so crawlers never see them. If your agent made a page or a small app, host0 hosts it at a link you can share.

Methodology#

We took the Tranco list 56Z8N (generated 22 October 2026 from 30 days of Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Umbrella data; pay-level domains), the same list as our llms.txt crawl, and requested https://<domain>/robots.txt from the top 10,000 on 23 October 2026, between 13:18 and 14:04 UTC, from a UK connection. Our crawler identified itself (host0-research (+https://host0.ai)), kept at most 8 domains and one request per host in flight, used 10-second timeouts and followed up to five redirects, including to other hosts. When the bare domain failed at DNS, TCP or TLS we tried www.; timeouts were retried once.

• Known policy (6,927): a robots.txt (5,778), or a clear absence (1,149: a 4xx, an HTML page or an empty body), which the standard treats as "allow everything". 962 domains that returned 401, 403, 429 or a challenge page, 81 server errors and other statuses, and 2,030 with no HTTP answer are unknown, and stay out of the denominator.
• Parser: our own, following RFC 9309 and Google's open-source parser: the agent name is matched on its leading letters, hyphens and underscores, case-insensitively; groups naming the same agent are merged; an unnamed agent follows *; the longest matching path wins, Allow wins ties. It agrees with Scrapy's Protego parser on all but at most 12 sites per agent, except Applebot-Extended and omgilibot, where Protego applies an Applebot or omgili group too. Per-agent numbers can move a point or two with the matching rule.
• Fully blocked means the group an agent follows disallows /, with or without a few Allow exceptions that leave an ordinary article path blocked. AI-specific means fully blocked while Googlebot isn't.
• Purposes come from each vendor's crawler documentation, read on 23 October 2026. We found no current vendor page for Bytespider, cohere-ai or Diffbot, and no official xAI crawler page.
• Publishers are sites whose robots.txt declares a Google News sitemap. OpenAI partners are domains we mapped by hand to the publishers in OpenAI's announced content deals (July 2023 to April 2025), by ownership at signing.
• Other studies are cited with their own population, date and definition. The Cloudflare comparison recounts our data by Cloudflare's definition (rules that name GPTBot in a full or partial disallow, over files found).

The crawler, parser, per-domain results table and the full data pack are in our repository under docs/research/2026-10-23-ai-crawlers-robots-txt.md.

Open questions#

• The blocked tenth. 962 domains wouldn't serve robots.txt to a self-identified crawler. How many of them publish AI rules that polite bots can't even read?
• Who actually complies. TollBit's network is the best evidence there is on bypass rates. Nobody outside it publishes per-vendor compliance for the user-triggered fetchers.
• What the deals say. The partner pattern is plain in the files, but no public deal text mentions crawler access.
• A clean trend. Our comparison with Cloudflare crosses two lists and two crawlers. We'll rerun this crawl with the same code to get a real growth rate.