Busiest .uk websites block AI crawlers twice as often as smaller sites
Data study
Among the most-visited .uk websites, about one in five ask at least one AI crawler to stay out of the whole site — double the share among smaller .uk sites. Bots that collect material to train AI models are turned away more often than the bots that fetch pages for AI search answers.
Among the most-visited .uk websites, about one in five ask at least one AI crawler to stay out of the whole site. That is double the share among smaller .uk sites. Bots that collect material to train AI models are turned away more often than the bots that fetch pages for AI search answers.
Type /robots.txt after a website's address and you may find a short, plain text file that reads like a door policy. Search engines welcome. Checkout pages private. And on some sites there is a line naming an AI company's crawler, followed by a single forward slash. That slash asks the bot to leave the whole site alone.
The story often told in marketing and media circles is that the web is pulling up the drawbridge on AI, and that the big names with the most to protect are pulling hardest. To check it, we read the robots.txt files of every .uk website in the Chrome UX Report's global top 500,000. In all, 4,377 sites answered.
Half of the story is right. The most-visited of those sites block AI crawlers at twice the rate of the rest. Blocking is still the exception across the group as a whole, though, and the bots refused most often are the ones that train AI models.
Most popular .uk sites still let every AI bot in
Start with the idea that the web is shutting AI out. Across all the sites checked, 11.7% block at least one of the 11 AI crawlers from the entire site. That is fewer than one in eight. The large majority set no whole-site block on any of them.
Three limits shape that figure.
- Requests, not locks. A robots.txt file asks crawlers to stay away, and well-behaved crawlers are expected to honour it. It cannot stop a bot that ignores it.
- Whole-site blocks only. A site that fences off a few folders from AI bots is counted here as open.
- Responding sites only. Sites that did not answer when the files were fetched are left out of every share.
The files were read in October 2026, so this is a single snapshot of where things stood, not a measure of change.
The most-visited sites block twice as often
On size, the received wisdom holds. Among .uk sites in the global top 100,000, 20.4% block at least one AI crawler. For .uk sites ranked from 100,001 to 500,000, the share is 10.2%.
| Group of .uk sites (Chrome UX Report rank) | Sites that answered | Block at least one AI crawler (%) |
|---|---|---|
| Global top 100,000 | 648 | 20.4 |
| Ranked 100,001 to 500,000 | 3,729 | 10.2 |
| All sites checked | 4,377 | 11.7 |
The obvious objection is size. The top group is 648 sites, set against 3,729 below it, and a smaller group can produce a dramatic percentage by accident. That is the reason this piece does not lead on "one in five" alone. Two things count against a fluke. The top group is a complete count of every .uk site in the global top 100,000 that answered, so it is not a sample. The gap between the two groups is also far wider than chance would plausibly produce at these group sizes.
The files record what sites chose. They do not record why, and this data cannot say.
Training bots are refused more than search bots
AI crawlers do different jobs. Some gather pages to train AI models. Others index pages so a site can turn up in AI search results in ChatGPT, Claude or Perplexity. One, ChatGPT-User, fetches a page live when a ChatGPT user asks about it.
UK sites refuse the training bots most. The share of sites blocking at least one training crawler is the same as the share blocking any AI crawler at all. The shares blocking an AI search crawler, or ChatGPT's live fetcher, are lower.
| Type of AI crawler | Sites blocking at least one (%) |
|---|---|
| Training crawlers (7 checked) | that figure |
| AI search crawlers (3 checked) | 7.0 |
| ChatGPT user-triggered fetcher | 5.4 |
Of the 11 crawlers checked, OpenAI's GPTBot is blocked most: 9.1% of sites shut it out, with Anthropic's ClaudeBot close behind at 8.6%. Both are training crawlers. The full list of 11 is in the table at the foot of this piece.
The distinction matters for anyone who wants customers to find them. Blocking a training crawler keeps your pages out of the material a model learns from. Blocking a search crawler such as OAI-SearchBot or PerplexityBot asks that your pages not be indexed for those tools' search results, which can make it harder for your site to be found or cited when someone asks an AI assistant a question. Locking the stockroom is one decision. Taking your shop off the map is another.
One name causes confusion. Google-Extended controls whether Google can use a site's pages for Gemini training. Blocking it does not remove a site from Google Search or from AI Overviews.
Blocking is a minority habit, heaviest at the top
On this evidence, the web as a whole has not shut AI out. Blocking is something a minority of popular .uk sites do. It is roughly twice as common among the most-visited, and it is aimed more at the bots that train models than at the bots that send readers back.
What this means for UK businesses
- Look at your own file. Add /robots.txt to your domain and read it. Some sites block crawlers through a catch-all rule or a template setting that nobody chose on purpose.
- Make two decisions. Whether AI firms may train on your content and whether AI search tools may index you are separate questions, and robots.txt lets you answer each one separately.
- Check before blocking Google-Extended. Blocking it affects Gemini training only. Your Google Search listing is unaffected.
- Treat robots.txt as a request. It signals your wishes to crawlers that follow the rules. It is not a security measure.
- Use folder rules where they fit. You can keep AI bots away from some sections and leave the rest open.
Most of the busiest .uk sites still leave that forward slash off. Whether yours should carry one is a business decision, and it starts with knowing what the file already says.
How we built it
This analysis fetched robots.txt and llms.txt directly from every .uk website in the Chrome UX Report's top 500,000, a ranking of sites by real Chrome traffic. The site list came from the HTTP Archive crawl. The checks ran in October 2026. The 4,377 sites that returned a response form the base for every percentage. Sites that did not respond are excluded rather than counted as open or closed.
A site is counted as blocking a crawler only when that crawler is disallowed from the whole site, either by a rule naming it or by the catch-all rule for all bots. Partial blocks of individual folders are not counted. robots.txt is a voluntary standard, so a block records a site's stated wish. It does not show whether any crawler complied.
The seven training crawlers are GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent and Bytespider. The three AI search crawlers are OAI-SearchBot, Claude-SearchBot and PerplexityBot. ChatGPT-User is counted separately as a user-triggered fetcher.
The difference between the top 100,000 group and the 100,001 to 500,000 group is statistically significant: the probability of a gap this size arising by chance at these group sizes is effectively zero. Figures are a point-in-time snapshot. llms.txt files are counted only when they return a text file opening with a Markdown "#" title, as the llms.txt proposal specifies.
Platform figures in the last table come from HTTP Archive's September 2026 crawl of .uk home pages. They cover .uk domains only, so UK businesses on .com and other domains are not included. Technologies are detected by fingerprint on the home page and can be missed if they are used deeper in a site.
AI crawlers blocked from the whole site, .uk sites that answered (October 2026, base 4,377)
| Crawler (operator, purpose) | Sites blocking (%) |
|---|---|
| GPTBot (OpenAI, training): most-blocked of the 11 checked | 9.1 |
| ClaudeBot (Anthropic, training) | 8.6 |
| Claude-SearchBot (Anthropic, AI search): least-blocked of the 11 checked | 4.8 |
llms.txt on .uk sites that answered (October 2026, base 4,377)
| Measure | Value |
|---|---|
| Sites publishing an llms.txt | 17.7% (775) |
| Of those, files using the same generated "Agent Instructions" template | 58.2% (451) |
Platforms detected on .uk home pages (HTTP Archive, September 2026)
| Platform | Share of .uk home pages (%) | Rank |
|---|---|---|
| WordPress | 38.44 | First of the 8 website platforms (CMS) measured |
| WooCommerce | 9.31 | First of the 8 e-commerce platforms measured |
| Shopify | 8.13 | Second of the 8 website platforms (CMS) measured; second of the 8 e-commerce platforms measured |
