Data study
A direct check of robots.txt files on more than four thousand popular .uk websites finds that blocking AI crawlers is a minority habit — but one that is about twice as common among the very biggest sites.
Every website has a small plain-text file at its root called robots.txt. Visitors never see it. It works like a sign taped inside a shop door, telling automated visitors which aisles they may walk down. Since AI companies began sending their own crawlers across the web, that sign has become one of the few places where a business can say "not you".
The usual story is that site owners are slamming the door. You hear it in marketing meetings and read it in trade columns: publishers and brands are locking out the AI bots in droves.
The .uk web tells a quieter story. This analysis fetched robots.txt directly from the most popular .uk websites, more than four thousand of them, drawn from the top half-million sites in Chrome's own usage data. Roughly one in nine blocks even a single AI crawler from the whole site. The habit is concentrated at the top, though. The biggest sites are about twice as likely to have that sign on the door.
Most popular .uk sites leave the door open
Start with the broad claim that everyone is blocking AI. Across the sites that answered, the large majority set no whole-site block against any of the 11 AI crawlers checked. Only about one in nine refused at least one.
Three limits shape that figure. A site counts only if it shuts a crawler out of every page, either by naming it or through a catch-all rule. Sites that fence off a few folders are not counted, so partial blocking goes unrecorded. Sites that did not respond are left out entirely. Most importantly, robots.txt is a request. Well-behaved crawlers are expected to honour it, but nothing in the file can force them to.
Even read generously, this is far from a stampede. The doors are mostly open.
OpenAI's training bot is turned away most often
When sites do block, they reach first for the best-known name. GPTBot, which OpenAI uses to gather material for training its models, is the most-blocked of the 11 AI crawlers checked, shut out by 9.1% of sites. Anthropic's ClaudeBot and ByteDance's Bytespider are joint second, at 8.6%.
share of checked .uk sites blocking each AI crawler (October 2026)
| Rank of 11 | Crawler | Operator | Purpose | Sites blocking | Share (%) |
|---|---|---|---|---|---|
| 1 | GPTBot | OpenAI | Training | 400 | 9.1 |
| =2 | ClaudeBot | Anthropic | Training | 378 | 8.6 |
| =2 | Bytespider | ByteDance | Training | 375 | 8.6 |
| 4 | CCBot | Common Crawl | Open web archive used for AI training | 350 | 8.0 |
| =5 | meta-externalagent | Meta | Training | 329 | 7.5 |
| =5 | Applebot-Extended | Apple | Apple Intelligence training | 328 | 7.5 |
| 7 | Google-Extended | Gemini training and grounding | 314 | 7.2 | |
| 8 | PerplexityBot | Perplexity | AI search | 292 | 6.7 |
| 9 | OAI-SearchBot | OpenAI | AI search (ChatGPT search) | 280 | 6.4 |
| 10 | ChatGPT-User | OpenAI | User-triggered live fetch | 237 | 5.4 |
| 11 | Claude-SearchBot | Anthropic | AI search (Claude) | 210 | 4.8 |
The spread is narrow. All seven training crawlers are blocked by between 7.2% and 9.1% of sites, with no single crawler far ahead of the rest.
Training bots are refused far more often than AI search bots
The 11 crawlers do different jobs, and site owners treat them differently. Training crawlers collect pages to build AI models. AI search crawlers index pages so that a chatbot's search feature can cite them. A user-triggered fetcher visits a page live when someone asks ChatGPT about it.
Training crawlers draw the most refusals. AI search crawlers are turned away by a noticeably smaller share, about one site in fourteen, and ChatGPT's live fetcher by fewer still.
| Type of AI crawler | Share of sites blocking at least one (%) |
|---|---|
| Training crawlers (7 checked) | 11.7 |
| AI search crawlers (3 checked) | 7.0 |
| User-triggered fetcher (ChatGPT-User) | 5.4 |
One crawler needs explaining because it is so often misunderstood. Google-Extended controls whether Google can use a site's pages to train and ground Gemini. Blocking it does not remove a site from Google Search or from AI Overviews.
The pattern suggests that site owners who block are making a distinction. Many are content to be found and cited by AI tools but would rather their pages were not used as training material. Whether that pattern reflects careful strategy or simply the defaults of popular blocklists cannot be told from robots.txt alone.
The biggest sites block about twice as often
At this point the received wisdom has looked weak at every turn. Then the size split brings it partly back.
Among the .uk sites ranked in the global top 100,000, about one in five blocks at least one AI crawler. Among those ranked from 100,001 to 500,000, the share is about one in ten.
| Summary measure | Sites | Share (%) |
|---|---|---|
| Block at least one AI crawler (all 4,377) | 514 | 11.7 |
| Block at least one training crawler | 514 | 11.7 |
| Block at least one AI search crawler | 306 | 7.0 |
| Block the user-triggered fetcher | 237 | 5.4 |
| Top 100,000: block at least one (of 648) | 132 | 20.4 |
| Ranked 100,001 to 500,000: block at least one (of 3,729) | 382 | 10.2 |
| Popularity band (global rank) | Sites checked | Share blocking at least one AI crawler (%) |
|---|---|---|
| Top 100,000 | 648 | 20.4 |
| 100,001 to 500,000 | 3,729 | 10.2 |
| All checked sites | 4,377 | 11.7 |
The obvious objection is that the big-site group is too small to mean much. It is not a handful. It holds several hundred sites, and a doubling across groups of this size is far too wide to put down to chance.
The data does not say why big sites block more. Larger organisations are more likely to have legal teams, content they license, or staff whose job includes watching which bots arrive. These explanations are plausible, but they remain guesses.
A minority habit, concentrated at the top
The idea that the web is barring its doors to AI is partly true and mostly overstated. The overall picture shows a small minority of popular .uk sites blocking AI. Among those that do, training crawlers are the main target, while AI search tools are less often refused. The one part of the myth with real support is at the top of the rankings, where blocking runs at about double the rate seen elsewhere.
What this means for UK businesses
If you have never opened your robots.txt, you are in the majority of popular .uk sites. That means you are allowing AI crawlers by default rather than by choice. That may be fine. You should still know it.
Treat training and search as separate decisions. Blocking a training crawler limits whether your pages feed future models. Blocking an AI search crawler can stop tools such as ChatGPT search or Perplexity from finding and citing you. Many businesses will want the first block without the second.
Remember the limits of the file. It is a polite request. It also does nothing to your normal Google Search listing unless you block Googlebot itself.
The sign on the door is yours to write. Most of the .uk web has left it blank.
How we built it
This analysis fetched robots.txt directly from every .uk website in the Chrome UX Report's top 500,000 sites, using the site list from the HTTP Archive crawl, in October 2026. Of 5,026 .uk sites, 4,377 answered. Sites that did not answer are excluded from every share. These are measured figures: a direct count of each answering site's file. They show the position in October 2026 and say nothing about change over time.
A site "blocks" a crawler only if that crawler is disallowed from the whole site, either by its own rule or by the catch-all (User-agent: *) rule. Partial blocks, such as disallowing a single folder, are not counted. robots.txt is a voluntary standard that crawlers are asked to honour, and it provides no enforcement.
Eleven crawlers were checked. The training crawlers were GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent and Bytespider. The AI search crawlers were OAI-SearchBot, Claude-SearchBot and PerplexityBot. The user-triggered fetcher was ChatGPT-User. Google-Extended governs Gemini training and grounding only. It has no effect on Google Search or AI Overviews.
The popularity bands use Chrome UX Report global rank buckets: the top 100,000 (648 answering .uk sites) and 100,001 to 500,000 (3,729). The difference between the bands (20.4% against 10.2%) is statistically significant (p < 0.001). The universe covers .uk domains only. UK businesses trading on .com or other domains are not included.
Joint ranks (=) mark crawlers that are level at one decimal place.
