Data study
A direct check of 4,130 of the most-visited .uk websites finds sites block AI training crawlers about twice as often as the crawlers that gather pages for AI search answers, meaning most businesses are turning away bots that learn from content while leaving the door open to bots that point customers towards it.
Behind the home page of a typical website sits a plain text file that no customer ever reads. It is called robots.txt, and it works like a note taped to the stockroom door. It tells visiting machines which rooms they may enter and which they should leave alone. Those notes were written with search engines in mind. Now a newer set of visitors is reading them: the crawlers that feed artificial intelligence.
A popular story says British websites have bolted that door against AI altogether. The notes themselves tell a narrower story. Across 4,130 of the most-visited .uk websites, checked in November 2026, sites turned away the bots that train AI models about twice as often as the bots that collect pages for AI search answers.
The difference matters to anyone who sells online. Training crawlers gather text to build the models. Search crawlers index pages so that a shop, a guide or a price list can be found and linked when someone asks ChatGPT, Claude or Perplexity a question. Each crawler is asked in and turned away separately. A site can refuse one and welcome the other.
Training crawlers are turned away about twice as often as search crawlers
About one in seven sites in the check, 14.5%, blocked at least one AI training crawler from the whole site, against 6.9% that blocked at least one AI search crawler. ChatGPT's user-triggered fetcher, which loads a page live when a ChatGPT user asks about it, was blocked least often of the three groups. Because it acts on a single person's request, it is reported separately and should not be read as a search crawler.
| Group of AI crawlers checked | Sites blocking at least one (%) |
|---|---|
| Training crawlers (seven checked) | 14.5 |
| AI search crawlers (three checked) | 6.9 |
| ChatGPT's user-triggered fetcher | 5.8 |
Two limits apply to every figure in this piece. A site counts as blocking only when it shuts a crawler out of the whole site, either by naming it or through a catch-all rule; blocking a few folders does not count. And robots.txt is a request that crawlers are asked to honour. It does not physically stop them.
GPTBot is the most-blocked of the eleven crawlers checked
OpenAI's training crawler, GPTBot, tops the blocked list: 11.9% of sites in the check shut it out. Anthropic's training crawler, ClaudeBot, sits a hair behind, level with two other training crawlers.
At the other end, Anthropic's search crawler, Claude-SearchBot, was the least blocked of the eleven, refused by 5.3% of sites. Ranked in full, the seven training crawlers fill the top seven places and the three search crawlers and ChatGPT's fetcher fill the bottom four. The same operator can be treated two ways on one site: welcome to index a product page for its search answers, unwelcome to copy it into its training data.
| Crawler | What it does | Sites blocking it (%) |
|---|---|---|
| GPTBot | OpenAI, trains AI models | 11.9 |
| ClaudeBot | Anthropic, trains AI models | 11.5 |
| Bytespider | ByteDance, trains AI models | 11.5 |
| CCBot | Common Crawl, open web archive widely used for AI training | 11.5 |
| meta-externalagent | Meta, trains AI models | 11.2 |
| Applebot-Extended | Apple, Apple Intelligence training | 10.6 |
| Google-Extended | Google, Gemini training and grounding | 10.0 |
| PerplexityBot | Perplexity, indexes pages for its search results | 6.5 |
| OAI-SearchBot | OpenAI, indexes pages for ChatGPT search results | 6.2 |
| ChatGPT-User | OpenAI, fetches a page live when a ChatGPT user asks | 5.8 |
| Claude-SearchBot | Anthropic, indexes pages for Claude search results | 5.3 |
Google's AI answers sit behind a different door
Google does not fit neatly into either column. Its Google-Extended token is a training control: it governs whether pages are used for Gemini, and blocking it does not remove a site from Google Search or from AI Overviews. Those answers are served by ordinary Googlebot, the same crawler that powers the familiar results page. That is why Google-Extended is counted with the training crawlers here, and why no Google crawler appears in the search column.
The biggest .uk sites are the readiest to say no
Size changes the picture. Among the .uk sites in the global top 100,000, 21.1% blocked at least one AI crawler, against 13.4% of those ranked 100,001 to 500,000. Put plainly, roughly one big site in five sends some AI crawler away, while further down the rankings it is closer to one in seven or eight.
| Measure (November 2026, .uk sites that answered) | Sites | Share (%) |
|---|---|---|
| Block at least one AI crawler, all sites | 598 of 4,130 | 14.5 |
| Block at least one training crawler | 597 of 4,130 | 14.5 |
| Block at least one AI search crawler | 285 of 4,130 | 6.9 |
| Block ChatGPT's user-triggered fetcher | 240 of 4,130 | 5.8 |
| Block at least one AI crawler, global top 100,000 | 126 of 597 | 21.1 |
| Block at least one AI crawler, ranked 100,001 to 500,000 | 472 of 3,533 | 13.4 |
| Publish an llms.txt file | 745 of 4,130 | 18.0 |
Could a handful of heavyweight publishers be skewing the totals? Each site counts once in these shares, whatever its traffic, and the top tier is a small slice of the check. The busiest sites shape their own figure, not everyone's. One limit is worth carrying forward: this size comparison covers blocking of any AI crawler, so it does not say whether big sites lean harder against training bots or search bots.
So the received wisdom gets one thing right. A clear minority of busy .uk sites do refuse AI crawlers, and the largest sites refuse more often. What the notes on the door do not show is a blanket ban. The refusals fall mostly on the bots that learn from content, and far less on the bots that point shoppers towards it.
What this means for UK businesses
Training and search are separate decisions. Refusing to let AI companies train on your content does not require you to vanish from AI search answers. You can make each choice on its own terms.
Check your catch-all rule first. A blanket "Disallow" aimed at every user-agent shuts out AI search crawlers along with everything else. Many unintended blocks start there.
Treat Google separately. Blocking Google-Extended affects Gemini training. It will not take you out of Google Search or AI Overviews, and it will not put you into them either.
Remember it is a request. robots.txt tells well-behaved crawlers what you would like. If you need to enforce a block, that is a job for your server or hosting provider.
The note on the stockroom door is short, and few customers will ever see it. For shoppers who put their questions to AI assistants, though, it decides whether an assistant can find the shop at all.
How we built it
The check fetched robots.txt and llms.txt directly from every .uk website in the Chrome UX Report's top 500,000 sites, using the site list from the HTTP Archive crawl, in November 2026. Of 4,865 .uk sites, 4,130 answered; the rest are left out of every share. These are measured figures: a direct read of each site's file, with no sampling or modelling.
A site counts as blocking a crawler only when that crawler is disallowed from the whole site, through its own rule or the catch-all. Partial blocks of some folders are not counted. robots.txt is a request crawlers are asked to honour; the check records what sites ask for.
Eleven crawlers were checked and grouped using each operator's own description. Training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User-triggered: ChatGPT-User. Google-Extended sits with training because it governs Gemini training only; Google's AI Overviews and AI Mode are served by ordinary Googlebot, so Google has no entry in the search group. Group figures count a site once if it blocks at least one crawler in that group. The size comparison uses the Chrome UX Report's global rank buckets. All figures are a single snapshot, so no change over time is claimed. No test of statistical significance was run on the gaps; they are described as observed shares of a complete set of responding sites.

