Back to guides

Popular .uk websites bar AI training bots more than AI search bots

Share
Popular .uk websites bar AI training bots more than AI search bots

Data study

A direct check of 4,130 of the most-visited .uk websites finds sites block AI training crawlers about twice as often as the crawlers that gather pages for AI search answers, meaning most businesses are turning away bots that learn from content while leaving the door open to bots that point customers towards it.

14.5%
Sites blocking at least one AI training crawler
Of 4,130 responding .uk sites, November 2026
6.9%
Sites blocking at least one AI search crawler
Of 4,130 responding .uk sites, November 2026
11.9%
Sites blocking GPTBot, the most-blocked crawler
Of 4,130 responding .uk sites, November 2026
5.3%
Sites blocking Claude-SearchBot, the least-blocked crawler
Of 4,130 responding .uk sites, November 2026

Behind the home page of a typical website sits a plain text file that no customer ever reads. It is called robots.txt, and it works like a note taped to the stockroom door. It tells visiting machines which rooms they may enter and which they should leave alone. Those notes were written with search engines in mind. Now a newer set of visitors is reading them: the crawlers that feed artificial intelligence.

A popular story says British websites have bolted that door against AI altogether. The notes themselves tell a narrower story. Across 4,130 of the most-visited .uk websites, checked in November 2026, sites turned away the bots that train AI models about twice as often as the bots that collect pages for AI search answers.

The difference matters to anyone who sells online. Training crawlers gather text to build the models. Search crawlers index pages so that a shop, a guide or a price list can be found and linked when someone asks ChatGPT, Claude or Perplexity a question. Each crawler is asked in and turned away separately. A site can refuse one and welcome the other.

Training crawlers are turned away about twice as often as search crawlers

About one in seven sites in the check, 14.5%, blocked at least one AI training crawler from the whole site, against 6.9% that blocked at least one AI search crawler. ChatGPT's user-triggered fetcher, which loads a page live when a ChatGPT user asks about it, was blocked least often of the three groups. Because it acts on a single person's request, it is reported separately and should not be read as a search crawler.

051015Training crawlers (seven checked) · Sites blocking at least one (%): 14.514.5Training crawlers (seven …AI search crawlers (three checked) · Sites blocking at least one (%): 6.96.9AI search crawlers (three…ChatGPT's user-triggered fetcher · Sites blocking at least one (%): 5.85.8ChatGPT's user-triggered …Sites blocking at least one (%)Group of AI crawlers checked
Group of AI crawlers checked Sites blocking at least one (%)
Training crawlers (seven checked) 14.5
AI search crawlers (three checked) 6.9
ChatGPT's user-triggered fetcher 5.8

Two limits apply to every figure in this piece. A site counts as blocking only when it shuts a crawler out of the whole site, either by naming it or through a catch-all rule; blocking a few folders does not count. And robots.txt is a request that crawlers are asked to honour. It does not physically stop them.

GPTBot is the most-blocked of the eleven crawlers checked

OpenAI's training crawler, GPTBot, tops the blocked list: 11.9% of sites in the check shut it out. Anthropic's training crawler, ClaudeBot, sits a hair behind, level with two other training crawlers.

At the other end, Anthropic's search crawler, Claude-SearchBot, was the least blocked of the eleven, refused by 5.3% of sites. Ranked in full, the seven training crawlers fill the top seven places and the three search crawlers and ChatGPT's fetcher fill the bottom four. The same operator can be treated two ways on one site: welcome to index a product page for its search answers, unwelcome to copy it into its training data.

051015GPTBotGPTBot · Sites blocking it (%): 11.911.9ClaudeBotClaudeBot · Sites blocking it (%): 11.511.5BytespiderBytespider · Sites blocking it (%): 11.511.5CCBotCCBot · Sites blocking it (%): 11.511.5meta-externalagentmeta-externalagent · Sites blocking it (%): 11.211.2Applebot-ExtendedApplebot-Extended · Sites blocking it (%): 10.610.6Google-ExtendedGoogle-Extended · Sites blocking it (%): 10.010.0PerplexityBotPerplexityBot · Sites blocking it (%): 6.56.5OAI-SearchBotOAI-SearchBot · Sites blocking it (%): 6.26.2ChatGPT-UserChatGPT-User · Sites blocking it (%): 5.85.8Claude-SearchBotClaude-SearchBot · Sites blocking it (%): 5.35.3Sites blocking it (%)
Crawler What it does Sites blocking it (%)
GPTBot OpenAI, trains AI models 11.9
ClaudeBot Anthropic, trains AI models 11.5
Bytespider ByteDance, trains AI models 11.5
CCBot Common Crawl, open web archive widely used for AI training 11.5
meta-externalagent Meta, trains AI models 11.2
Applebot-Extended Apple, Apple Intelligence training 10.6
Google-Extended Google, Gemini training and grounding 10.0
PerplexityBot Perplexity, indexes pages for its search results 6.5
OAI-SearchBot OpenAI, indexes pages for ChatGPT search results 6.2
ChatGPT-User OpenAI, fetches a page live when a ChatGPT user asks 5.8
Claude-SearchBot Anthropic, indexes pages for Claude search results 5.3

Google's AI answers sit behind a different door

Google does not fit neatly into either column. Its Google-Extended token is a training control: it governs whether pages are used for Gemini, and blocking it does not remove a site from Google Search or from AI Overviews. Those answers are served by ordinary Googlebot, the same crawler that powers the familiar results page. That is why Google-Extended is counted with the training crawlers here, and why no Google crawler appears in the search column.

The biggest .uk sites are the readiest to say no

Size changes the picture. Among the .uk sites in the global top 100,000, 21.1% blocked at least one AI crawler, against 13.4% of those ranked 100,001 to 500,000. Put plainly, roughly one big site in five sends some AI crawler away, while further down the rankings it is closer to one in seven or eight.

0102030Block at least one AI crawler, all sites · Share (%): 14.514.5Block at least one AI cra…Block at least one training crawler · Share (%): 14.514.5Block at least one traini…Block at least one AI search crawler · Share (%): 6.96.9Block at least one AI sea…Block ChatGPT's user-triggered fetcher · Share (%): 5.85.8Block ChatGPT's user-trig…Block at least one AI crawler, global top 100,000 · Share (%): 21.121.1Block at least one AI cra…Block at least one AI crawler, ranked 100,001 to 500,000 · Share (%): 13.413.4Block at least one AI cra…Publish an llms.txt file · Share (%): 18.018.0Publish an llms.txt fileShare (%)Measure (November 2026, .uk sites that answered)
Measure (November 2026, .uk sites that answered) Sites Share (%)
Block at least one AI crawler, all sites 598 of 4,130 14.5
Block at least one training crawler 597 of 4,130 14.5
Block at least one AI search crawler 285 of 4,130 6.9
Block ChatGPT's user-triggered fetcher 240 of 4,130 5.8
Block at least one AI crawler, global top 100,000 126 of 597 21.1
Block at least one AI crawler, ranked 100,001 to 500,000 472 of 3,533 13.4
Publish an llms.txt file 745 of 4,130 18.0

Could a handful of heavyweight publishers be skewing the totals? Each site counts once in these shares, whatever its traffic, and the top tier is a small slice of the check. The busiest sites shape their own figure, not everyone's. One limit is worth carrying forward: this size comparison covers blocking of any AI crawler, so it does not say whether big sites lean harder against training bots or search bots.

So the received wisdom gets one thing right. A clear minority of busy .uk sites do refuse AI crawlers, and the largest sites refuse more often. What the notes on the door do not show is a blanket ban. The refusals fall mostly on the bots that learn from content, and far less on the bots that point shoppers towards it.

What this means for UK businesses

Training and search are separate decisions. Refusing to let AI companies train on your content does not require you to vanish from AI search answers. You can make each choice on its own terms.

Check your catch-all rule first. A blanket "Disallow" aimed at every user-agent shuts out AI search crawlers along with everything else. Many unintended blocks start there.

Treat Google separately. Blocking Google-Extended affects Gemini training. It will not take you out of Google Search or AI Overviews, and it will not put you into them either.

Remember it is a request. robots.txt tells well-behaved crawlers what you would like. If you need to enforce a block, that is a job for your server or hosting provider.

The note on the stockroom door is short, and few customers will ever see it. For shoppers who put their questions to AI assistants, though, it decides whether an assistant can find the shop at all.

How we built it

The check fetched robots.txt and llms.txt directly from every .uk website in the Chrome UX Report's top 500,000 sites, using the site list from the HTTP Archive crawl, in November 2026. Of 4,865 .uk sites, 4,130 answered; the rest are left out of every share. These are measured figures: a direct read of each site's file, with no sampling or modelling.

A site counts as blocking a crawler only when that crawler is disallowed from the whole site, through its own rule or the catch-all. Partial blocks of some folders are not counted. robots.txt is a request crawlers are asked to honour; the check records what sites ask for.

Eleven crawlers were checked and grouped using each operator's own description. Training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User-triggered: ChatGPT-User. Google-Extended sits with training because it governs Gemini training only; Google's AI Overviews and AI Mode are served by ordinary Googlebot, so Google has no entry in the search group. Group figures count a site once if it blocks at least one crawler in that group. The size comparison uses the Chrome UX Report's global rank buckets. All figures are a single snapshot, so no change over time is claimed. No test of statistical significance was run on the gaps; they are described as observed shares of a complete set of responding sites.

Share
AI search

Does AI search recommend your business?

Get a free AI visibility snapshot: how ChatGPT, Google's AI answers and other AI platforms see your brand, emailed to you within one working day.

Free. No spam, and we never share your details. What’s in the report?

or see our AI SEO services
Practical SEO, PPC and web tips One email a month. Unsubscribe anytime.