Back to guides

Most-visited .uk sites block AI crawlers twice as often

Share
Most-visited .uk sites block AI crawlers twice as often

Data study

Among the most popular .uk websites, the biggest shut AI crawlers out at double the rate of the rest. Both groups aim mostly at the bots that train AI models, and less often at the ones that fetch pages for AI answers.

11.7%
Popular .uk sites blocking at least one AI crawler
Site-wide blocks only; partial folder blocks not counted. Base: 4,377 responding sites, October 2026.
20.4%
.uk sites in the global top 100,000 blocking AI crawlers
648 .uk sites ranked in the global top 100,000. October 2026.
10.2%
.uk sites ranked 100,001 to 500,000 blocking AI crawlers
3,729 sites. Compared with 20.4% for the top 100,000 group.
9.1%
Sites blocking GPTBot, the most-blocked AI crawler
400 of 4,377 responding sites. October 2026.

Every website keeps a small plain-text file at its front door. It is called robots.txt, and most visitors never see it. For automated crawlers, it is the house rules: which rooms they may enter, and which doors are shut.

Since AI companies began sending their own crawlers across the web, that file has turned into a quiet statement of policy. Marketing circles tend to hold two opposite ideas at once: that the web has slammed the door on AI, or that blocking is something only the giants bother with. A check of the robots.txt files on 4,377 of the most-visited .uk websites tests both ideas.

Neither idea fits the data. Most popular .uk sites leave the door open. Among those that do close it, size makes a clear difference: the biggest .uk sites block AI crawlers about twice as often as the smaller sites beneath them. The bots they turn away are mostly the ones that collect material to train AI models.

The biggest .uk sites block twice as often

The second idea, that blocking is a big-site habit, comes closer to the truth. Among the 648 checked .uk sites ranked in the global top 100,000, 20.4% block at least one AI crawler. Below that line, across the rest of the checked sites, the share is 10.2%.

0%20%40%60%Any AI crawler, all checked sites · Share: 11.7%11.7%Any AI crawler, all check…Any AI crawler, global top 100,000 · Share: 20.4%20.4%Any AI crawler, global to…Any AI crawler, ranked 100,001 to 500,000 · Share: 10.2%10.2%Any AI crawler, ranked 10…At least one AI training crawler · Share: 11.7%11.7%At least one AI training …At least one AI search crawler · Share: 7.0%7.0%At least one AI search cr…User-triggered AI fetcher · Share: 5.4%5.4%User-triggered AI fetcherPublish an llms.txt · Share: 17.7%17.7%Publish an llms.txtOf those llms.txt files, same generated "Agent Instructions" template · Share: 58.2%58.2%Of those llms.txt files, …ShareGroup
Group Share Sites
Any AI crawler, all checked sites 11.7% 514 of 4,377
Any AI crawler, global top 100,000 20.4% 132 of 648
Any AI crawler, ranked 100,001 to 500,000 10.2% 382 of 3,729
At least one AI training crawler 11.7% 514 of 4,377
At least one AI search crawler 7% 306 of 4,377
User-triggered AI fetcher 5.4% 237 of 4,377
Publish an llms.txt 17.7% 775 of 4,377
Of those llms.txt files, same generated "Agent Instructions" template 58.2% 451 of 775

The obvious objection is the size of the top group. A few hundred sites is a small crowd, and a handful of unusual cases can shift a percentage. The gap is wide enough to answer that objection. A doubling across several hundred sites is very unlikely to come from chance alone, and the smaller group, at several thousand sites, gives a steady comparison point.

The gap does not tell us why. Bigger sites tend to have legal teams, licensing conversations and developers who maintain the robots.txt file. Smaller sites often run whatever default their platform ships with. Those explanations are plausible, but the data cannot confirm them. What the data shows is the size of the difference.

Training bots are the main target

The next idea is that "blocking AI" means one thing. In practice, a crawler can do one of three jobs. Training crawlers gather pages to build AI models. AI search crawlers index pages so an assistant can cite them in answers. User-triggered fetchers load a page live when someone asks a chatbot about it.

Those three jobs are treated very differently. The share of sites blocking at least one training crawler is exactly the same as the share blocking any AI crawler at all, so every blocker in the check shuts out a training bot. Search crawlers are blocked less often, by 7% of sites, and ChatGPT's user-triggered fetcher by 5.4%.

The pattern suggests a deliberate choice by many site owners. They decline to feed model training but stay reachable when a person asks an assistant a question.

GPTBot is the most-blocked crawler

OpenAI's training crawler, GPTBot, is turned away more often than any of the other 10 AI crawlers checked: 9.1% of sites block it from the whole site. Anthropic's ClaudeBot comes next at 8.6%, level with ByteDance's Bytespider.

0%2%4%6%8%10%GPTBotGPTBot · Share blocking: 9.1%9.1%ClaudeBotClaudeBot · Share blocking: 8.6%8.6%BytespiderBytespider · Share blocking: 8.6%8.6%CCBotCCBot · Share blocking: 8.0%8.0%meta-externalagentmeta-externalagent · Share blocking: 7.5%7.5%Applebot-ExtendedApplebot-Extended · Share blocking: 7.5%7.5%Google-ExtendedGoogle-Extended · Share blocking: 7.2%7.2%PerplexityBotPerplexityBot · Share blocking: 6.7%6.7%OAI-SearchBotOAI-SearchBot · Share blocking: 6.4%6.4%ChatGPT-UserChatGPT-User · Share blocking: 5.4%5.4%Claude-SearchBotClaude-SearchBot · Share blocking: 4.8%4.8%Share blocking
Crawler Operator and purpose Share blocking Sites
GPTBot OpenAI, training (most-blocked of the 11 checked) 9.1% 400
ClaudeBot Anthropic, training 8.6% 378
Bytespider ByteDance, training 8.6% 375
CCBot Common Crawl, open web archive widely used for AI training 8% 350
meta-externalagent Meta, training 7.5% 329
Applebot-Extended Apple, Apple Intelligence training 7.5% 328
Google-Extended Google, Gemini training and grounding (not Search or AI Overviews) 7.2% 314
PerplexityBot Perplexity, AI search 6.7% 292
OAI-SearchBot OpenAI, AI search 6.4% 280
ChatGPT-User OpenAI, user-triggered fetcher 5.4% 237
Claude-SearchBot Anthropic, AI search (least-blocked of the 11 checked) 4.8% 210

One label causes frequent confusion. Google-Extended, one of the training crawlers counted here, controls only whether Google may use a site's pages for Gemini training and grounding. Blocking it does not remove a site from Google Search or from AI Overviews. A business that blocks Google-Extended expecting to vanish from Google's AI summaries will not get that result.

Screenshot: Google's own list of its crawlers, including what Google-Extended does and does not control

Google's own list of its crawlers, including what Google-Extended does and does not control. Source: developers.google.com, captured 12 October 2026.

Blocking a search bot keeps a site out of AI answers

Blocking a training crawler and blocking a search crawler have different effects. A training block asks an AI company not to fold a site's pages into its next model. A search block asks the assistant not to index the site for its answers. If ChatGPT, Claude or Perplexity cannot index a page, they are far less likely to cite or link it when a customer asks a relevant question.

Screenshot: OpenAI's documentation separating its training crawler from its search crawler and user-triggered fetcher

OpenAI's documentation separating its training crawler from its search crawler and user-triggered fetcher. Source: platform.openai.com, captured the same day.

Even a block is only a request. Robots.txt asks crawlers to stay out; it has no means of forcing them to comply. Well-behaved crawlers follow the file, but nothing in it stops one that chooses to ignore it.

So the front-door file tells a fairly measured story. Most popular .uk sites leave every door open. The biggest are twice as likely to lock at least one. The doors that do get locked lead mostly to model training, while the routes into AI answers are left open more often.

What this means for UK businesses

  • Open your own robots.txt. Add /robots.txt to your domain and read it. Many sites run a platform default that nobody on the team has checked.
  • Decide on training and search separately. Blocking training crawlers and blocking AI search crawlers have different effects. You can decline one and allow the other.
  • Do not block Google-Extended to escape AI Overviews. It governs Gemini training and grounding, not Google Search.
  • Treat robots.txt as a request. It signals your wishes to crawlers that choose to follow it. It does not protect content.

Example only: a robots.txt that declines AI training crawlers but allows AI search crawlers and user-triggered fetchers.

# Decline AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Allow AI search crawlers and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /

# Everyone else
User-agent: *
Allow: /

How we built it

This analysis fetched robots.txt and llms.txt directly from every .uk website in that Chrome UX Report list, using the site list from the HTTP Archive crawl, in October 2026. Of the .uk sites in that list, 4,377 answered. Sites that did not answer are excluded from every share, so all percentages use the 4,377 responding sites as the base.

A site counts as blocking a crawler only when that crawler is disallowed from the whole site, either through a rule naming it or through the catch-all User-agent: * rule. Partial blocks covering only some folders are not counted. Robots.txt is a voluntary standard. A block records what a site asks for, not what crawlers actually do.

Crawlers were grouped as follows. Training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User-triggered fetcher: ChatGPT-User.

For the size comparison, sites were split by global Chrome UX Report rank: the top 100,000 (648 .uk sites) against the rest of the list (3,729 sites). The difference between the two blocking shares is statistically significant (p < 0.001). This shows the size of the gap but not its cause.

The check covers .uk domains only, so UK businesses on .com and other domains are not included. All figures are a point-in-time snapshot, and none describes a change over time.

Data table

AI crawlers blocked from the whole site, .uk sites in the Chrome UX Report top 500,000 (October 2026, base 4,377 responding sites)

Blocking by group

Context: platforms on .uk home pages (HTTP Archive, September 2026, 343,930 home pages; detected by home-page fingerprint)

0%10%20%30%40%WordPress · Share of .uk home pages: 38.44%WordPressWooCommerce · Share of .uk home pages: 9.31%9.31%WooCommerceShopify · Share of .uk home pages: 8.13%8.13%ShopifyShare of .uk home pagesPlatform
Platform Share of .uk home pages Scope
WordPress 38.44% Highest of the 8 CMS platforms measured
WooCommerce 9.31% Highest of the 8 e-commerce platforms measured
Shopify 8.13% Second of the 8 in both lists
Share
AI search

Does AI search recommend your business?

Get a free AI visibility snapshot: how ChatGPT, Google's AI answers and other AI platforms see your brand, emailed to you within one working day.

Free. No spam, and we never share your details. What’s in the report?

or see our AI SEO services
Practical SEO, PPC and web tips One email a month. Unsubscribe anytime.