Data study
Among the most popular .uk websites, the biggest shut AI crawlers out at double the rate of the rest. Both groups aim mostly at the bots that train AI models, and less often at the ones that fetch pages for AI answers.
Every website keeps a small plain-text file at its front door. It is called robots.txt, and most visitors never see it. For automated crawlers, it is the house rules: which rooms they may enter, and which doors are shut.
Since AI companies began sending their own crawlers across the web, that file has turned into a quiet statement of policy. Marketing circles tend to hold two opposite ideas at once: that the web has slammed the door on AI, or that blocking is something only the giants bother with. A check of the robots.txt files on 4,377 of the most-visited .uk websites tests both ideas.
Neither idea fits the data. Most popular .uk sites leave the door open. Among those that do close it, size makes a clear difference: the biggest .uk sites block AI crawlers about twice as often as the smaller sites beneath them. The bots they turn away are mostly the ones that collect material to train AI models.
Most popular .uk sites let AI crawlers in
Start with the idea of a web in lockdown. Across the 4,377 sites that answered, 11.7% block at least one of the 11 AI crawlers checked from the whole site. Put plainly, the large majority of popular .uk sites place no site-wide block on any of them.
| Crawler type (at least one blocked) | Share of 4,377 sites |
|---|---|
| AI training crawlers | 11.7% |
| AI search crawlers | 7% |
| User-triggered AI fetchers | 5.4% |
That figure is a floor. The check counts only a block on the entire site, either through a rule naming the crawler or through a catch-all rule covering every bot. A site that fences off a few folders is not counted. The check also covers only sites that responded: 4,377 of the 5,026 .uk sites in the Chrome UX Report's top 500,000 answered, and the others are left out of every share. Every figure here is a snapshot from October 2026, so none of it says whether blocking is becoming more or less common.
The biggest .uk sites block twice as often
The second idea, that blocking is a big-site habit, comes closer to the truth. Among the 648 checked .uk sites ranked in the global top 100,000, 20.4% block at least one AI crawler. Below that line, across the rest of the checked sites, the share is 10.2%.
| Group | Share | Sites |
|---|---|---|
| Any AI crawler, all checked sites | 11.7% | 514 of 4,377 |
| Any AI crawler, global top 100,000 | 20.4% | 132 of 648 |
| Any AI crawler, ranked 100,001 to 500,000 | 10.2% | 382 of 3,729 |
| At least one AI training crawler | 11.7% | 514 of 4,377 |
| At least one AI search crawler | 7% | 306 of 4,377 |
| User-triggered AI fetcher | 5.4% | 237 of 4,377 |
| Publish an llms.txt | 17.7% | 775 of 4,377 |
| Of those llms.txt files, same generated "Agent Instructions" template | 58.2% | 451 of 775 |
The obvious objection is the size of the top group. A few hundred sites is a small crowd, and a handful of unusual cases can shift a percentage. The gap is wide enough to answer that objection. A doubling across several hundred sites is very unlikely to come from chance alone, and the smaller group, at several thousand sites, gives a steady comparison point.
The gap does not tell us why. Bigger sites tend to have legal teams, licensing conversations and developers who maintain the robots.txt file. Smaller sites often run whatever default their platform ships with. Those explanations are plausible, but the data cannot confirm them. What the data shows is the size of the difference.
Training bots are the main target
The next idea is that "blocking AI" means one thing. In practice, a crawler can do one of three jobs. Training crawlers gather pages to build AI models. AI search crawlers index pages so an assistant can cite them in answers. User-triggered fetchers load a page live when someone asks a chatbot about it.
Those three jobs are treated very differently. The share of sites blocking at least one training crawler is exactly the same as the share blocking any AI crawler at all, so every blocker in the check shuts out a training bot. Search crawlers are blocked less often, by 7% of sites, and ChatGPT's user-triggered fetcher by 5.4%.
The pattern suggests a deliberate choice by many site owners. They decline to feed model training but stay reachable when a person asks an assistant a question.
GPTBot is the most-blocked crawler
OpenAI's training crawler, GPTBot, is turned away more often than any of the other 10 AI crawlers checked: 9.1% of sites block it from the whole site. Anthropic's ClaudeBot comes next at 8.6%, level with ByteDance's Bytespider.
| Crawler | Operator and purpose | Share blocking | Sites |
|---|---|---|---|
| GPTBot | OpenAI, training (most-blocked of the 11 checked) | 9.1% | 400 |
| ClaudeBot | Anthropic, training | 8.6% | 378 |
| Bytespider | ByteDance, training | 8.6% | 375 |
| CCBot | Common Crawl, open web archive widely used for AI training | 8% | 350 |
| meta-externalagent | Meta, training | 7.5% | 329 |
| Applebot-Extended | Apple, Apple Intelligence training | 7.5% | 328 |
| Google-Extended | Google, Gemini training and grounding (not Search or AI Overviews) | 7.2% | 314 |
| PerplexityBot | Perplexity, AI search | 6.7% | 292 |
| OAI-SearchBot | OpenAI, AI search | 6.4% | 280 |
| ChatGPT-User | OpenAI, user-triggered fetcher | 5.4% | 237 |
| Claude-SearchBot | Anthropic, AI search (least-blocked of the 11 checked) | 4.8% | 210 |
One label causes frequent confusion. Google-Extended, one of the training crawlers counted here, controls only whether Google may use a site's pages for Gemini training and grounding. Blocking it does not remove a site from Google Search or from AI Overviews. A business that blocks Google-Extended expecting to vanish from Google's AI summaries will not get that result.

Google's own list of its crawlers, including what Google-Extended does and does not control. Source: developers.google.com, captured 12 October 2026.
Blocking a search bot keeps a site out of AI answers
Blocking a training crawler and blocking a search crawler have different effects. A training block asks an AI company not to fold a site's pages into its next model. A search block asks the assistant not to index the site for its answers. If ChatGPT, Claude or Perplexity cannot index a page, they are far less likely to cite or link it when a customer asks a relevant question.

OpenAI's documentation separating its training crawler from its search crawler and user-triggered fetcher. Source: platform.openai.com, captured the same day.
Even a block is only a request. Robots.txt asks crawlers to stay out; it has no means of forcing them to comply. Well-behaved crawlers follow the file, but nothing in it stops one that chooses to ignore it.
So the front-door file tells a fairly measured story. Most popular .uk sites leave every door open. The biggest are twice as likely to lock at least one. The doors that do get locked lead mostly to model training, while the routes into AI answers are left open more often.
What this means for UK businesses
- Open your own robots.txt. Add /robots.txt to your domain and read it. Many sites run a platform default that nobody on the team has checked.
- Decide on training and search separately. Blocking training crawlers and blocking AI search crawlers have different effects. You can decline one and allow the other.
- Do not block Google-Extended to escape AI Overviews. It governs Gemini training and grounding, not Google Search.
- Treat robots.txt as a request. It signals your wishes to crawlers that choose to follow it. It does not protect content.
Example only: a robots.txt that declines AI training crawlers but allows AI search crawlers and user-triggered fetchers.
# Decline AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
# Allow AI search crawlers and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /
# Everyone else
User-agent: *
Allow: /
How we built it
This analysis fetched robots.txt and llms.txt directly from every .uk website in that Chrome UX Report list, using the site list from the HTTP Archive crawl, in October 2026. Of the .uk sites in that list, 4,377 answered. Sites that did not answer are excluded from every share, so all percentages use the 4,377 responding sites as the base.
A site counts as blocking a crawler only when that crawler is disallowed from the whole site, either through a rule naming it or through the catch-all User-agent: * rule. Partial blocks covering only some folders are not counted. Robots.txt is a voluntary standard. A block records what a site asks for, not what crawlers actually do.
Crawlers were grouped as follows. Training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User-triggered fetcher: ChatGPT-User.
For the size comparison, sites were split by global Chrome UX Report rank: the top 100,000 (648 .uk sites) against the rest of the list (3,729 sites). The difference between the two blocking shares is statistically significant (p < 0.001). This shows the size of the gap but not its cause.
The check covers .uk domains only, so UK businesses on .com and other domains are not included. All figures are a point-in-time snapshot, and none describes a change over time.
Data table
AI crawlers blocked from the whole site, .uk sites in the Chrome UX Report top 500,000 (October 2026, base 4,377 responding sites)
Blocking by group
Context: platforms on .uk home pages (HTTP Archive, September 2026, 343,930 home pages; detected by home-page fingerprint)
| Platform | Share of .uk home pages | Scope |
|---|---|---|
| WordPress | 38.44% | Highest of the 8 CMS platforms measured |
| WooCommerce | 9.31% | Highest of the 8 e-commerce platforms measured |
| Shopify | 8.13% | Second of the 8 in both lists |

