AI crawler access is a commercial decision for UK businesses
Data study
Blocking an AI training crawler and blocking an AI search crawler are two separate choices — and most UK businesses only need to make one of them. New data from 4,377 of the most-visited .uk websites shows many site owners are already treating them differently.
Publishers are making a licensing choice
The New York Times is among a wave of new members joining SPUR, a publisher coalition seeking fair payment terms from AI companies, Press Gazette reports. For large publishers, their content is the product. Controlling which AI firms can read it, and on what terms, is a licensing decision.
Most UK businesses are in a different position. A plumber, a law firm or an online shop is not trying to license its content. It wants customers to find it. The useful question is not "should AI pay me?" It is "do I want to appear when someone asks ChatGPT, Gemini or Perplexity for a recommendation?"
Training bots and search bots do different jobs
AI companies run several crawlers, and they do different jobs. Some collect pages to train future models. Others fetch pages so an AI assistant can find and cite them in answers. A robots.txt file can treat each one differently.
A check of the 4,377 most-visited .uk websites found that 11.7% block at least one AI training crawler from the whole site. That compares with 7% that block at least one AI search crawler, and 5.4% that block ChatGPT's user-triggered fetcher. These figures come from robots.txt files in October 2026. They show what sites ask crawlers to do, not whether crawlers obey, and they only count whole-site blocks.
GPTBot, OpenAI's training crawler, was the most blocked of the 11 AI crawlers checked: 9.1% of those sites shut it out. Claude-SearchBot, which Anthropic uses to index pages for Claude's search results, was the least blocked of the 11, at 4.8%. The gap suggests many site owners already treat training and search as separate decisions.
Size matters too. Among the 648 .uk sites in the global top 100,000, 20.4% block at least one AI crawler. That compares with 10.2% of the 3,729 sites ranked 100,001 to 500,000. This is a single snapshot by traffic rank, so it shows a difference in level, not a trend. Bigger sites are more often publishers with content worth licensing.
OpenAI's documentation lists its training crawler and its search crawler as separate user agents. Source: platform.openai.com, captured in October 2026.
What this means for UK businesses
- Decide on purpose, not by default. Check your robots.txt. Some plugins, templates and security tools add AI blocks without you choosing them.
- Treat training and search separately. If you have no plans to license your content, blocking AI search crawlers may simply hide you from AI answers your customers are already reading.
- Blocking training bots is a fair choice. If you do not want your content used to train models, you can say so without dropping out of AI search.
- Review it regularly. AI companies add and rename crawlers. Check their own documentation when you update your file.
Example: a robots.txt that blocks AI training crawlers but allows AI search crawlers
# Block AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Allow AI search and user-requested fetches
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
This is an example only. Google-Extended controls whether Google can use your content for its AI models. It does not affect normal Google Search crawling. Test any change before you publish it, and keep your existing rules for other bots.
Analysis compiled by Sayu (sayu.co.uk)
