Cloudflare pilot widens ChatGPT search: check robots.txt allows its search bot
AI search guidance
ChatGPT search can only quote pages its crawler is allowed to reach. UK businesses should check whether their robots.txt is quietly keeping them out of AI search answers.
What was reported
Search Engine Journal reports that Cloudflare says its network signals helped OpenAI index hundreds of millions of new pages for ChatGPT search. This is Cloudflare's account of a pilot, as reported. It does not say whether any particular website was included. Nothing in the report suggests it changes how any single site is indexed.
The report does show OpenAI working to widen the pool of pages ChatGPT search can draw on. That raises a fair question for every UK business. When someone asks ChatGPT about the products or services you sell, can it reach your pages at all?
The answer often sits in one small file on your website: robots.txt. It tells crawlers which parts of a site they may visit. If it shuts out ChatGPT's search crawler, your pages are unlikely to be among those it can quote, however many new pages the index gains.
Training bots and search bots do different jobs
OpenAI runs more than one crawler, and each has its own name in robots.txt:
- GPTBot collects pages that may be used to train OpenAI's AI models.
- OAI-SearchBot finds and indexes pages so they can appear, with links, in ChatGPT search answers.
- ChatGPT-User fetches a page when a person asks ChatGPT to look at it.
Anthropic draws a similar line between ClaudeBot, for training, and Claude-SearchBot, for its search results.
Blocking a training bot and blocking a search bot are two separate choices with different effects. Blocking GPTBot keeps your content out of future model training. Blocking OAI-SearchBot keeps your pages out of the index ChatGPT search quotes from. That means fewer chances to be named, linked or recommended when customers ask it a question.
Many sites made their AI crawler decision in one sweep. They used a block list copied from elsewhere, a plug-in setting or a one-click option from a hosting or security service. Any of these can treat every AI bot the same way. That may be exactly what a business wants. It is still worth checking that it was a choice made on purpose.
Allowing a search crawler removes a barrier. It does not guarantee your pages will be indexed or quoted. Page quality, clear content and a site that loads properly still matter.
What this means for UK businesses
- Look at your robots.txt today. Visit yourdomain.co.uk/robots.txt and search for the names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and Claude-SearchBot.
- Treat training and search as two decisions. Agree on each one with whoever owns your website and your marketing.
- Check settings outside the file. Hosting, firewall and security services can block AI crawlers before they ever read robots.txt.
- Keep expectations realistic. The reported pilot does not change how your own site is indexed. Letting the search crawler in makes your pages eligible. It does not promise they will be quoted.
- Recheck after changes. A new plug-in, theme or hosting setting can quietly rewrite the rules.
Analysis compiled by Sayu (sayu.co.uk), a UK digital marketing agency, in response to reporting by Search Engine Journal.
