Judging AI visibility? First check your site lets AI search bots in
Data study
Retailers are comparing how often AI tools mention them, but a short file on their own website may decide whether those tools can read their pages at all.
Why AI visibility is in the news
Retail Week and PPC Land are both reporting on the new Semrush AI Visibility Index. It ranks retail brands by how often they appear in answers from ChatGPT, Gemini and Perplexity. This is third-party data, and it measures mentions and citations, not the reasons behind them.
PPC Land's coverage notes that UK AI platforms share 63 to 70 of their top 100 brands, against 70 to 78 in the US. In other words, the tools do not all agree, and a plan built for US AI search may not carry over to Britain.
The stakes are real. Online accounted for 27.3% of UK retail sales in August 2026, according to ONS figures in Sayu's study of online retail spending. That is one month's figure, and monthly shares move around.
The technical choice behind AI answers
Before any business judges how visible it is in AI answers, it should check one simple thing: does it let AI search crawlers in?
AI companies run different bots for different jobs. Some collect pages to train AI models. Others fetch pages so an AI tool can find and cite them in search-style answers. Blocking a training bot and blocking a search bot are separate choices with separate results. A site cannot be cited by an AI search tool if it shuts out that tool's crawler.
Sayu's study of AI crawler blocking checked the robots.txt files of the 4,377 most-visited .uk websites in October 2026. It covers .uk domains only, and only whole-site blocks. A robots.txt file states what a site asks of bots. It does not prove that every bot obeys.
The results:
- 11.7% of the sites checked block at least one AI training crawler, against 7% that block at least one AI search crawler and 5.4% that block ChatGPT's user-triggered fetcher.
- GPTBot, OpenAI's training crawler, is the most-blocked of the 11 AI crawlers checked, shut out by 9.1% of sites.
- Claude-SearchBot, which indexes pages for Claude's search results, is the least-blocked of the 11, shut out by 4.8%.
- Bigger sites block more often: 20.4% of the 648 checked .uk sites in the global top 100,000 block at least one AI crawler, against 10.2% of the 3,729 ranked 100,001 to 500,000.
These are aggregates. They say nothing about any single retailer or any brand in the index. But they show that many sites treat training and search bots differently, and that some search bots are shut out too, whether on purpose or by accident.
OpenAI's own page listing its crawlers and what each one is for. Source: platform.openai.com, captured October 2026.
What this means for UK businesses
- Read your robots.txt. It sits at yourdomain.co.uk/robots.txt. Look for any AI crawler names with "Disallow: /" under them.
- Separate training from search. Decide whether you mind your content being used to train AI models. That is a different question from whether you want AI search tools to find and cite you.
- Check platform defaults. Some website platforms, plug-ins and security tools add bot rules on your behalf. Confirm the rules match what you actually want.
- Audit before you compare. If your brand looks weak in AI answers, rule out a blocked crawler before you change your content or spend more.
Analysis compiled by Sayu (sayu.co.uk), using its own crawler and online retail studies alongside published third-party reporting.
