Before AI shopping arrives, check AI assistants can read your shop
Retail research
Consumer adoption of AI-powered shopping is still in its early stages — but UK retailers who act now can make sure their products are visible when shoppers start asking AI assistants what to buy.
A slow start, according to Paxon
Retailers are still waiting for AI-powered shopping to disrupt how people buy, according to retail logistics business Paxon. Its view, reported by InternetRetailing, is that consumer adoption is still in its early stages. These are Paxon's conclusions, drawn from its own research.
A slow start is not a reason to ignore the shift. It is a window to prepare. Online selling already matters a great deal to UK retail. Online accounted for 27.3% of UK retail sales in August 2026, according to ONS figures gathered in Sayu's study. That was up from 26.1% in August 2025, a rise of 1.2 percentage points. These are single-month readings, and the online share moves from month to month. For context, the highest monthly share on record is 37.8%, in January 2021, during lockdown. That was an unusual peak, not a normal level.
When shoppers do start asking AI assistants what to buy, much of that spending could be shaped by whose products the assistants can see.
AI assistants can only recommend what they can read
ChatGPT, Gemini, Perplexity and Claude all fetch web pages to build their answers. Each site's robots.txt file tells these automated visitors, called crawlers, what they may and may not read.
The detail that matters is that AI companies run different crawlers for different jobs:
- Training crawlers collect pages to help build future AI models. OpenAI's GPTBot and Anthropic's ClaudeBot are examples.
- AI search crawlers index pages so the assistant can cite and link to them in answers. Examples include OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and PerplexityBot.
- User-triggered fetchers visit a page when a person asks the assistant about it. Examples include ChatGPT-User, Claude-User and Perplexity-User.
Many businesses block AI crawlers because they don't want their content used for training. That is a fair choice. The risk is a blanket rule that also shuts out the search crawlers and fetchers. If that happens, a shop may be left out of AI answers when a shopper asks for product suggestions.
This is about visibility in AI assistants' answers. It is a separate question from how a site ranks in ordinary Google results.
OpenAI's own list of its crawlers and what each one does. Source: platform.openai.com, captured in October 2026.
What this means for UK businesses
- Read your robots.txt file. Open yoursite.co.uk/robots.txt in a browser and look for any AI crawler names.
- Decide on training and AI search separately. You can say no to training and still say yes to being found in AI answers.
- Check what your platform or plug-ins add. Some security tools and themes add AI blocks without telling you.
- Recheck after changes. A site migration or new firewall rule can quietly undo your settings.
- Do it before the Golden Quarter. Peak trading is the worst time to find out an assistant can't see your products.
Each AI company publishes its own crawler names, and these can change. Check the operators' documentation before you copy any list. Keep any existing rules for checkout, basket and account pages in place.
Analysis compiled by Sayu (sayu.co.uk), drawing on Paxon's findings as reported by InternetRetailing and ONS retail figures.
