SEO & AI search
OpenAI, Google, Microsoft and Anthropic are aligning on a single opt-out signal for AI training — but blocking everything AI-related can quietly remove your site from the answers that send customers your way.
What the companies announced
The Verge reports that OpenAI, Google, Microsoft and Anthropic will adopt a new standard that publishers can use to opt out of having their content scraped for AI. The story was first covered by Search Engine Roundtable. Today, a site owner has to set training preferences company by company. The new shared "policy token" signal lets a publisher set them once and have them apply across several AI companies.
The full technical details may still be emerging. So far, the companies have described it as a single way to state AI training preferences. Until the documentation is published, website owners should treat it as a promising new option and keep their existing settings in place.
Why training and search are separate choices
In the same week, Search Engine Roundtable reported new Bing AI content controls. Microsoft says they let site owners make choices about training, grounding, or both, while remaining eligible for Bing search results. "Grounding" is when an AI tool reads live web pages to build an answer and often links to them.
Training and grounding are two different jobs. Training is when a company uses your pages to build its AI model. Search is when an AI assistant fetches your pages to answer a user's question, and it can send that user to your site.
That second job matters for traffic. Sayu's study, AI chat apps cite gov.uk most for everyday UK questions, looked at the share of answers to everyday UK searches that linked to at least one website. Perplexity linked out in 100% of answers (about 7.8 sites per answer). ChatGPT did so in 69.4% (about 3.1 sites per answer), Gemini in 53.6% (about 3.5 sites per answer) and Claude in 26.9% (about 4 sites per answer). These figures come from a set of everyday UK questions checked in October 2026. Results will vary with the question asked and will change as the tools are updated.
The same study found Google showed an AI Overview on 69.4% of the 360 everyday UK searches checked in October 2026. That figure reflects one month and one set of searches.
Many AI companies already run separate crawlers for training and for search. A blanket block on everything AI-related can take a site out of the answers that link back to businesses. Blocking training alone can leave it eligible to appear in them.
What this means for UK businesses
- Check your current settings first. Open yoursite.co.uk/robots.txt and read what it says about AI crawlers. Also check any bot protection your hosting or security service has switched on.
- Make two separate decisions. Decide whether you are happy for your content to train AI models. Then decide, separately, whether you want to appear and be linked in AI search answers.
- Watch for the new standard's documentation. Once the companies publish how the shared signal works, add it alongside your existing settings rather than in place of them.
- Look at Bing's controls. If you use Bing Webmaster Tools, review the new options that separate training from grounding.
Analysis compiled by Sayu (sayu.co.uk), drawing on reporting by The Verge and Search Engine Roundtable.
