Meta's new crawling terms: decide what AI crawlers take from your site
AI crawling guidance
Meta's updated terms mean many UK businesses have quietly agreed to AI training they never considered. Knowing what your robots.txt file says — and what it does not say — is a decision you can make today.
What Meta has changed
Search Engine Journal reports that Meta's new terms let it crawl websites tied to its business users and use that content to train its products. According to the report, this widens rights that ad buyers had already granted.
This piece describes the terms only as reported. It does not set out their legal reach. Any business that wants to know exactly what it has agreed to should read Meta's terms in full or take its own advice.
The practical point is simple. Many UK firms run ads or keep a business page on Meta. Most did not sign up thinking about AI training. The new terms are a good prompt to check what you have agreed to, and what your website currently allows.
The wider debate about AI and web content is active too. Press Gazette reports that the Guardian and the Sun agree on a ban for deceptive AI bots. CEOs and editors are calling for the Prime Minister to back an AI transparency bill. The rules may change, but your own settings are something you can review today.
Training crawlers and search crawlers are not the same
AI crawlers do not all do the same job. Treating them as one group can cost you visibility.
- Training crawlers collect pages to help build AI models. Examples include OpenAI's GPTBot, Anthropic's ClaudeBot and Meta's meta-externalagent. Google-Extended is a separate control that lets you opt out of your content being used for Google's AI models without affecting Google Search.
- AI search crawlers and fetchers collect pages so an assistant can show or cite them in answers. Examples include OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot for Claude, and PerplexityBot for Perplexity.
If you block a training crawler, your content is less likely to be used to train that model. If you block a search crawler, you may simply vanish from answers in tools like ChatGPT, Gemini or Perplexity. For a business that wants to be found, that is a very different result.
Your robots.txt file is where most of these choices are set. It is worth knowing what it says, who wrote it and when it was last checked. Bear in mind that robots.txt is a request, not a lock. Well-run crawlers follow it, but it does not override any contract terms you have accepted elsewhere.
What this means for UK businesses
- Read the terms. If you advertise on Meta or link your website to a Meta business account, read the updated terms so you know what you have agreed to.
- Check your robots.txt. Look at which AI crawlers it allows and blocks. Make sure the choices are deliberate.
- Separate training from search. Decide whether you mind your content being used to train AI models. Then decide, separately, whether you want to appear in AI answers. Most businesses that rely on being found will want the second.
- Review regularly. New crawlers appear and terms change. Put a short check in the diary.
Test any change before it goes live, and keep a note of why each line is there. That way the next person to edit the file knows the decision was made on purpose.
Analysis compiled by Sayu (sayu.co.uk)
