AI search & crawlers
Publisher trade bodies are challenging Meta's scraping terms — and every UK business with a website already faces the same underlying question: which AI bots should be allowed to read your pages?
Publishers take a collective stand
Several publisher trade bodies have published letters to Meta pushing back on its new scraping terms, Digiday reports. They have also urged lawmakers to act. Publishers rarely move together like this, which makes this a live trade story.
The debate covers how AI companies collect and reuse content from the web. Publishers and Meta will work through the details of the terms in their own talks and through lawmakers. This article offers no view on those terms and gives no legal advice. If your business has questions about any company's terms, speak to a legal adviser.
Every UK business with a website already faces a practical version of the same question: which AI bots should be allowed to read your pages?
AI bots do different jobs
Most AI companies run more than one crawler, and each does a different job. Broadly, there are three kinds:
- Training crawlers collect pages to help build and improve AI models.
- Search crawlers index pages so an AI assistant can find them and cite them in its answers.
- User-triggered fetchers visit a page when a person asks an assistant to look at it.
Each crawler identifies itself with its own name, called a user-agent. You can allow or block each one separately in your robots.txt file, the plain text file at the root of your website that tells bots where they may go.
That gives you a set of separate choices. If you block training crawlers, your pages are less likely to go into future AI models. If you block search crawlers, AI assistants may struggle to find your pages and cite them when people ask questions your business could answer. Blocking everything protects your content but may cost you visibility in AI answers. Allowing everything keeps you visible but hands over your content for every purpose.
Robots.txt works as a request. Well-behaved crawlers follow it, but it does not physically stop a bot. It is still the standard way to tell AI companies your preferences.
Sayu's study of the most-visited .uk websites looks at which AI crawlers those sites block in robots.txt. It also looks at how often AI answers appear in everyday UK searches.
What this means for UK businesses
- Check your robots.txt today. Visit yourdomain.co.uk/robots.txt and see which bots it names. Many sites carry rules added years ago by a developer or a plug-in.
- Decide crawler by crawler. Sort the AI bots into training, search and user-triggered fetchers, then decide on each group. Most companies list their crawler names in their own documentation.
- Weigh visibility against control. If people find you by asking questions, blocking AI search crawlers may cost you more than it saves.
- Keep the terms debate separate. Watch how the publisher talks with Meta develop, but take legal advice on any terms that affect your business.
- Review the file regularly. AI companies add and rename crawlers, so set a reminder to check your file again.
Analysis compiled by Sayu (sayu.co.uk)
