Can AI crawlers see your site?
Check whether ChatGPT, Claude, Perplexity, Google’s AI and 12 other crawlers can reach your page — by your robots.txt rules and by what your server actually does.
About the AI crawler checker
Training crawlers (like GPTBot and ClaudeBot) collect pages to train AI models. Search and user bots (like OAI-SearchBot, Claude-SearchBot and PerplexityBot) fetch pages to answer questions and cite sources. Blocking training bots is a business choice; blocking search bots means AI assistants can’t recommend or link to you.
Your robots.txt might allow a bot while your firewall or CDN (Cloudflare, for example) quietly blocks it. We request the page as GPTBot, ClaudeBot and PerplexityBot and compare the answer with what a normal visitor gets.
A proposed plain-text file at the root of your site that gives AI tools a curated map of your most useful pages. It’s optional and not every AI company reads it yet, but it’s cheap to add.
It depends on your business. Publishers often block training bots but allow search bots. Most businesses that want to be found should let the search and user bots in — they’re how AI assistants discover and cite you.
More free tools
SEO page checker
Titles, meta descriptions, headings, alt text, structured data and speed for any page, scored in seconds.
Use the tool NewHTTP status checker
Status codes and the full redirect chain for up to 10 URLs, with redirect problems flagged.
Use the toolSitemap checker
Find your XML sitemaps, check they load and count the URLs in them.
SERP snippet preview
See how your title and description fit in Google, measured in pixels.
Schema generator
Build valid JSON-LD for your organisation, FAQs, products and articles.