Robots.txt checker

Five things for the crawler you pick:

• Which robots.txt group applies to it, and the exact line that allows or blocks the URL
• What your server returns when the page is requested with that crawler’s user agent
• noindex, nofollow and nosnippet rules aimed at it in meta tags or the X-Robots-Tag header
• Mistakes in robots.txt: typos, rules outside a group, full URLs in rules and unsupported directives
• Every redirect hop, since each new URL is checked against its own host’s robots.txt

Google uses the group for its most specific name (a Googlebot group before User-agent: *) and combines every group with that name. Within it, the longest matching rule wins, and Allow wins a tie. Wildcards (*) and end anchors ($) work as Google documents. If robots.txt returns a 4xx error, Google treats the site as having no rules; a 5xx error stops it crawling the site until the file loads again.

Yes. Press “Edit and test” under the robots.txt, change the rules and test them against the URL. Switch crawler to see how the same rules treat other bots. Nothing on your site changes.

Firewalls, CDNs and security plugins often refuse AI crawlers by user agent, whatever robots.txt says. Some also check that a request claiming to be Googlebot or Bingbot really comes from Google or Microsoft, and refuse our test while letting the real crawler in. Search Console’s crawl stats show what the real Googlebot gets.

Not reliably. robots.txt controls crawling. A blocked URL can still be indexed from links on other sites, without its content. To keep a page out of results, allow crawling and add a noindex tag.

Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot) fetch pages to answer questions and cite sources, so blocking them keeps you out of AI answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended and CCBot) collect content for training models; blocking them is a business decision and doesn’t affect AI search. Our AI crawler checker tests them all at once.