AI crawler & ai.txt checker
Enter any site to see which AI crawlers its robots.txt lets in — split into training, AI search and user-triggered bots — and what its ai.txt file declares.
What this checks
AI companies crawl the web with named bots, and the place they say they look for your permission is robots.txt. This tool reads your robots.txt the way those crawlers do (following RFC 9309: the most specific group wins, the longest matching rule wins) and reports, for each AI crawler, whether it may fetch your homepage and which line decided it. It also fetches /ai.txt and shows which content types it opts out of AI training.
The AI crawlers we check
GPTBot— OpenAI, trainingOAI-SearchBot— OpenAI, ai search (Results in ChatGPT search)ChatGPT-User— OpenAI, user-triggered (Fetches a page when a ChatGPT user asks for it)ClaudeBot— Anthropic, trainingClaude-SearchBot— Anthropic, ai searchClaude-User— Anthropic, user-triggeredGoogle-Extended— Google, training (Gemini training & grounding only — no effect on Google Search)Applebot-Extended— Apple, training (Apple Intelligence training — no effect on Siri/Spotlight)Meta-ExternalAgent— Meta, trainingPerplexityBot— Perplexity, ai searchPerplexity-User— Perplexity, user-triggeredCCBot— Common Crawl, training (Open dataset used to train many AI models)Bytespider— ByteDance, trainingAmazonbot— Amazon, ai search (Alexa and Amazon AI answers)
Blocking AI training without leaving AI search
The most common setup: keep your content out of training data, but stay visible in AI search answers.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot Disallow: / User-agent: * Allow: /
For the full background — what ai.txt is, where it came from, and why robots.txt still does the real work — read What is ai.txt, and does it actually stop AI crawlers?
Frequently asked questions
What is ai.txt?
ai.txt is a text file proposed by Spawning in 2023 that declares whether your content may be used to train AI. It borrows robots.txt syntax but applies rules to file types, e.g. "Disallow: *.jpg" to opt images out. It is a declaration, not a block.
Do AI companies respect ai.txt?
None of the major AI companies document reading ai.txt. OpenAI, Anthropic, Google, Apple and Perplexity all tell site owners to use robots.txt, which is why this checker reads both files.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search crawls with Googlebot, which is a separate rule.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) keeps your content out of future models. Blocking AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) also removes your pages from AI answers, which can cost traffic.
Why do user-triggered fetchers show as allowed even though I block them?
They are listed the way robots.txt treats them. But OpenAI and Perplexity document that ChatGPT-User and Perplexity-User act on behalf of a person, so robots.txt rules may not apply to them in practice.
More free tools
- llms.txt CheckerValidate a site’s llms.txt against the format, find llms-full.txt, and test the links inside.
- robots.txt CheckerFind syntax errors, accidental “block everything” rules, and whether search engines can reach your site.
- Sitemap CheckerFind and validate an XML sitemap: size limits, dates, robots.txt listing, and a sample of broken or redirecting URLs.
- AI Website DetectorWas a site built with an AI website builder or coding tool?