What Is ai.txt? And Does It Actually Stop AI Crawlers?

If you want to keep your website out of AI training data, you will quickly run into ai.txt: a small text file that promises to tell AI companies what they may and may not train on. It sounds like exactly the right tool. The catch is that none of the major AI companies say they read it. Here is what ai.txt actually is, why it doesn't do what most people hope, and the rules that do control AI crawlers today.

What ai.txt is

ai.txt was proposed by Spawning, a company that builds AI opt-out tools, in May 2023. It is a plain-text file at the root of your site (yoursite.com/ai.txt) that borrows the syntax of robots.txt but applies it to types of content instead of paths. A rule like Disallow: *.jpg means “don't train on my images”:

# ai.txt — opt images and text out of AI training
User-Agent: *
Disallow: *.jpg
Disallow: *.jpeg
Disallow: *.png
Disallow: *.txt
Disallow: *.html

Spawning offers a generator that writes the file for you, grouped into text, images, audio, video and code. Its pitch was a machine-readable way to state the text-and-data-mining opt-out that EU copyright law allows rights holders to make.

The catch: it's a declaration, not a block

ai.txt doesn't stop anything on its own. A crawler has to choose to read it and honor it, and as of October 2026 none of the big AI companies documents doing so. OpenAI, Anthropic, Google, Apple and Perplexity all publish crawler documentation, and all of them point site owners to the same place: robots.txt.

It doesn't help that the name is crowded. Several unrelated proposals use “ai.txt”, including a 2026 individual draft at the IETF that describes itself as a superset of Spawning's format. None of them is a finished standard. An ai.txt file is still a reasonable public statement of your wishes, but it is not what keeps a crawler out.

What actually controls AI crawlers: robots.txt

Every major AI company crawls with named bots and documents a robots.txt token for each. The important detail is that they do different jobs. Blocking a training crawler keeps your content out of future models. Blocking an AI search crawler also removes your pages from that assistant's answers and citations, which can cost you traffic.

The most common goal is to stay out of training data but remain visible in AI search. That looks like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Three mistakes we see constantly

1. Blocking AI with “User-agent: *”. A site-wide Disallow: / under * does block AI crawlers, and it also blocks Googlebot and Bingbot. To block only AI, name each crawler in its own group, as above.
2. Blocking AI search when you only meant training. Putting OAI-SearchBot, Claude-SearchBot or PerplexityBot in the block list takes you out of those answers entirely.
3. Fearing Google-Extended will hurt rankings. It won't. Google-Extended only controls Gemini training and grounding. Google Search uses Googlebot, which is a separate rule.

What big sites actually do

We ran our AI crawler checker against a few well-known sites on October 31, 2026:

• nytimes.com blocks 13 of the 14 AI crawlers we check, including every training crawler and three of the four AI search crawlers.
• theguardian.com blocks 9 of 14, again including most AI search crawlers.
• github.com blocks just 1 of 14 and leaves AI search fully open.

None of the three has an ai.txt file. Large publishers that care about this put their rules in robots.txt.

What robots.txt can't do either

robots.txt is also voluntary: it works because the major companies have chosen to follow it, not because anything enforces it. Two more limits are worth knowing. First, user-triggered fetchers such as ChatGPT-User and Perplexity-User act on behalf of a person who asked about your page, and OpenAI and Perplexity document that robots.txt rules may not apply to them. Second, blocking a crawler today doesn't remove anything that was collected before you added the rule.

So should you add an ai.txt?

It costs nothing and states your intent in a machine-readable way, which some people value for the EU opt-out argument. Just don't let it be the only thing you do. If you want AI crawlers to stay out, the rules have to be in robots.txt.

Want to see where your own site stands? Check any site's AI crawler access and ai.txt. It shows, for each crawler, whether it is allowed and the exact robots.txt line that decided it. While you're at it, you can also check whether the site was built with AI.