robots.txt checker
Check any site’s robots.txt for syntax errors, accidental site-wide blocks and a missing Sitemap line — and see exactly which rule decides whether Googlebot and Bingbot can reach the homepage.
The mistakes this catches
- A staging block left in production.
Disallow: /underUser-agent: *removes the whole site from search over time. - An HTML page instead of a file. Some hosts serve the homepage for every unknown path, including /robots.txt. Crawlers treat that as invalid.
- Typos.
Dissallow:or a missing colon is silently ignored, so the rule you think you have doesn't exist. - Rules before any User-agent. They belong to no group and apply to no one.
- No Sitemap line, or a relative sitemap URL.
- Files over 500 KiB, where Google stops reading.
Want to control AI crawlers specifically? Use the AI crawler checker, which reads the same file for GPTBot, ClaudeBot, Google-Extended and others.
Frequently asked questions
What does a robots.txt file do?
It tells crawlers which parts of a site they may fetch. It lives at /robots.txt, groups rules by User-agent, and uses Allow and Disallow lines. It controls crawling, not indexing: a blocked page can still appear in search results if other sites link to it.
What happens if a site has no robots.txt?
Crawlers treat a missing file (404) as "everything is allowed". That is fine for many sites, but you lose the easiest way to point crawlers at your sitemap.
What happens if robots.txt returns a server error?
Google treats a 5xx response as "the whole site is disallowed" until it can read the file again. If it keeps failing for a long time, Google falls back to its last cached copy or crawls without rules.
How does a crawler pick which rules apply?
It uses the group that names it most specifically, falling back to "User-agent: *". Within that group the longest matching path wins, and Allow wins a tie. This checker follows the same rules (RFC 9309).
Does Google respect Crawl-delay?
No. Bing and Yandex honor Crawl-delay; Google ignores it and adjusts its crawl rate automatically.
More free tools
- AI Crawler & ai.txt CheckerSee which AI crawlers (GPTBot, ClaudeBot, Google-Extended, …) your robots.txt lets in, and what your ai.txt declares.
- llms.txt CheckerValidate a site’s llms.txt against the format, find llms-full.txt, and test the links inside.
- Sitemap CheckerFind and validate an XML sitemap: size limits, dates, robots.txt listing, and a sample of broken or redirecting URLs.
- AI Website DetectorWas a site built with an AI website builder or coding tool?