Robots.txt Validator & Tester
Check a robots.txt for syntax problems and see which user-agent groups apply, before a stray disallow quietly removes a section from search.
Browser tool
robots.txt
Paste robots.txt or upload a file. Everything is checked on your device; no website is contacted.
About Robots.txt Validator
One mistyped line in robots.txt can remove a whole section of a site from search, and nothing announces it — traffic simply declines. Checking the file against the rules before it ships is considerably cheaper than diagnosing the drop afterwards. Free, and no login is wanted. The rules are parsed inside this page, which means a private staging file stays private.
How to use Robots.txt Validator
- Paste the fileThe whole thing, including comments and blank lines, exactly as it will be served.
- Read which groups applyDirectives belong to the user-agent group above them, and a stray blank line ends a group early. Seeing the grouping is usually enough to spot the mistake.
- Fix and re-checkBefore deploying, rather than after the reporting shows a decline.
The rule that surprises people most
Disallow controls crawling, not indexing. A URL that is blocked can still appear in results, listed without a description, because Google saw it linked from elsewhere and was never allowed to fetch it to find out more. Worse: if you block a page in order to hide it, the crawler can no longer see the noindex on it, so blocking actively prevents removal. To keep something out of the index, allow crawling and use noindex.
Most specific match wins
When several rules match a URL, the longest matching path takes precedence rather than the first one written — so an Allow can override a broader Disallow regardless of order. This is why a file that reads correctly top to bottom can still behave unexpectedly.
What it does not do
- Syntax validity says nothing about whether the rules express what you meant. A perfectly valid file can block your most important section.
- Crawlers differ. Wildcards and Allow are widely supported but not universal, and Crawl-delay is ignored by Google entirely.
Robots.txt Validator: questions people ask
- Will blocking a page in robots.txt remove it from Google?
- No, and it can prevent removal. A blocked URL can still be listed, and because the crawler cannot fetch it, it never sees any noindex you added. Allow crawling and use noindex when the goal is to get something out of the index.
- Do I need a robots.txt at all?
- Not strictly — its absence means everything may be crawled, which is often the correct policy for a small site. It is worth having to point at your sitemap and to keep crawlers out of areas with no public value.
- Can I test whether Googlebot is allowed to crawl a specific URL?
- Yes, enter a URL or path under Test a URL and pick Googlebot, Bingbot, GPTBot, ClaudeBot or a custom user agent to see Allowed or Blocked, with the group that applied. Matching follows RFC 9309 and Google's longest-match rule. It tests the rules you pasted, not the file your site currently serves.
Questions about ToolsVerse itself — privacy, cost, what happens to your files — are answered in the frequently asked questions.
Related tools
The tools people most often need alongside Robots.txt Validator.
Can’t find the tool you need?
Tell us what you’re looking for and we’ll consider adding it.