robots.txt Validator
Paste a robots.txt or fetch one from a URL. See the parsed rules, get the syntax checked, and test whether any path is blocked for a given crawler.
What robots.txt does and does not do
robots.txt tells well-behaved crawlers which paths to avoid requesting. It is a crawling instruction, not an access control and not an indexing instruction. A blocked URL can still appear in search results if other pages link to it — Google just will not have read the page. To keep something out of the index, use a noindex meta tag on a page that is crawlable, or protect it properly with authentication.
How path matching works
Matching is prefix-based and case-sensitive. Two wildcards are supported: * matches any run of characters and $ anchors the end of the URL.
| Pattern | Matches | Does not match |
|---|---|---|
| /fish | /fish, /fish.html, /fish/salmon.html, /fishheads | /Fish.asp, /catfish |
| /fish/ | /fish/, /fish/salmon.htm, /fish/?id=1 | /fish, /fish.html |
| /*.php | /index.php, /folder/any.php | / , /windows.PHP |
| /*.php$ | /filename.php, /folder/filename.php | /filename.php?p=1, /filename.php5 |
Which rule wins
When several rules match, the most specific one wins — that is, the one with the longest path pattern. If an Allow and a Disallow are the same length, Allow wins. This is why Allow: /folder/public overrides Disallow: /folder/ for a URL underneath it.
User-agent: * Disallow: /folder/ Allow: /folder/public
Given Disallow: /folder/ and Allow: /folder/public, the path /folder/public/page is allowed because the Allow pattern is longer.
How groups work
Consecutive User-agent lines share one group of rules. A crawler uses the single group whose user-agent token matches it most specifically, and ignores every other group — including the * group. That is the most common robots.txt mistake: adding a Googlebot group and assuming Googlebot still obeys the rules under *.
User-agent: * Disallow: /private User-agent: Googlebot Disallow: /admin
Common mistakes
Disallow: / left over from staging
This blocks the entire site. It is the single most common cause of a site vanishing from search after a launch.
Expecting a named crawler to also follow the * rules
It will not. Repeat any shared rules inside the named group.
Using Disallow to remove a page from search
Blocked pages can still be indexed from external links. Allow crawling and serve a noindex tag instead.
Blocking /css/ or /js/
Google renders pages before ranking them. Blocking assets can make a page look broken to the crawler.
A relative Sitemap path
Sitemap must be an absolute URL, including https://.