How to use the robots.txt tester
- Either enter a domain and press “Fetch robots.txt” to load the live file, or edit the rules in the box. A sample file is there to start with.
- Type the path or full URL you want to check, such as
/blog/orhttps://example.com/cart/. - Pick the crawler. Choose “Custom” for any other bot and type its name, such as AhrefsBot.
- The verdict updates as you type: “Allowed” or “Blocked”, the rule that decided it, its line number and the group that applied.
How crawlers read robots.txt
The file is split into groups. A group starts with one or more User-agent lines and holds the Allow and Disallow rules below them. A crawler obeys only one group: the one naming it most specifically. Googlebot-Image follows a googlebot-image group if the file has one, otherwise the googlebot group, and only then *. Groups that name the same crawler are merged.
Inside that group, every rule whose path matches the start of the URL path is a candidate, and the longest one wins. When an Allow and a Disallow rule are the same length, Allow wins. * matches any run of characters and a trailing $ marks the end of the URL. Paths are case-sensitive, an empty Disallow: blocks nothing, and /robots.txt itself is always readable. These are the rules of RFC 9309, which Google and Bing follow.
Worked example
With the sample file loaded, test /search?q=shoes as Googlebot. The file has no Googlebot group, so the * group applies, and Disallow: /search is the longest match: Blocked. Now test /search/help. Both Disallow: /search (7 characters) and Allow: /search/help (12 characters) match, so the longer Allow rule wins. The tool shows Allowed and the deciding rule Allow: /search/help (line 4). Switch the crawler to GPTBot and every path is blocked by its own Disallow: / group.
Common mistakes the tester catches
- Using robots.txt to hide pages. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a
noindexrobots meta tag. - Blocking CSS and JavaScript. Google renders pages like a browser, so blocked style and script files can make pages look broken to it.
- Missing trailing slash.
Disallow: /adminalso blocks/administrators-guide;/admin/blocks only the folder. - Wrong case.
Disallow: /Photos/does not block/photos/. - Unsupported lines. Google ignores
Crawl-delayandNoindex:. The tester lists such lines in its notes, along with misspelled directives.
When the live file returns an error, the status decides what crawlers do: a 404 means no restrictions, while a 5xx error makes Google pause crawling the whole site. Google also reads only the first 500 KiB of the file.