Free robots.txt Tester
Paste a robots.txt file, list the URLs you care about and see, for each one, whether a chosen crawler may fetch it and which rule decided.
Free, with no sign-up. Fair-use limits apply. Text you paste is processed and not stored. To enforce the limits we keep any domain or address you enter and a scrambled form of your IP address for 30 days, as described in our privacy policy.
What a robots.txt tester tells you
A robots.txt file tells crawlers which parts of a site they may request. It is easy to get wrong: one stray slash can block a whole section, and rules for different crawlers interact in ways that are hard to judge by eye. This tester applies the same matching logic that major crawlers use, so you can check a change before it goes live.
How to use it
Paste the contents of your robots.txt, list one URL or path per line, choose the crawler to test as, and press Test URLs. Each URL is marked allowed or blocked and the exact rule that decided it is shown. You can also see syntax problems, such as a rule that appears before any user-agent line.
How rules are matched
- The most specific crawler group wins: a group for Googlebot is used instead of the group for *, not in addition to it.
- Inside a group, the longest matching path pattern wins, and when an Allow and a Disallow match equally, Allow wins.
- An asterisk matches any run of characters and a dollar sign anchors the end of the address.
- An empty Disallow allows everything.
Blocking versus removing
Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of search results, let it be crawled and use a noindex tag instead. Our meta robots generator writes one for you.
Frequently asked questions
Does this fetch my live robots.txt?
No. You paste the text, so you can test a draft before publishing it. Nothing you paste is stored.
Can I test AI crawlers?
Yes. The crawler menu includes GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other common AI user agents.
Why is a URL blocked when I did not expect it?
Check the rule shown next to the result. The usual cause is a broad pattern such as Disallow: /*? or a rule that is longer than the Allow you expected to win.
More free tools
See all free tools, or read about the Domain Tools in the dashboard.