Robots.txt Tester
A single wrong line in robots.txt can hide your whole site from Google. Paste your robots.txt, enter a URL and a crawler, and see instantly whether it's allowed or blocked - and exactly which rule decided it.
Blockedfor * on /admin/settings
Matched rule: Disallow: /admin
How to use the robots.txt tester tool
- 1Paste your robots.txt.
- 2Enter the URL or path to test.
- 3Set the user-agent, or leave it as *.
- 4Read the allowed/blocked verdict and the matching rule.
Frequently asked questions
How does robots.txt decide access?
For a given crawler, the most specific matching user-agent group applies, and within it the longest matching path rule wins. On a tie, Allow beats Disallow.
Does a specific user-agent override the wildcard?
Yes. If a crawler has its own User-agent group, it uses only that group and ignores the User-agent: * rules entirely.
Does robots.txt remove pages from Google?
No. It blocks crawling, but a blocked URL can still appear in results if it's linked elsewhere. To keep a page out of the index, allow crawling and use a noindex tag.
What does an empty Disallow mean?
Disallow: with no path means 'allow everything'. It's how you grant a crawler group full access.
Why is my page blocked when I didn't intend it?
Usually a broad Disallow (like Disallow: /) or a path rule that unintentionally matches. This tester shows the exact rule responsible so you can fix it.
More free SEO tools
Skip the manual work - let Spook rank you
These tools help with the pieces. Spook does the whole job: it finds winnable queries, writes the articles, and publishes them on autopilot.