# Robots.txt Tester

A single wrong line in robots.txt can hide your whole site from Google. Paste your robots.txt, enter a URL and a crawler, and see instantly whether it's allowed or blocked - and exactly which rule decided it.

## How to use it

1. Paste your robots.txt.
2. Enter the URL or path to test.
3. Set the user-agent, or leave it as *.
4. Read the allowed/blocked verdict and the matching rule.

## Frequently asked questions

### How does robots.txt decide access?

For a given crawler, the most specific matching user-agent group applies, and within it the longest matching path rule wins. On a tie, Allow beats Disallow.

### Does a specific user-agent override the wildcard?

Yes. If a crawler has its own User-agent group, it uses only that group and ignores the User-agent: * rules entirely.

### Does robots.txt remove pages from Google?

No. It blocks crawling, but a blocked URL can still appear in results if it's linked elsewhere. To keep a page out of the index, allow crawling and use a noindex tag.

### What does an empty Disallow mean?

Disallow: with no path means 'allow everything'. It's how you grant a crawler group full access.

### Why is my page blocked when I didn't intend it?

Usually a broad Disallow (like Disallow: /) or a path rule that unintentionally matches. This tester shows the exact rule responsible so you can fix it.

Use the interactive tool: https://www.tryspook.com/tools/robots-txt-tester