# AI Crawler Access Checker

AI companies crawl the web to train models and to answer questions with live citations. This checker reads your robots.txt and shows, crawler by crawler, whether each major AI bot is allowed or blocked - so you can decide who gets access to your content.

## How to use it

1. Paste the contents of your current robots.txt (from yourdomain.com/robots.txt).
2. Review the status of each AI crawler - allowed or blocked.
3. Decide which to allow (for AI answer visibility) or block (to protect training data).
4. Use our robots.txt generator to create the exact directives you want.

## Frequently asked questions

### Should I block or allow AI crawlers?

It's a trade-off. Allowing crawlers like OAI-SearchBot and PerplexityBot can get your content cited in AI answers, driving referral traffic. Blocking bots like GPTBot and CCBot keeps your content out of model training. Many sites allow answer and search bots while blocking training bots.

### What's the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content to train OpenAI's models, while OAI-SearchBot fetches pages to cite in ChatGPT's live search answers. If you want visibility in ChatGPT's answers but not in training data, allow OAI-SearchBot and block GPTBot.

### Does blocking an AI crawler remove me from its answers?

It stops that company from crawling your site directly, but models may still reference content they learned earlier or find through third parties. Blocking is about future access, not erasing existing knowledge.

### Where do I add AI crawler rules?

In your robots.txt at the root of your domain, as separate User-agent groups. Our robots.txt generator can build the exact directives for you.

### Do all AI companies respect robots.txt?

Most major ones - OpenAI, Anthropic, Google, and Perplexity - honor robots.txt, but compliance isn't universally guaranteed. For content you must protect, use server-side access controls, not robots.txt alone.

Use the interactive tool: https://www.tryspook.com/tools/ai-crawler-checker