Tools
Robots.txt checker: which AI bots do you let in?
Type a domain. I fetch the robots.txt and tell you, bot by bot, who may crawl the site: search engines, AI training crawlers and real-time fetch agents. With the rule that decides each verdict and what it means for your visibility.
Note: I read the file as it is right now, using Google's interpretation rules. I keep neither the domain nor the file, beyond a 10-minute cache so I do not hammer other people's sites.
The bots this tool checks
The same matrix as the article on AI crawlers and bots, plus Claude-SearchBot and the two opt-out switches.
| Bot | Operator | Render JS | Purpose | robots.txt |
|---|---|---|---|---|
| Googlebot | Yes | Search index (feeds AI Overviews and Gemini) | Respects | |
| Bingbot | Microsoft | Yes (limited) | Search index (feeds Copilot and ChatGPT) | Respects |
| Applebot | Apple | Yes | Search (Siri, Spotlight) and training (Apple Intelligence) | Respects |
| GPTBot | OpenAI | No | Model training | Respects |
| OAI-SearchBot | OpenAI | No | ChatGPT search index | Respects |
| ChatGPT-User | OpenAI | No | Real-time fetch (on request) | User action |
| ClaudeBot | Anthropic | No | Model training | Respects |
| Claude-User | Anthropic | No | Real-time fetch (on request) | User action |
| PerplexityBot | Perplexity | No | Perplexity index (not training) | Respects |
| Perplexity-User | Perplexity | No | Real-time fetch (on request) | User action |
| Claude-SearchBot | Anthropic | No | Claude search index | Respects |
| Meta-ExternalAgent | Meta | No | Meta AI training | Respects |
| Amazonbot | Amazon | No | Alexa and AI services | Respects |
| Bytespider | ByteDance | No | Training | Historically ignores |
| CCBot | Common Crawl | No | Open dataset that feeds many LLMs | Respects |
| Google-Extended | - | Gemini training opt-out | Token | |
| Applebot-Extended | Apple | - | Apple training opt-out | Token |
To decide with context: AI crawlers and bots: who crawls, renders and indexes and Block or allow AI? The GSC control and robots.txt, without the myths.