Which robots.txt line blocks your AI crawlers?
Three rules decide everything: the most specific user-agent group applies, the longest matching path rule wins, and Allow beats Disallow at equal length. Here is how to read your file like the checker does — and fix only what needs fixing.
Check your domain now1. Find the right group for each bot
A group starting with User-agent: GPTBot applies only to GPTBot. A group with User-agent: * applies to any bot with no specific group. The token must match exactly — GPT-Bot, ChatGPT, or OpenAI match nothing. Our checker evaluates all 9 canonical identities separately so a training-bot block is never confused with a search-bot block.
2. Decide training vs search first
This is the decision most sites get wrong — they block everything when they only meant to opt out of training:
Allow for AI search visibility: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot.
Decide deliberately (training): GPTBot, ClaudeBot, Google-Extended.
Never block blindly: live-fetch agents that act for a real user at request time.
3. Read 403 and 429 honestly
A 403 observed at fetch time was refused by the server or firewall — it does not name Cloudflare, Vercel, or any layer by itself. A 429 means rate limiting was observed, not a permanent block. And a homepage result never establishes the whole domain: re-check the exact page path the client cares about.
Limits
Point-in-time configuration evidence for the submitted page path only. Not proof of a crawler visit, firewall behavior, indexing, training, ranking, citation, traffic, or revenue. No tool can promise an AI answer will cite you.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used for model training. OAI-SearchBot crawls specifically for ChatGPT search results. They are independent robots.txt user-agents: you can block training crawls with User-agent: GPTBot / Disallow: / while allowing search crawls with User-agent: OAI-SearchBot and an empty Disallow. Blocking one does not block the other.
How do I allow an AI crawler in robots.txt?
Add a group for its exact user-agent token with an empty Disallow line, e.g. User-agent: PerplexityBot followed by Disallow: (empty). An empty Disallow means allowed — that is not a typo. Verify the live file at yourdomain.com/robots.txt afterwards, because CMS apps and deploys can overwrite it.
What does a 403 mean for an AI crawler?
A 403 observed at fetch time means access was refused by the server or firewall layer — it does not name the layer. robots.txt itself never returns 403; it is a text file. Say '403 observed', not 'Cloudflare blocked the crawler', unless you have rule-level evidence from that layer.
What does a 429 mean?
Rate limiting was observed — the crawler asked too fast and was slowed down. It is not permanent blocking. Recheck off-peak before concluding anything, and do not treat one 429 as proof the bot can never reach you.
Does blocking AI training bots remove me from AI answers?
Only the bots you block stop being asked. Training bots (GPTBot, ClaudeBot) and search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) are separate controls. If you block only training, search-index bots you allowed can still read you. Check each token individually rather than blocking everything at once.
Last updated: September 10, 2026