AI Crawler Access
What does AI Crawler Access mean?
AI crawler access describes whether a website's robots.txt allows the automated bots that AI companies use to fetch content — for example GPTBot, OAI-SearchBot, and ChatGPT-User from OpenAI, or Google-Extended from Google. Blocking a crawler removes your content from that company's training or search-grounding pipeline, depending on the bot.
Why does it matter?
If AI crawlers cannot fetch your pages, those engines have nothing to read from you, so you are far less likely to appear in the answers they generate. Access is a prerequisite for being cited, not a guarantee of it.
What does it look like in practice?
A site blocks all bot traffic with a wildcard robots.txt rule and disappears from ChatGPT web-search answers. Allowing OAI-SearchBot and GPTBot restores access; OpenAI notes robots.txt changes can take about a day to take effect.
Frequently asked questions
Does blocking Google-Extended hurt my Google rankings?+
Which bots should I allow for AI visibility?+
Which sources support this definition?
Related terms
Try it with Optimus Pass
Run a Visibility Audit on your site and see how it scores on the checks behind this term.
Visibility Audit →Browse the full glossary for more definitions.
All glossary terms →