Free crawler check · no account needed
AI crawler access checker: check robots.txt rules for a page path.
Enter a domain or page URL. The checker evaluates the robots.txt rules that apply to that path for nine search and AI crawler identities, then shows the matching rule and whether it is Allowed, Blocked, or Not evaluated. Your first check is free and needs no account.
By Grady Coleman (Founder) · Last updated:
SearchTrainingAds
Key takeaways
- Nine crawler identities are checked against the rules for the submitted page path.
- Allowed, Blocked, and Not evaluated show what the matching rule evidence supports.
- Training and search controls are separate. You can allow one and block another.
- No readable rule means Not evaluated. The checker does not assume access.
How to read each state
The robots.txt directives that apply to this crawler permit the requested page path. The engine found real evidence for this conclusion.
A Disallow rule in the crawler's applicable group covers the requested path. Whether that is intentional is a decision only you can make.
No robots.txt evidence could be read, so access is reported as unknown — never as allowed. Absence of a file is not proof of anything.
What this checker does not claim
The result is a reading of directives, not a measurement of outcomes.
Frequently asked questions
How do I check if AI crawlers can read my website?+
Enter a domain or page URL. The checker evaluates the robots.txt rules that apply to that page path for each of 9 crawler identities, then shows the matching rule and an Allowed, Blocked, or Not evaluated state. Firewall or CDN reachability is not tested.
What does an AI crawler access checker actually check?+
The checker fetches the domain's robots.txt file and evaluates the rules for the submitted page path, separately for each supported crawler identity. It follows RFC 9309 matching conventions: the most-specific user-agent group applies, the longest matching path rule wins, and Allow overrides Disallow at equal length.
Which crawlers does the checker evaluate?+
The checker evaluates 9 identities: Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, GPTBot, Google-Extended, ClaudeBot, and OAI-AdsBot. The result labels each identity by its intended purpose; robots.txt rules do not show whether it made a request.
If a crawler shows Allowed, will ChatGPT or Perplexity cite my site?+
No. Allowed means the matching robots.txt rule permits that crawler to request the submitted path. The result does not show a crawler visit, indexing, rankings, or citations.
Does blocking GPTBot hurt my visibility in ChatGPT search?+
OpenAI documents separate controls for OAI-SearchBot and GPTBot. OAI-SearchBot is used for ChatGPT search; GPTBot is used for potential model-training collection. The checker reports their robots.txt rules separately. It does not report search ranking or citation outcomes.
Why does everything say Not evaluated for my site?+
The checker could not read usable robots.txt rules for the submitted origin. The file may be missing or unavailable. It reports Not evaluated rather than assuming access.
Is this checker free?+
Yes. Your first check is free with no account and no credit card — run it on any public domain or page URL. After that, create a free account to keep checking. The full visibility audit — which adds the 3-lens SEO, AEO, and GEO diagnostics alongside this crawler view — also has a free snapshot.
What can you check after reviewing crawler access?
The full visibility audit checks one public page for SEO, GEO, and AEO Readiness (Beta), then reviews crawler-access rules separately. It reports page findings and suggested fixes, not rankings, visits, or citations.
Related crawler guidance
Check crawler-specific rules against provider documentation before changing your robots.txt.
Sources: official crawler documentation
- OpenAI — Overview of OpenAI Crawlers
- Anthropic — Does Anthropic crawl data from the web?
- Perplexity — Perplexity Crawlers
- Google — Common crawlers (Googlebot, Google-Extended)
- RFC 9309 — Robots Exclusion Protocol
By Grady Coleman (Founder) · Last updated: