Skip to content

Free crawler check · no account needed

AI crawler access checker: check robots.txt rules for a page path.

Enter a domain or page URL. The checker evaluates the robots.txt rules that apply to that path for nine search and AI crawler identities, then shows the matching rule and whether it is Allowed, Blocked, or Not evaluated. Your first check is free and needs no account.

By · Last updated:

Check a public page · free snapshotSEARCH · TRAINING · ADS CRAWLERS
CRAWLER DIRECTORY1 / 9 · SEARCH

Key takeaways

  • Nine crawler identities are checked against the rules for the submitted page path.
  • Allowed, Blocked, and Not evaluated show what the matching rule evidence supports.
  • Training and search controls are separate. You can allow one and block another.
  • No readable rule means Not evaluated. The checker does not assume access.
Reading

How to read each state

Allowed

The robots.txt directives that apply to this crawler permit the requested page path. The engine found real evidence for this conclusion.

Blocked

A Disallow rule in the crawler's applicable group covers the requested path. Whether that is intentional is a decision only you can make.

Not evaluated

No robots.txt evidence could be read, so access is reported as unknown — never as allowed. Absence of a file is not proof of anything.

Boundaries

What this checker does not claim

The result is a reading of directives, not a measurement of outcomes.

Live crawler visits. An Allowed rule does not show that a crawler requested the page.Firewall or CDN reachability. Those controls are not tested.Successful indexing or search rankings.AI answer appearances or citations.
Questions

Frequently asked questions

How do I check if AI crawlers can read my website?+

Enter a domain or page URL. The checker evaluates the robots.txt rules that apply to that page path for each of 9 crawler identities, then shows the matching rule and an Allowed, Blocked, or Not evaluated state. Firewall or CDN reachability is not tested.

What does an AI crawler access checker actually check?+

The checker fetches the domain's robots.txt file and evaluates the rules for the submitted page path, separately for each supported crawler identity. It follows RFC 9309 matching conventions: the most-specific user-agent group applies, the longest matching path rule wins, and Allow overrides Disallow at equal length.

Which crawlers does the checker evaluate?+

The checker evaluates 9 identities: Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, GPTBot, Google-Extended, ClaudeBot, and OAI-AdsBot. The result labels each identity by its intended purpose; robots.txt rules do not show whether it made a request.

If a crawler shows Allowed, will ChatGPT or Perplexity cite my site?+

No. Allowed means the matching robots.txt rule permits that crawler to request the submitted path. The result does not show a crawler visit, indexing, rankings, or citations.

Does blocking GPTBot hurt my visibility in ChatGPT search?+

OpenAI documents separate controls for OAI-SearchBot and GPTBot. OAI-SearchBot is used for ChatGPT search; GPTBot is used for potential model-training collection. The checker reports their robots.txt rules separately. It does not report search ranking or citation outcomes.

Why does everything say Not evaluated for my site?+

The checker could not read usable robots.txt rules for the submitted origin. The file may be missing or unavailable. It reports Not evaluated rather than assuming access.

Is this checker free?+

Yes. Your first check is free with no account and no credit card — run it on any public domain or page URL. After that, create a free account to keep checking. The full visibility audit — which adds the 3-lens SEO, AEO, and GEO diagnostics alongside this crawler view — also has a free snapshot.

What can you check after reviewing crawler access?

The full visibility audit checks one public page for SEO, GEO, and AEO Readiness (Beta), then reviews crawler-access rules separately. It reports page findings and suggested fixes, not rankings, visits, or citations.

Related crawler guidance

Check crawler-specific rules against provider documentation before changing your robots.txt.

Sources: official crawler documentation

By · Last updated: