Skip to content

01 / Crawler access

Can AI crawlers reach the pages that matter?

By · Last updated:

Check the live robots.txt rules for nine crawler identities. See the applicable rule, the crawler's purpose, and the boundary of what that evidence can prove.

Free verified account. No card. Point-in-time configuration evidence.

OPTIMUS / ACCESS SIGNALREADY TO CHECK

CRAWLER IDENTITIES

Which crawler policies apply to one page?

Search, training, and advertising crawlers are reported separately so an Allowed result never masquerades as a citation promise.

Googlebot SEARCHOAI-SearchBot SEARCHPerplexityBot SEARCHGPTBot TRAININGClaudeBot TRAININGOAI-AdsBot ADS

Key takeaways

  • — Nine crawler identities are checked separately: 5 search, 3 training, 1 ads.
  • — Every answer cites the exact robots.txt rule behind it.
  • — Allowed means permission, never a visit, citation, or ranking.
  • — Blocking training bots does not block search crawlers.

02 / What the check asks

Which crawler question matters to your page?

Nine identities are grouped by purpose, so search access, training preferences, and ad validation stay separate.

01 / SEARCH

Can search crawlers reach the page?

The checker reviews Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Bingbot against the requested path. It shows the rule behind each state, so search and AI-search access are not confused with training permissions.

How is search access checked?

The checker resolves the most specific crawler group for each search token, then applies the longest matching path rule for the submitted URL. According to the robots exclusion conventions, the most specific group wins over wildcard groups, and the longest path wins within a group. The result is Allowed, Blocked, or Not evaluated — always with the exact rule shown beside it.

02 / TRAINING

Which training controls are separate?

GPTBot, Google-Extended, and ClaudeBot are reported separately from search crawlers. Blocking a training token does not automatically block a search crawler, so the checker keeps those decisions visible instead of treating every bot as one switch.

How are training tokens checked?

Each training token is evaluated against the published robots.txt groups independently of the search crawlers. The checker labels the training purpose beside every result, so a GPTBot or ClaudeBot decision never hides inside a search verdict. An Allowed state shows permission only — never a training visit, dataset inclusion, or future model use.

03 / ADS

Can an ad landing crawler reach the page?

OAI-AdsBot is reviewed as a separate advertising identity for ChatGPT Ads landing-page validation. It is not an organic visibility signal, and its result should not be read as proof of ad approval or delivery.

How is ad access checked?

The OAI-AdsBot token is applied to the requested path exactly like any other crawler identity. The checker reports the matching robots evidence for ad-landing validation only. Firewall and CDN behavior may still affect a request after robots.txt is read, so an Allowed result never counts as proof of ad approval or delivery.

04 / EVIDENCE

What does an Allowed result prove?

Allowed means the published robots.txt directives permit that crawler to request the page path. It does not prove a visit, indexing, ingestion, ranking, citation, or recommendation.

How is evidence kept honest?

No evidence means no confirmation: when robots.txt cannot be read, the checker reports Not evaluated instead of assuming access. The result is point-in-time configuration evidence for the submitted path — the exact directives, the crawler they apply to, and nothing beyond what the file shows.

03 / The boundary

Does Allowed mean you will be cited?

The checker reads published directives. Compared with a search block, which can remove retrieval evidence, a training opt-out only affects model data eligibility. No rule file can reveal what an AI system will select.

  • — AI rankings or placement in answers. robots.txt says nothing about ranking.
  • — Citation eligibility in ChatGPT, Perplexity, or any assistant.
  • — Successful indexing. A directive is not an index report.
  • — Guaranteed crawling. robots.txt is a request; visits depend on each operator.
  • — Firewall behavior. A WAF or CDN rule can block a crawler that robots.txt allows.

The rest of this guide

Go deeper on the part you need

04 / The next move

What comes after the access check?

The full Visibility audit adds SEO, GEO, and AEO Readiness — Beta for the same URL, then turns the evidence into a prioritized Fix First list.

01Check crawler accessconfiguration
02Run the visibility auditpage conditions
03Fix and verifyre-scan