How the crawler check works — and what it never claims.
By Grady Coleman (Founder) · Last updated:
The check fetches the domain's public robots.txt, resolves the directives per crawler under RFC 9309 conventions (most-specific group, longest-path wins), and reports one state per identity with the rule behind it.
Key takeaways
- — Every result is one of three evidence states: confirmed, observed, or inconclusive.
- — The check never claims visits, indexing, ranking, citations, or traffic.
- — Results are point-in-time with bounded rechecks — never continuous monitoring.
- — Ownership is disclosed: these guides feed the Optimus Pass audit product.
What evidence states does the check report?
Confirmed configuration: a relevant published directive is visible in the fetched robots.txt.
Observed response: the system genuinely observed an HTTP response (e.g. 403 observed, rate limiting observed) — the layer is named only with rule-level evidence.
Inconclusive / Not evaluated: the available evidence does not establish a result. No pass or failure is inferred.
What is never claimed
Crawler visits, indexing, ingestion, training use, ranking, citation, traffic, or revenue. Compared with rank trackers, which record positions over weeks, this check reports one path at one moment. The check is point-in-time evidence for the submitted page path, with bounded rechecks — not continuous monitoring. For the full Visibility methodology (SEO 18, GEO 11, AEO Beta 17, 9 crawlers separate), see the visibility methodology.
When should you re-check a domain?
The check is point-in-time evidence, so re-check whenever the underlying facts can change: after a deploy, a template edit, a new security plugin, a CDN settings change, or a provider documentation update from OpenAI, Anthropic, Google, or Perplexity. According to the method, each result belongs to one submitted path at one moment — a fresh check on the same path is a new observation, not a continuation. Agencies commonly re-check per client onboarding for the same reason. Bounded rechecks after fixes are part of the workflow; continuous monitoring is a different product the checker does not offer.
Who operates the checker and its guides?
These guides and the checker are operated by Optimus Pass at optimuspass.com as educational content for the Optimus Pass visibility product. They are a separate reading experience, not a concealment of ownership: every page links directly to the product that performs the deeper audit. The relationship is stated here so no reader mistakes an educational guide for an unaffiliated review.
By Grady Coleman (Founder) · Last updated:
Method questions, answered
Does Allowed mean my page will be crawled?
No, Allowed means the published directives permit a request — nothing more. The crawler operator decides when and whether to visit, and firewall or CDN layers can refuse requests that robots.txt permits. The checker reports configuration evidence for the submitted path, so treat every result as permission state, not a visit log.
Why report Not evaluated instead of guessing?
The alternative to Not evaluated is an assumption, and assumptions about access cause exactly the misconfigurations this check exists to catch. When robots.txt cannot be read, the honest result is unknown — the page states what was attempted and stops there, rather than filling the gap with a plausible-sounding verdict.
How do training and search controls stay separate?
The check resolves each of the 9 tokens against its own applicable group, so a GPTBot training decision never leaks into an OAI-SearchBot search verdict. According to the providers' crawler documentation, these are independent controls with independent directives, and the report keeps them in separate purpose groups for that reason.