How the crawler check works — and what it never claims.
The check fetches the domain's public robots.txt, resolves the directives per crawler under RFC 9309 conventions (most-specific group, longest-path wins), and reports one state per identity with the rule behind it.
Evidence states
Confirmed configuration: a relevant published directive is visible in the fetched robots.txt.
Observed response: the system genuinely observed an HTTP response (e.g. 403 observed, rate limiting observed) — the layer is named only with rule-level evidence.
Inconclusive / Not evaluated: the available evidence does not establish a result. No pass or failure is inferred.
What is never claimed
Crawler visits, indexing, ingestion, training use, ranking, citation, traffic, or revenue. The check is point-in-time evidence for the submitted page path, with bounded rechecks — not continuous monitoring. For the full Visibility methodology (SEO 18, GEO 11, AEO Beta 17, 9 crawlers separate), see the visibility methodology.
Ownership
These guides and the checker are operated by Optimus Pass at optimuspass.com as acquisition content for the Optimus Pass visibility product. They are a separate reading experience, not a concealment of ownership: every page links directly to the product that performs the deeper audit.
Last updated: September 10, 2026