Skip to content

SEOCrawler access on Shopify

How to Check AI Crawler Access on a Shopify Store

AI systems can discover Shopify products through more than one route. Open-web crawlers may fetch storefront pages, while Shopify Catalog supplies structured product data to eligible, activated agentic storefronts. This guide explains how to check open-web access on Shopify and where that check stops.

Key takeaways

  1. Shopify's robots.txt governs open-web crawler requests; Shopify Catalog is a separate product-data route for activated agentic storefronts
  2. Shopify documents Liquid edits for crawler-specific rules and recommends keeping its maintained defaults
  3. Six checks in order: file groups, page directives, served HTML, token meanings, user-initiated fetchers, change timing
  4. An open file is a request, not proof. Shopify calls its rules advisory, and some fetchers ignore them by design
  5. A robots.txt check cannot establish Shopify Catalog eligibility or inspect a separate CDN or firewall

01 Two different routes to product discovery

A robots.txt check answers a narrow question: what access instructions does the store publish for open-web crawlers? It does not tell you whether a product is supplied to an agentic storefront through Shopify Catalog. Shopify documents Catalog as a product-data source for eligible products and activated channels, separate from traditional web crawling and indexing.

Shopify says Catalog eligibility, product-data mapping, and channel activation determine whether products are available through that route. The documentation describes controls for removing Catalog access for specific channels, including ChatGPT and Microsoft Copilot. A product may still be discoverable through other external methods. Treat the Catalog route and open-web crawling as separate checks, and do not infer one from the other.

This article covers robots.txt, page directives, and the HTML served to open-web crawlers. For Catalog availability, review Shopify's Agentic sales-channel settings, eligibility, and product-data mapping. Shopify's current agentic storefront guidance explains both discovery routes and their limits.

Open-web access is still relevant when people or crawlers use search engines and web retrieval. For background on those separate questions, see our guides to how ChatGPT finds sources and what an AI Overview can use.

02 Where robots.txt lives on Shopify, and who controls it

Every public store serves the file at yourdomain.com/robots.txt, and Shopify generates a default file. To customize it, Shopify documents a theme template named robots.txt.liquid. You can add or remove directives in Liquid, which lets Shopify keep updating the default rules. Shopify recommends this approach over replacing the template with plain-text rules, which can become outdated.

Read Shopify's own warnings before you touch it. The Help Center calls it an unsupported customization. Shopify Support can't help with questions about how to use this feature. And, more bluntly: Incorrect use of the feature can result in loss of all traffic. Incorrect rules can affect search crawling as well as AI crawler access.

Shopify labels this an unsupported customization and says Support cannot help with edits. The Help Center documents adding or removing rules for specific crawlers. Its caution says incorrect use can result in loss of all traffic, so follow the current instructions and preserve Shopify's maintained rules where possible.

03 What the default file blocks, and what it doesn't

Shopify's documented default robots.txt includes these disallowed paths: /admin, /cart/, /checkout, /account, /orders, plus collection filtering and sorting parameters. The default also allows crawlers to request public product, collection, page, blog, and policy content.

Shopify's documented default includes an Allow: / rule and no rule naming GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended. A custom theme template can change the file a store serves, so inspect the live file instead of assuming it matches the default.

Shopify notes that an existing robots.txt.liquid template may still output older rules such as Disallow: /search and Disallow: /policies/, which are absent from its current default. If you find those directives, review the custom template and Shopify's current guidance before changing it.

The documented default allows public paths, but the live file and Shopify's network layer are separate checks. The next section covers what those checks establish.

04 The six-step open-web access check

Work through these in order. Each one can fail independently of the others, and a clean result at step one is not a clean result at step three.

1. Read the file per crawler. Fetch yourdomain.com/robots.txt and look for a User-agent: group naming each token you care about: OAI-SearchBot for ChatGPT search, GPTBot for training, PerplexityBot, Claude-SearchBot, and Googlebot. RFC 9309 says crawler product-token matching is case-insensitive. When multiple groups match a crawler, combine their rules; use the wildcard group only when no matching group exists. Google documents an additional rule: it selects the most-specific matching user-agent group, combining multiple groups for that specific token. Within a group, apply the most-specific matching path rule. Google uses the least restrictive result when matching Allow and Disallow rules conflict. The robots.txt guide walks the rule syntax, and the free AI crawler access checker evaluates all 9 identities against your file in one pass.

2. Check the page itself. robots.txt gates paths; a page directive like noindex gates URLs. The two interact, and not in the direction people assume. Google documents that a noindex on a page its crawler can't fetch is never seen, because reading it requires the fetch, and where rules conflict the more restrictive result applies. So a closed robots.txt can hide a page even with no directive on it, and an open one only helps if the page itself doesn't say stay away. Check a live product page's <head> for the directive before concluding the file is the whole story. One boundary: Google's documentation says the noindex rule applies to search engine crawlers, and the AI-crawler vendors say nothing about it, so nothing settles how an AI-specific fetcher treats the directive.

3. Look at the served HTML. View Source on a product page. Is the description text in the markup, or assembled after load? Google documents that Googlebot renders pages for search; none of the AI-crawler vendors documents anything comparable. If descriptions or pricing appear only after JavaScript runs, treat their visibility to those systems as unproven. View Source is a signal about the raw fetch, not a verdict on any specific bot.

4. Know which token controls what. Training and search are separate switches. OpenAI's crawler documentation states each setting is independent: GPTBot relates to model training, OAI-SearchBot to surfacing sites in ChatGPT search results, and a site opted out of search will not be shown in ChatGPT search answers, though can still appear as navigational links. The GPTBot vs OAI-SearchBot reference maps the identities and what blocking each one does, and doesn't do.

5. Separate a person asking from a crawler running. ChatGPT-User and Perplexity-User fetch when a human prompts them. OpenAI notes that for those requests robots.txt rules may not apply, and Perplexity documents that Perplexity-User generally ignores robots.txt rules. Pasting your URL into ChatGPT and getting your page read back is a real spot-check of user-initiated fetch. It proves nothing about whether the automated crawler reaches the store.

6. Give changes a day. Shopify serves a template change instantly; crawlers don't re-read on your schedule. OpenAI: it can take ~24 hours from a site's robots.txt update for our systems to adjust. Google caches robots.txt for up to 24 hours. Re-check after a crawl cycle, not after a coffee.

RFC 9309, the 2022 standard that formalised robots exclusion, states the limit plainly: the rules are not a form of access authorization. Shopify's own phrasing agrees: its file is "directional and advisory, and not all crawlers are guaranteed to follow them."

Free diagnostic · Point-in-time check

Check open-web access in one pass

The free scan reads your robots.txt against all 9 crawler identities plus page-level directives, alongside 18 SEO checks, GEO findings, and AEO Readiness (Beta). It does not evaluate Shopify Catalog eligibility or channel activation, and does not see CDN or firewall rules.

Run a free diagnostic

05 What an open robots.txt does not prove

An allow in the file is a necessary condition, not a sufficient one. Three separate things can sit between "the file says open" and "an AI answer quotes the page."

First, the file can fail to arrive cleanly. If robots.txt returns a 4xx error, RFC 9309 says a crawler MAY access any resources on the server, and Google documents that it treats every 4xx except 429 as if no file existed at all, with no restrictions. If it returns 5xx, the RFC says a crawler MUST assume complete disallow, and Google stops crawling for up to 12 hours and then leans on a cached copy. A broken robots.txt has no single meaning across engines, and a check that can't fetch the file cleanly can't return a clean verdict.

Second, something can intercept before the file is ever consulted. Shopify says bot management for Shopify stores is handled at its network layer and discourages a proxy in front of Shopify. Its documentation does not name which bots are affected or whether any particular crawler is blocked. A separate CDN or WAF configured by the merchant may also affect requests. This is an open-web access question, distinct from Shopify Catalog, which supplies eligible product data to activated agentic storefronts. Changing robots.txt or network-layer access does not stop Catalog distribution to channels the merchant has activated. Shopify documents separate channel controls, including ChatGPT and Microsoft Copilot options. Cloudflare's one-click AI block has existed since July 2024, and its current docs describe its own access controls for sites using Cloudflare. Shopify advises against putting a proxy in front of its platform, so do not assume a Shopify store uses Cloudflare.

Third, a policy statement is not a performed visit. Vendors share work where they can: OpenAI notes that if a site allows both of its bots, it may use the results from one crawl for both use cases. That is a statement about OpenAI's documented process, not proof that a particular site was visited.

So can you see the crawlers that do arrive? Shopify's documentation distinguishes Catalog availability and channel controls from open-web crawler activity. Do not treat a Catalog listing as proof of a crawler fetch, or an open robots.txt as proof that Catalog data is enabled. What does show some open-web crawler activity: Google Search Console for Googlebot only, Cloudflare bot analytics if your plan includes it, and the published IP ranges OpenAI and Google maintain, if you have request logs somewhere to check them against. No AI crawler in your logs, where you even have logs, is not proof no crawler came.

06 What to check next

Match the fix to the failed check. A default file with no AI rules and clean product pages does not call for a robots.txt change based on those checks alone. Move to the page-directive and served-HTML checks. If the question is Shopify Catalog availability, review eligibility, channel activation, and product-data mapping separately. Don't edit the liquid template to feel productive; it's an unsupported file carrying a loss-of-all-traffic warning.

For a leftover robots.txt.liquid, review the crawler rules against Shopify's current default. Keep administrative, cart, checkout, account, and duplicate-filter paths handled as intended, then test the live result after saving. Shopify Support cannot review the custom file, so make changes carefully.

For a separate CDN or WAF, review that provider's settings. Editing robots.txt does not change a request intercepted before the file is read.

And if the check raises questions you can't settle yourself, about which pages matter most or what the findings mean for the catalogue, the done-for-you audit is a human-reviewed assessment of the specific pages you agree on, findings prioritized. This article is the self-serve version; that one reads the store with you.

Questions

Frequently Asked Questions

Does Shopify block AI crawlers by default?

Shopify's generated robots.txt and its network-layer bot management describe open-web crawler access. Shopify separately distributes eligible product data to activated agentic storefronts through Shopify Catalog. These are distinct routes, and Shopify's docs do not support inferring a network-layer block or a guaranteed crawler visit from the robots.txt file.

Where do I find robots.txt on my Shopify store?

At your domain root: yourdomain.com/robots.txt. Shopify generates the file. To change it you add a robots.txt.liquid theme template in the code editor, which Shopify labels an unsupported customization and warns can cause loss of all traffic if used incorrectly.

Does robots.txt control whether products appear in Shopify agentic storefronts?

No. Shopify says robots.txt and network-layer controls affect open-web discoverability. Product data sent through Shopify Catalog to activated agentic storefronts is a separate mechanism. Shopify documents separate channel controls, including opt-out options for ChatGPT and Microsoft Copilot; changing robots.txt does not replace those controls.

My product descriptions load in the browser. Will a crawler see them?

Unproven, and that distinction matters. Google documents that Googlebot renders pages for search; the AI-crawler vendors do not document equivalent rendering. Use View Source to see what a raw fetch receives: if the description is absent from the HTML, treat its visibility to those systems as unproven. This describes served markup, not any specific bot's behavior.

What is the fastest way to confirm AI crawlers can reach one product page?

For open-web access, check the store's robots.txt for the crawler's matching user-agent group and path rules, then inspect the product page's directives and served HTML. That does not establish whether the product is available through Shopify Catalog. Check Shopify's Agentic sales-channel settings and Catalog eligibility separately. A prompt pasted into ChatGPT or Perplexity is a user-initiated fetch spot check, not proof of automated crawler access.

Next step

Check open-web access against 9 crawler identities

The free scan reads robots.txt and page-level directives on the URL you enter. It does not inspect Shopify Catalog eligibility or channel activation. Both are evidence checks, not guarantees of discovery or citation.

Run a free diagnostic

← Back to the blog