robots.txt says Allowed. The bot still never arrives.
That means the second gate is closed: your platform template or firewall is stopping the crawler upstream. Here are the exact places to look on Shopify, WordPress, Vercel, Netlify, and Nginx.
Check your domain firstShopify: the template and the app overwrite
Shopify generates robots.txt for you and allows AI crawlers by default. When a Shopify store blocks bots, the cause is usually a robots.txt.liquid edit or a third-party app that rewrote the file. Re-read the live file, fix the Liquid template, and confirm the rendered output. Keep one rule per bot, exact token match, and never paste a full catalog into agents.md — it has a size limit and a different job (agent instructions, not crawler context).
WordPress: no plugin required
Read yourdomain.com/robots.txt directly and edit the file (or virtual rule via your SEO plugin) by hand. Then check the second gate: security plugins and host firewalls with bot-management features can silently deny AI user-agents. If the bot token never appears in your access logs despite an Allowed rule, the block lives there — not in robots.txt.
Vercel, Netlify, Nginx: rule order decides
WAF custom rules execute around your app: a deny for an AI user-agent fires before robots.txt is ever consulted. On Cloudflare, review AI Crawl Control alongside WAF custom rules and move the intended rule to the top. On Nginx, check deny directives and limit zones. Allowed in robots but absent from logs always means upstream.
Limits
Point-in-time configuration evidence for the submitted page path only. Fixing access makes you eligible to be read — it never promises indexing, ranking, citation, traffic, or revenue.
Frequently asked questions
Does Shopify block AI crawlers by default?
No. Shopify's default robots.txt allows AI crawlers. Blocks almost always come from a robots.txt.liquid customization or a third-party app that edited the file. Check your live /robots.txt first, then the Liquid template and installed apps.
How do I edit robots.txt on Shopify for AI crawlers?
In Shopify admin go to Online Store, Themes, Edit code, add a robots.txt.liquid template, and append your per-bot rules at the bottom without removing the default checkout, cart, and search blocks. Save and verify the live /robots.txt in a browser — changes are live in seconds.
Does llms.txt control crawler access?
No. llms.txt suggests what matters; it blocks nothing. Anyone relying on llms.txt to keep content out of AI systems has blocked nothing. Conversely, blocking the wrong bots in robots.txt makes even the best llms.txt useless. Keep the two jobs separated.
Why does my checker say Allowed but the crawler never arrives?
Because a firewall, CDN, or WAF rule is stopping it upstream. On Cloudflare check Security, Configure AI bot policies (note the September 2026 defaults blocking Training and Agent bots on ad pages for new domains). On Vercel/Netlify check WAF custom rules and their order. On Nginx check deny rules and rate-limit zones, plus access.log for the bot token.
Last updated: September 10, 2026