Guide

robots.txt for e-commerce: what to allow and what to block

A single wrong line in robots.txt can hide your shop from Google or from AI assistants like ChatGPT. Here is how to get it right.

Scan your site free

Allow your public pages

Product, category, blog and landing pages should always be crawlable. Start with User-agent: * and Allow: /.

Block pages with no search value

Cart, checkout, account and internal search result pages waste crawl budget.

Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?q=

Decide on AI crawlers

GPTBot, ClaudeBot, PerplexityBot and Google-Extended feed AI assistants. Allowing them helps your shop appear in AI answers. Blocking them keeps your content out.

Add your sitemap and llms.txt

Add a Sitemap: line pointing to your sitemap.xml, and publish an llms.txt that briefly describes your shop for AI tools.

Frequently asked questions

Does robots.txt remove pages from Google?

No. It stops crawling, not indexing. Use a noindex tag to keep a page out of results.

Can Trendifyn check my robots.txt?

Yes — the Site Audit flags problems and gives you an improved version to copy.

See what Trendifyn finds on your site

Free scan · one opportunity unlocked · no card needed

Scan your site free