← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read
What AI Crawler Policy Should an Ecommerce Store Run?
The short answer: a store's policy writes itself once the assets are named. Block the training class and FirecrawlAgent — your catalogue, photography and pricing are inventory, not dataset. Allow the major retrieval agents — AI answers increasingly answer 'where can I buy', and product citations are the new shelf space. Fence /checkout, /account and /admin for every agent. Keep facebookexternalhit allowed and allowlisted — shares are a store's word of mouth. Set rate caps from your log, because stores are where crawler load and trading peaks collide.
Build my block kit — free · Test a policy in the simulator first
What is an ecommerce crawler policy?
an AgentBlock policy shaped for a store: training and scraping classes blocked to protect catalogue inventory, retrieval agents allowed for product citations, private commerce paths fenced for all agents, the link-preview fetcher protected, and rate caps sized to trading-peak load.
The catalogue is the asset under negotiation
A store's public pages are not marketing about the product; in machine terms they are the product's data — specifications, photography, stock levels, prices — in a form any fetcher can inventory wholesale. Training crawlers take that inventory into datasets. A scraping API takes it to whoever rents the scraper, including, in the cases that concentrate minds, the competitor whose prices begin shadowing yours within the week. Both classes meet the store's refusal for the same reason a wholesaler locks a warehouse that happens to have a glass front.
The refusal is cheap to make precise because the classes are clean: nine training entries, one scraping entry, every one verified and nameable. A store does not need a stance on AI in general; it needs ten decisions that take ten minutes in the builder.
Retrieval is shelf space now
The countervailing fact: shopping questions are moving into answer engines, and the answers cite fetchable product and guide pages. A store invisible to retrieval agents has vacated that shelf entirely — not demoted, absent — while competitors' products are quoted with prices. The retrieval decision for a store is therefore commercial, not technical, and should be made with the referral analytics open: most stores that measure it keep the major retrieval agents allowed and fence them out of transactional space instead.
Freshness argues the same way. User-triggered fetchers read today's page — today's stock, today's price. A store that allows them is quoted accurately; a store that blocks them is quoted from stale copies or not at all, and stale price quotes generate the support tickets nobody wants.
Fences are a store's non-negotiables
Checkout, payment, account, order-status and admin paths are refused to all agents, full stop. No citation is earned on a confirmation page; no legitimate agent business exists at a password reset. Path rules make this a property of the policy rather than a hope about agent behaviour, and the refusals log with their rules — evidence a payment provider's security review will ask for by name, as the booking-site example in the path-rules guide shows.
Add preview and staging paths during sale preparations. A store's unpublished sale prices, fenced in advance, cannot be fetched into an answer before the sale opens — the kind of leak that is invisible until a customer quotes your Friday price on Wednesday.
Load, peaks and the preview fetcher
Stores live on peaks — a promotion, a seasonal spike — and peaks are when crawler load stops being funny. Rate caps sized from an ordinary month's baseline, with the per-minute cap as the flood gate, keep agent traffic from competing with buyers during the hours that pay for the year. Revisit the caps before your known peak season with the impact preview run on last year's peak week, not on a quiet Tuesday.
And protect the previews. A store's products are shared in chats constantly; facebookexternalhit is quite literally the colleague who builds the card that sells the product. Allowed by default, allowlisted by discipline, share-tested after every deploy — the three habits from the preview guide, applied without exception in a store's policy.
Five actions to take today
- Block training and scraping in full. Nine training entries plus FirecrawlAgent: the catalogue is inventory, and inventory is not a dataset.
- Allow major retrieval agents. Product citations in AI answers are shelf space; fence the agents out of transactional paths instead of out of the store.
- Fence commerce paths for all agents. Checkout, payment, account, order-status, admin and preview paths — refused, logged, evidence-ready.
- Protect previews like revenue. facebookexternalhit allowed, allowlisted, and share-tested on real product pages after every deploy.
- Size caps to your peak, not your average. Run the impact preview on last peak season's log before the next one arrives.
Worked example: a WooCommerce store's complete policy
Beaumont Beauty, the fictional skincare store, runs the template nearly as written. Its policy blocks the nine training entries and FirecrawlAgent; allows five retrieval agents including PerplexityBot and ChatGPT-User; fences /checkout, /my-account, /wp-admin and /preview-sale for all agents; caps every allowed agent at four times its measured baseline; and carries facebookexternalhit on the allowlist.
A quarter under the policy produced the shape the template predicts: 96,000 training and scraping refusals in the plugin's block log, 12,400 retrieval fetches served, AI-answer referrals up 34% quarter on quarter, and zero agent verdicts of any kind on the fenced commerce paths — the line its payment provider's reviewer underlined.
The one adjustment came from the log, as adjustments should: Bytespider-name attempts arrived in bursts during a competitor's sale week, and the per-minute cap was halved for the training class's remnants without touching anything else.
Frequently asked questions
Will blocking training crawlers hurt my store's Google sales?
No. The training class has no role in Google Search rankings — Google-Extended, the one Google entry, is a training opt-out token that explicitly does not affect Search. Your organic visibility is governed by Googlebot, which is not in AgentBlock's database and never appears in a kit.
Should I block price-comparison scrapers by name?
The named, honest channel for that traffic is FirecrawlAgent — block it, and fence or cap the catalogue paths behaviourally for the unnamed remainder. A policy that names the scraping class and fences the asset is stronger than a list of guessed scraper names.
Do AI answers really send stores buyers?
On the evidence of the stores in these guides, yes — retrieval referrals arrive with ordinary conversion rates and grow as answer engines take shopping questions. Your own analytics is the authority: measure the channel for a quarter before considering any retrieval block.
What is the first thing to check after a sale goes wrong?
Previews, then fences. Share-test the sale page (the utility fetcher), then read path-rule verdicts to confirm nothing transactional leaked or broke. Both checks take minutes and cover the two failure classes a promotion actually suffers.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- What Can Robots.txt Actually Do About AI Crawlers?
- How Do You Design an Allowlist for AI Agents?
- What Are the Most Common AI Blocklist Design Mistakes?
- Per-Agent Policy or Block All AI Bots: Which Is Right?
- How Do You Test a Block Policy Before Deploying It?
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded