← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read
Per-Agent Policy or Block All AI Bots: Which Is Right?
The short answer: the blanket is a legitimate starting posture — AgentBlock's own default blocks 20 of 21 verified agents and allows only the link-preview fetcher — but it is a starting point. Per-agent policy wins whenever your economics differ by class: training blocked for principle, retrieval allowed for citations, utilities protected by construction. The right answer is the one your own numbers support after the impact preview, and most content and commerce sites land on block-training, allow-retrieval within one review cycle.
Build my block kit — free · Test a policy in the simulator first
What is a per-agent policy?
a crawler policy in which each named agent carries its own decision — block, allow, capped, fenced — rather than inheriting a single blanket verdict. AgentBlock's builder is per-agent by construction, with class defaults as the starting position an owner then adjusts agent by agent.
The honest case for block-all
Block-all is not ignorance; for some sites it is the correct terminal position. A paywalled research business whose summaries substitute for sales, a private community whose content was never meant for models, a data product whose catalogue is the inventory — these owners rationally refuse every class, retrieval included, and accept invisibility in answers as a cost already priced in. If that is your business, the blanket is not a phase; it is the policy, and per-agent tooling serves you by making the blanket precise — verified names, utility protected, fences on private paths.
Block-all is also the right temporary posture while you gather evidence. A new site with no log history cannot price citations it has never received; defaulting strict and loosening from data beats defaulting open and discovering costs later. AgentBlock's default exists for exactly this owner.
The honest case for per-agent
Most sites are not selling answers; they are selling things answers can recommend. For them the classes have opposite signs. Training is a cost with no return channel. Retrieval is a return channel with no inventory cost — the fetched page is read, cited, and left exactly as found. A blanket treats a cost and a channel identically, which is another way of saying it does not treat them at all.
Per-agent policy also ages better. When a vendor launches a new retrieval agent or your citation traffic doubles, a per-agent owner adjusts one flip; a blanket owner re-litigates the whole posture. Granularity is not fussiness — it is the difference between a policy that can be maintained and one that gets abandoned at the first change.
The middle most sites actually choose
The modal mature policy, across the sites these guides draw on: training class blocked in full (nine entries), the two token opt-outs expressed, retrieval allowed agent by agent after a log review (usually all four to six major retrieval entries), ads and scraping blocked, utility allowed and allowlisted, path fences on private subtrees, rate caps set from baselines. It is neither blanket nor open — it is a priced position, each line defensible with a number or a principle.
Reaching it takes one sitting with three inputs: the database (what exists), the impact preview (what visits you), and your analytics (what the visits are worth). Owners who skip the third input guess at the retrieval line; owners who skip the second guess at the fences; the builder cannot supply either, which is why the tools put them on the same screen.
Deciding with the substitution test
One question sorts most retrieval decisions: does a faithful summary of this page, with a citation, replace a visit I needed — or create one? Product pages, local services, how-to content with a business behind it: summaries create visits, because the reader still needs the thing. Pure information products: summaries can replace them. Apply the test per section of your site, not per domain — a publisher may fence its paid research paths while opening its marketing guides, and path rules exist precisely for owners whose answer differs by folder.
Apply the same test honestly in the other direction for training: no summary is involved, no citation flows back, so the only question is whether you consent to inventory use. Owners who consent — open documentation, public research — should allow training deliberately and say why; consent recorded is as much a policy as refusal.
Revisiting without drama
Whatever you choose, schedule the revisit when you choose it. Crawler economics are moving quarterly; a policy written as a permanent constitution will be wrong by silence within a year. The stale check handles the world's changes — new agents, changed grades — and a dated calendar review handles yours: citation volumes, substitution evidence, new site sections with new path needs.
The review's output should be boring: a few flips, a cap adjusted, the note updated, the kit regenerated and verified. Policies that change in small dated steps stay trustworthy; policies that change in annual revolutions usually change because something broke loudly enough to force the meeting.
Five actions to take today
- Start from the default, knowingly. Twenty blocked, one allowed is a defensible opening position — adopt it as a decision, not an accident.
- Run the impact preview before loosening. Per-agent visit counts from your real log turn the retrieval debate from theology into arithmetic.
- Apply the substitution test per section. Summaries that create visits argue allow; summaries that replace sales argue block — possibly by path, not by domain.
- Set the training line by principle. Consent or refusal, stated once, applied to all nine entries and both opt-out tokens.
- Calendar the review. Quarterly, alongside the stale check; small dated adjustments are the maintenance pattern that keeps a policy true.
Worked example: a SaaS site's one-sitting decision
LoopLedger, a fictional bookkeeping SaaS, opened its policy review expecting an argument between its founder (block everything) and its marketer (allow everything). The impact preview settled the shape in twenty minutes: 31,000 training fetches a month against 2,900 retrieval fetches, and analytics showing answer-engine referrals converting to trials at 4.8% — double the site's average.
The policy they shipped: training blocked in full, retrieval allowed for the five verified retrieval entries with caps at five times baseline, scraping and ads blocked, fences on /app, /billing and /reset, facebookexternalhit allowlisted. Neither founder nor marketer got their blanket; both got their numbers.
Two quarters on, trial referrals from AI answers are up 61% and the training refusal count sits at 58,000. The review note records the decision rule for next time: classes are priced separately, always.
Frequently asked questions
Is AgentBlock's default block-all or per-agent?
The default policy is the strict blanket expressed per-agent: 20 of 21 verified agents blocked, facebookexternalhit allowed. The builder is per-agent by construction, so loosening any line — a retrieval agent, say — is one visible flip from that starting point.
Who should stay on block-all permanently?
Businesses whose content is the product being summarised: paywalled research, data products, private communities. If a cited summary replaces your sale, invisibility is the correct price. The substitution test, applied to your own conversion data, is the way to know.
Does allowing retrieval weaken my training block?
No. The entries are independent: training agents stay blocked and logged while retrieval agents fetch. A summary engine citing you does not gain access to the training pipeline through your policy, and your evidence keeps the two streams separate.
How long does a per-agent decision actually take?
One sitting, with the database open, the impact preview run on a recent log slice, and your referral analytics beside it. Most of the hour is reading your own numbers; the flips themselves are minutes.
What if I choose wrong?
Every decision is reversible by regeneration: flip, rebuild, verify the new kit hash, and the policy history shows when each posture held. The expensive errors are the silent structural ones — phantoms, family rules, drift — not a weighed per-agent call that later data revises.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- What Crawler Policy Should a SaaS Documentation Site Run?
- How Should a Creator or Portfolio Site Handle AI Crawlers?
- Incident: What Happens When a Block Rule Breaks Your Link Previews?
- Incident: How Blocking a Retrieval Crawler Quietly Costs You Citations
- Incident: How a Stale Kit Missed Meta's New Crawler Wave
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded