← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

How Should a News Publisher Handle AI Crawlers: Citations vs Training?

The short answer: by section, not by masthead. Block the training class across the whole site — the archive's value should not be strip-mined into datasets without a deal. For retrieval, split the estate: open the public journalism you want cited and discovered, fence the subscriber archive by path where a cited summary substitutes for the subscription it would have sold. Set the split from your own substitution evidence, review it quarterly, and keep the evidence packs that let you negotiate licensing from counts instead of impressions.

Build my block kit — free · Test a policy in the simulator first

What is a publisher crawler policy?

an AgentBlock policy for a news or content publisher that separates the estate by economics: training blocked site-wide, retrieval allowed on public sections whose citations recruit readers, and paid sections fenced by path rule where answer summaries substitute for subscriptions.

Why publishers get the hardest version

Every other business in this series sells something an answer cannot deliver — a product, a booking, a service. A publisher sells the answer itself, which collapses the comfortable distinction between distribution and substitution. The same retrieval fetch can be tomorrow's subscriber (a cited investigation that demonstrates value) or today's lost sale (a summary that fully satisfies the question the article existed to answer). No fleet-wide default can make that call; it has to be made per section, with the publisher's own funnel data, and remade as the data moves.

Training, by contrast, is the easy line even here. The archive's training value is precisely what licensing deals price; allowing it gratis by default is not openness, it is an unnegotiated licence. Block the class, and let any future deal be a deliberate, evidenced exception.

The section split, concretely

Public explainers, news coverage and service journalism: retrieval allowed. These sections recruit — they are how new readers meet the masthead inside answers, and their citations carry the brand into rooms it cannot otherwise enter. Subscriber analysis, archives and data products: fenced by path rule for retrieval agents, because their summaries substitute. The fence is a path decision, not an agent decision, which is what makes the split administrable: sections move between the two treatments by editing a rule when their economics change, without touching the agent list at all.

Metering subtleties belong to the publisher's platform, not the crawler policy. The kit's job is the boundary it can actually hold: named agents, named paths, logged verdicts.

Evidence is a licensing asset

Publishers negotiate with platforms and AI vendors from a position the evidence pack directly strengthens. 'Your training crawler fetched our archive 400,000 times in a year we blocked it, here are the refusal logs by month' is a licensing conversation with exhibits. 'We feel our content has value' is not. The pack's discipline — per-agent counts, verdicts separated, database version stamped, filed monthly — converts a grievance into a ledger, and ledgers are what deals are priced from.

The same file serves the internal argument. When the retrieval question comes up in the newsroom — are answers sending us readers or eating them? — the pack's retrieval counts beside the analytics referrals give the only honest answer available: this publisher's, this quarter's, in numbers.

The robots layer and the public record

Publishers, more than most, should keep the declaration layer immaculate: the robots file is a public document journalists and researchers actually read, and a publisher's stated posture is part of its industry position. Generate it from the policy, audit it for drift like everyone else, and let the token opt-outs sit visibly in it. Where the industry moves toward collective signals and licensing frameworks, a clean, dated, generated declaration is the artifact that plugs into them without rework.

And expect scrutiny of the gap between word and enforcement: a publisher that declares a block it does not enforce will be found out by its own profession. The verifier's hash and the evidence rhythm are what let the declaration be made in a strong voice.

Five actions to take today

  1. Block training site-wide. The archive's dataset value belongs in negotiated licences, not in a default nobody signed.
  2. Split retrieval by section. Public journalism open for citation; subscriber sections fenced by path where summaries substitute.
  3. Price substitution from your funnel. Compare answer-referred sessions against summary-exposed sections quarterly; move sections between treatments as the data says.
  4. File evidence packs monthly. Per-agent counts are licensing exhibits and the internal debate's only honest input.
  5. Keep the declaration layer pristine. Robots generated from policy and audited — a publisher's public posture will be read by professionals.

Worked example: a regional publisher's two-treatment estate

The Harbour Gazette, the fictional regional news site, runs the split. Its news and explainer sections (roughly 3,800 public articles) allow the five major retrieval agents; its subscriber analysis archive (1,900 pieces under /plus) is fenced for all retrieval agents by path rule; the training class is blocked across the whole domain.

The first two quarters produced the discriminating data the policy was designed to generate: 88,000 training refusals logged, 26,500 retrieval fetches served on public sections, answer-referred sessions up 41%, and — inside the fence — 3,100 retrieval attempts on /plus paths refused, each one a summary that would have been given away.

When a national platform opened licensing talks, the Gazette's evidence file — six monthly packs, refusal counts by agent, the dated policy history — became the meeting's working document. The publisher's commercial lead describes the policy's value in one line: it made our position countable.

Frequently asked questions

Should a publisher block retrieval agents entirely?

Only if its own data shows summaries substituting across the whole estate — a finding some pure-subscription publishers genuinely reach. Most estates are mixed, and the section split captures most of the value on both sides. Decide from your funnel, per section, and revisit quarterly as answer behaviour changes.

Does blocking training affect our search traffic?

No. Training crawlers and search indexing are separate systems; the database's Google and Apple entries are opt-out tokens that explicitly leave search untouched. A publisher's Google News and Search visibility is not in the crawler policy's blast radius.

Can path fencing protect a metered paywall?

It protects the paths you name from the named agents your policy governs — a real and auditable boundary. It is not a paywall and does not replace one: access control for humans and unnamed automation remains your platform's job, as the path-rules guide states plainly.

What do we say to an AI vendor who asks why they are blocked?

The policy, with exhibits: training is blocked pending licence; retrieval is open on public sections and fenced on subscriber paths; here are the monthly evidence packs. Vendors negotiate with publishers who can count — the file is the difference between a posture and a position.

How often should the section split change?

As often as the substitution evidence changes — quarterly review is a sane cadence. Moving a section is a path-rule edit, deliberately cheap, so the policy can track the economics without drama or a rebuild of the agent list.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back