← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

What Is the Complete AI Training Opt-Out Checklist for a Small Business?

The short answer: nine training entries blocked (GPTBot, ClaudeBot, anthropic-ai, Meta-ExternalAgent, CCBot, Bytespider, Google-Extended, Applebot-Extended, cohere-ai), the scraping channel decided (FirecrawlAgent), the decisions enforced at the edge or server — not merely declared in robots — the kit verified by hash, and the whole position written down with its date. That is the complete refusal; the checklist below is the order to do it in, in one sitting, with the two tokens and the second Anthropic name that partial checklists forget.

Build my block kit — free · Test a policy in the simulator first

What is an AI training opt-out?

a site's complete, enforced refusal of training use of its content: all nine training-class entries in AgentBlock's database blocked through their own mechanisms — edge/server refusals for the fetching crawlers, the robots/policy layer for the Google-Extended and Applebot-Extended tokens — verified and dated.

The fetching seven

Seven of the nine training entries are programs that arrive at your site: GPTBot, ClaudeBot, anthropic-ai, Meta-ExternalAgent, CCBot, Bytespider and cohere-ai. Each is verified, each carries its vendor's documentation or directory evidence in the database, and each is refused the same way — a name match at the edge or server, logged per request. The set matters more than any member: a refusal naming six of seven is not a partial credit situation, it is an open door with a policy document leaning against it.

Work the set in the builder rather than from memory. The class view exists because human recall of crawler names is exactly the failure this checklist was written to retire — the second Anthropic token being its most famous victim.

The token two

Google-Extended and Applebot-Extended complete the nine, and they complete it in a different layer: neither fetches, both govern a downstream use of content their vendors' search crawlers collect, and both are expressed in the robots/policy layer your kit generates. Forgetting them is the respectable error — they appear in no one's server log, so no count ever reminds you — and it is the error with the largest vendors attached. Their consolation: both are zero-risk to search visibility, the only two decisions in the entire policy about which that can be said without a caveat.

Verify them by audit, not by log: the enforcement-gap audit confirms the tokens are present in the live robots file, which is the only place their existence can be checked.

The adjacent decisions the checklist includes

FirecrawlAgent is not training, and the checklist includes it anyway, because owners refusing training almost always mean the wider sentence — 'our content should not be machine-ingested for other people's products' — and the scraping API is that sentence's other half. Decide it in the same sitting, in the same note. Retrieval stays untouched by this checklist: the opt-out is about inventory, and the citation channel is distribution. Owners who conflate them mid-checklist produce the citation-loss incident; the classes are kept separate here on purpose.

And the utility agent is protected by construction throughout: nothing in the training refusal touches facebookexternalhit, whose allowlist entry should be confirmed in the same sitting, while the policy is open and the share test is cheap.

Enforcement, verification, and the written position

A checklist completed in robots.txt alone is a draft. Generate the artifact your stack enforces — the free Worker, the plugin — deploy it, and run the verifier: kit ID and hash against the policy, confirming the nine decisions and the fences are what production runs. Then the written position, three sentences in the policy note: the training class is refused as of this date, through these artifacts, verified by this hash; retrieval posture stated separately so no future reader merges them; review date set.

That note is the checklist's final item and its point. An opt-out that exists only as configuration will be re-litigated by every new contractor; an opt-out recorded as a dated organisational position gets maintained instead. The quarterly stale check then keeps it true against a world that adds training crawlers by the season.

Five actions to take today

  1. Block GPTBot — OpenAI's training crawler, the largest single source of training fetches on most sites.
  2. Block both Anthropic training tokens — ClaudeBot and anthropic-ai, together, or the refusal is two-thirds of itself.
  3. Block Meta-ExternalAgent, CCBot, Bytespider and cohere-ai — the rest of the fetching training class, including the open-corpus feed.
  4. Express Google-Extended and Applebot-Extended — the token pair, in the generated robots layer, confirmed afterwards by the gap audit.
  5. Decide FirecrawlAgent, protect facebookexternalhit, enforce, verify, write it down — scraping decided, previews allowlisted, Worker or plugin deployed, hash checked, position dated in the policy note.

Worked example: one sitting, one complete refusal

A fictional 6-person accountancy practice, Hartwell & Co., had 'blocked AI training' twice before — once in robots.txt by a marketing contractor, once via a hosting panel toggle whose scope nobody could later describe. Its AgentBlock sitting began with the audit of what existed: the robots file named four training crawlers, both tokens absent, no enforcement layer at all.

The checklist ran in order in a single afternoon: all nine training entries set, FirecrawlAgent blocked, retrieval left at a deliberate allow for the practice's tax guides, the preview fetcher allowlisted, the free Worker deployed and verified (hash filed), and the position written in three sentences with the review date.

The quarter's evidence pack gave the refusal its first real numbers: 14,600 training-class requests refused at the edge, zero training fetches served, both tokens confirmed present at the June gap audit. The practice's managing partner describes the difference from its previous attempts in one line: 'This time we can point at it.'

Frequently asked questions

How long does the complete opt-out take?

One sitting for a small business: the decisions are nine flips and two adjacent calls, the artifact is generated, deployment is a paste-and-route or a plugin upload, and verification is one check. The owners in these guides consistently spend longer reading their first impact preview than on the opt-out itself.

If I only block GPTBot, how much have I done?

The largest single share on most sites — and an incomplete refusal. Anthropic's two tokens, Meta's trainer, the Common Crawl feed and the token pair remain open, and any document claiming a training opt-out on that basis will fail an informed read. The class is nine decisions because the vendors are nine mechanisms.

Does the opt-out affect AI answers about my business?

Training and answers are separate pipelines. The opt-out governs training; retrieval agents — the answer pipeline — keep whatever posture you set for them, which for most small businesses is allow. Your facts remain citable while your inventory leaves the datasets.

What does it cost?

The complete opt-out runs on the free tier: policy building, the database and the Cloudflare Worker are free. WordPress sites wanting the plugin's in-house log take Pro ($29/mo); the opt-out's completeness does not depend on the tier, only the evidence conveniences do.

How do I keep the opt-out complete over time?

The stale check, quarterly. New training crawlers enter the database as the world ships them; your kit's version diff names them, and the refresh adds their decisions. An opt-out is a subscription to the world's changes, maintained in four sittings a year.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back