← All guides · Reviewed November 2026
How to Block AI Scrapers and Bots from Your Website
The short answer: you block AI scrapers by naming them — a policy that matches each known agent's user-agent and refuses it at your edge or server. AgentBlock builds that policy from a dated database of 25 agent entries (21 vendor-verified) and generates the deployable kit for your stack: a Cloudflare Worker, a WordPress plugin, or a plain-JS snippet. Build the policy free at the kit builder, deploy one artifact, then verify it. Database v2026-10-18, reviewed November 2026.
What are you actually blocking?
Not "bots" in general — specific, named agents whose vendors publish a user-agent string. AgentBlock's database sorts its 25 entries into classes, and the default decision follows the class:
| Class | Entries in the database | Default | Examples |
|---|---|---|---|
| Training | 9 | Block | GPTBot, ClaudeBot, CCBot, Bytespider |
| Retrieval (search + user-triggered fetch) | 12 | Block | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot |
| Indexing | 1 | Block | meta-webindexer |
| Ads | 1 | Block | meta-externalads |
| Scraping | 1 | Block | FirecrawlAgent |
| Utility | 1 | Allow | facebookexternalhit (link previews) |
Three of the retrieval entries are watch entries — ChatGPT Atlas, Meta's Muse agent, and Grok retrieval — with no vendor-published user-agent. There is no name to bind a rule to, so no kit blocks them; they classify UNKNOWN. Which classes deserve a pass instead is the subject of which AI crawlers you should allow.
How does the blocking work, step by step?
- Pick per agent. In the builder, each verified agent starts at its class default — the default policy blocks 20 and allows 1 — and you flip any of them.
- Generate the kit. AgentBlock emits the artifact for your stack, stamped with a kit ID, a kit hash, and the database version it was built from.
- Deploy one artifact. Edge, server, or client — the three deployments are step-mapped in blocking AI bots on WordPress, Cloudflare, or plain JS.
- Simulate, then verify. The pre-deploy simulator shows what a policy does before it ships; the kit verifier checks the deployed artifact against your policy afterwards.
- Re-check for staleness. Agent databases age. The stale check diffs your kit against the current database so a kit built on last season's agent list gets flagged, not forgotten.
What can a user-agent block not do?
AgentBlock prints the limit on every kit, and it belongs in this guide too: sophisticated spoofing can bypass any user-agent block — an agent sending a stock browser user-agent, or rotating residential IPs, walks past a name-based rule. A block is a control, not a guarantee. It stops the named, honest agents — which, for training crawlers and answer engines, is most of the traffic a small site is trying to govern. Agents with no published name stay UNKNOWN, never guessed. The wider fleet takes the opposite posture on its own public pages — see Ingenious AI — because blocking is a per-site choice, and this kit exists to make that choice deployable in one click.
Frequently asked questions
How do I block AI scrapers from my website?
Decide per agent, not all-or-nothing: AgentBlock's builder turns your per-agent choices into a deployable kit — a Cloudflare Worker, a WordPress plugin, or a plain-JS snippet — generated from its dated database of 25 agent entries (21 vendor-verified). Deploy the artifact for your stack, then verify it with the kit verifier.
How many AI agents can AgentBlock block?
The database holds 25 entries: 21 verified vendor-documented agents, 1 reported-but-unverified listing, and 3 watch entries with no vendor-published user-agent. The default policy blocks 20 verified agents and allows 1 utility agent; watch entries cannot be blocked by user-agent rules and are never guessed.
Can a user-agent block be bypassed?
Yes. AgentBlock states it on every kit: sophisticated spoofing can bypass any user-agent block — an agent sending a stock browser UA, or rotating residential IPs. This is a control, not a guarantee.
Which deployment should I choose?
The Cloudflare Worker enforces at the edge and is the free artifact. The WordPress plugin enforces on your server (Pro). The plain-JS snippet is a deterrent and notice layer only (Pro) — any agent that disables JavaScript bypasses it completely.
Is the blocking kit free?
Building a policy, browsing the database, and downloading the Cloudflare Worker template are free. Pro ($29/mo) adds the WordPress plugin and JS snippet downloads; Team ($79/mo) adds the AgentGate policy export and the classify/database API.
Build my block kit — free · Which crawlers should you allow?