← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read
What Are the Most Common AI Blocklist Design Mistakes?
The short answer: five mistakes account for nearly every bad AI blocklist: deciding by vendor family instead of by agent; copying unverified lists whose tokens were never vendor-published; blocking the link-preview fetcher and breaking your own shares; blocking one of a vendor's two training tokens and reporting a complete block; and enforcing nothing — a robots file standing alone. Every one of them is a decision made at the wrong granularity, and AgentBlock's database grades and per-agent builder exist to make the right granularity the easy one.
Build my block kit — free · Test a policy in the simulator first
What is a blocklist design mistake?
a structural error in how a blocking policy is assembled — wrong unit of decision, wrong grade of evidence, wrong layer of enforcement — as distinct from a wrong opinion about any single agent. Mistakes of design repeat across every agent the list touches, which is why they cost more than any individual decision.
Mistake one: the family switch
'Block Meta' feels decisive and is meaningless as policy. Meta publishes five database entries with five jobs — training, indexing, user-triggered fetch, ads, link previews — spanning every position a considered policy might take, from default-block to default-allow. A family rule necessarily gets at least three of the five wrong. OpenAI's three and Anthropic's four have the same property: the vendor is the worst available unit of decision, and the agent is the right one.
The tell in inherited lists: rules matching short substrings like 'meta' or 'gpt' that catch agents their author never heard of, including future ones. A rule whose blast radius its author cannot enumerate is not a rule; it is an incident scheduled for later.
Mistake two: phantom and reported tokens
Forum lists circulate tokens nobody can source. Some are directory-reported — GrokBot is the type specimen — and some are pure invention, tokens for crawlers that may not exist, copied from list to list for years. Each phantom rule inflates the blocked count a policy reports while matching nothing in the world, and the inflated count is then quoted in documents other people rely on. Evidence inflation is the quiet corruption of security reporting.
AgentBlock's grades are the antidote: verified tokens only in block lists, reported entries shown but never emitted, watch entries never guessed. An inherited list should be graded token by token before it is trusted, and the ones that fail grading should be removed even though — especially though — the blocked-agent count goes down.
Mistake three: friendly fire on utilities
The preview fetcher is the famous casualty, but the class is wider: any fetcher your own features depend on — partner previews, required checkers, your own monitoring under a stable token — can be swept up by a broad list. Friendly fire is usually discovered by symptom (shares sag, a dashboard gaps, a partner complains) weeks after the deploy, because nothing in the blocking stack reports 'you refused something you needed'.
Prevention is procedural: before any list ships, read the allow side of the evidence — the impact preview shows what your draft policy would refuse, per agent, against your real log. Thirty seconds of reading the refusal list catches the utilities. Skipping the preview is how launches break.
Mistake four: the forgotten second token
Anthropic publishes two training tokens; owners who block ClaudeBot and stop have blocked, in observed traffic, roughly two-thirds of the training crawl while believing it closed. The general form: vendors rename, split and add agents over time, and a hand-maintained list freezes the author's knowledge at writing date. Every month after, the list and the world diverge a little more.
The structural answer is a versioned database with a stale check, not better memory. Your kit should know which database version it was built from, and something — AgentBlock's stale check — should tap you on the shoulder when the world has moved. A blocklist without a freshness mechanism has an expiry date it does not print.
Mistake five: declaration without enforcement
The robots-only policy: carefully written, publicly filed, and binding on exactly the crawlers polite enough to be bound — with a documented bypass on the user-triggered class and silent erasure one plugin update away. Declaration is a layer, and a useful one for evidence and for the token opt-outs that live there. As the only layer, it is a press release.
Enforcement completes the design: the same decisions compiled into a Worker or a plugin that refuses requests mechanically, a verifier that proves the artifact matches the policy, and an audit that keeps the robots layer honest. Owners who adopt the full stack stop having blocklist incidents of every class above — not because their opinions improved, but because their structure stopped permitting the mistakes.
Five actions to take today
- Decide per agent, never per family. If a rule's substring can catch an agent you cannot name, the rule is too broad.
- Grade every token. Vendor-published or it does not ship; reported tokens are monitoring notes, not rules.
- Preview refusals against your real log. The impact preview's refusal list is where friendly fire is caught before launch.
- Block tokens in their sets. Both Anthropic training tokens, all nine training entries for a class decision — completeness is a checklist, not a memory feat.
- Enforce, verify, audit. Worker or plugin for refusals, kit verifier for artifact truth, enforcement-gap audit for the robots layer.
Worked example: one inherited list, four mistakes
Sable & Co., a fictional homeware retailer, inherited a 31-line block list from a previous agency and asked for a review before renewing the retainer. The grading exercise found all four content mistakes in one file: a 'meta' substring rule catching facebookexternalhit (mistakes one and three), two directory-reported tokens and one phantom token inflating the count (mistake two), ClaudeBot blocked while anthropic-ai fetched freely (mistake four) — and the whole list existed only in robots.txt (mistake five).
Measured against the retailer's log, the list's real effect was: 6,400 training fetches a month still served via the second token, previews intermittently broken for six months, and a reported '24 agents blocked' that an evidence check reduced to 17 verified names.
The rebuilt AgentBlock policy blocks 20 verified agents, allows the preview fetcher by construction, and enforces at the edge. The review's closing line is now the retailer's standing instruction to any agency: counts you cannot grade are not counts.
Frequently asked questions
My block list works fine — why audit it?
Because each mistake class is silent. Phantom tokens fail by matching nothing, friendly fire announces itself in other teams' metrics, a forgotten second token shows up only in a log read per-agent, and robots drift leaves no error. 'No symptoms' is the expected presentation of all five.
Is a bigger blocked count ever the goal?
No. The goal is a policy whose every line is evidence-graded and whose effects are measured. A list blocking 20 verified agents with clean logs is worth more than one claiming 31 with phantoms, friendly fire and an open second token.
How do I grade a token I found online?
Ask for the vendor's own crawler documentation naming it. If the vendor publishes it, the token is verified-grade. If only directories list it, it is reported-grade: monitoring material. If nothing sources it, it is a phantom: remove it and note the removal so the count change is explained.
What is the fastest full check of an existing list?
Three passes: grade the tokens against the database; run the impact preview to see refusals per agent on your real traffic; run the enforcement-gap audit to see whether robots, meta and edge agree. Together they take under an hour and cover all five mistake classes.
Can AgentBlock import my existing list?
Policies are built from the builder's per-agent decisions, which is the point — importing an ungraded list wholesale would import its mistakes. The productive route is to grade your list against the database and re-express the surviving decisions; owners typically find the rebuild takes one sitting.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- Should You Block or Allow PerplexityBot?
- What Is Meta-ExternalAgent and the October 2026 Crawl Clause?
- Why Can Meta-ExternalFetcher Bypass Your Robots.txt?
- Why Should You Never Block facebookexternalhit?
- What Is FirecrawlAgent and Should Your Site Block It?
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded