← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

How Do You Block ClaudeBot and Keep Claude Citations?

The short answer: Anthropic's agents split the same way OpenAI's do, with one extra token. ClaudeBot and the anthropic-ai token are training crawlers — block both to keep your pages out of training crawls. Claude-SearchBot indexes for Claude's answers and Claude-User fetches when a user asks — allow both if you want to remain citable. All four are verified entries in AgentBlock's database (v2026-10-18), so the split is four explicit decisions in the builder, not a vendor-wide guess.

Build my block kit — free · Test a policy in the simulator first

What is ClaudeBot?

Anthropic's training crawler, a verified entry in AgentBlock's database. Anthropic also publishes anthropic-ai (a second training token), Claude-SearchBot (search retrieval feeding Claude answers) and Claude-User (user-triggered fetching). The training pair and the retrieval pair can be decided independently.

Four Anthropic tokens, two jobs

Anthropic is the vendor most likely to be half-blocked by accident, because it publishes two training tokens rather than one. ClaudeBot is the known name; anthropic-ai is the older training crawl token sitting quietly in the same database class. A policy that blocks ClaudeBot and forgets anthropic-ai leaves half the training door open while reporting a complete block — the worst outcome, because it is believed.

The retrieval pair mirrors OpenAI's. Claude-SearchBot builds the retrieval index behind Claude's answers; Claude-User fetches a page at the moment a user asks Claude about it. Both are verified, both default to block under AgentBlock's retrieval-class default, and both must be deliberately allowed if citations are part of your distribution — which, for most content sites, they increasingly are.

Why the training pair matters together

Training crawls are inventory operations: the crawler works through your site systematically, collecting text and media for a dataset. Two tokens doing that job means two fetch patterns in your logs, and owners who watch only for the famous name undercount the crawl by whatever share the second token carries. In the worked example below, anthropic-ai accounted for roughly a third of the training fetches.

Blocking both tokens is also what makes the block auditable. The kit's block log records verdicts per agent, so a reviewer can see ClaudeBot refusals and anthropic-ai refusals as separate lines and confirm the pair is closed. A policy you cannot audit per agent is a policy you are taking on faith.

The citation case for the retrieval pair

Claude's answers cite sources, and the citation pipeline is the retrieval pair. Sites that allow Claude-SearchBot and Claude-User appear as linked sources when Claude answers questions in their niche; sites that block them are simply absent from those answers, with no notification and no error anywhere. The absence is silent, which is why it so often goes unnoticed for quarters.

There is also a freshness argument for Claude-User specifically. A user-triggered fetch reads the current page — today's price, today's opening hours, today's corrected guide. Blocking it does not just lose a citation; it leaves the model answering from older indexed copies or from competitors' pages instead.

Building and checking the split

In the builder: ClaudeBot block, anthropic-ai block, Claude-SearchBot allow, Claude-User allow. Run the draft through the pre-deploy simulator, which tests the policy against every known agent and your key paths, and use the impact preview against a slice of your real server log if you want to see the before picture — which named agents have actually been visiting, and what the new policy would have done to each visit.

After deployment, the kit verifier confirms the artifact on your stack matches the policy — blocked list, allowed list, kit ID and hash. Anthropic's tokens are exactly the kind that drift when someone hand-edits a Worker or a plugin file months later; the hash is what catches the edit.

When to block the retrieval pair as well

Some sites should block all four: a paywalled research archive whose summaries substitute for subscriptions, or a support knowledge base whose answers are a paid product. That is a coherent posture when it is chosen with the citation cost stated out loud. AgentBlock's default policy in fact blocks both retrieval agents — the default is the strict posture, and allowing retrieval is the deliberate exception owners make with the tradeoff in front of them.

Whichever way you decide, write the decision down with its date. Crawler policy rots quietly: staff change, stacks migrate, and a retrieval block nobody remembers choosing looks, a year later, like a mysterious traffic decline with no cause anyone can name.

Five actions to take today

  1. Block both training tokens. ClaudeBot and anthropic-ai are a pair — blocking one and not the other is a half-measure that reports as whole.
  2. Decide retrieval explicitly. Claude-SearchBot and Claude-User are your Claude citation supply; allow them unless substitution is your stated objection.
  3. Preview against your real log. Run the impact preview on recent server-log lines to see each Anthropic agent's actual visit counts before you commit.
  4. Simulate the draft kit. Confirm in the simulator that training paths refuse and retrieval paths pass on your key pages.
  5. Verify and date the decision. Run the kit verifier after deploy and record the policy date, so the next review starts from a known position.

Worked example: a recipe archive closes the second training token

Saffron & Smoke, a fictional recipe archive with 1,150 recipes, blocked ClaudeBot in a hand-written server rule in March and considered the job done. When its owner ran AgentBlock's impact preview against 30 days of logs in October, the preview showed 12,480 ClaudeBot requests already refused — and 6,090 anthropic-ai requests served, carrying full recipe text, because the second training token had never been in the rule.

The rebuilt policy blocked both training tokens and allowed the retrieval pair, which the archive wanted for citations in cooking questions. The first month under the kit logged 11,940 training refusals across the two agents — anthropic-ai alone accounted for 4,310 of them — while Claude-User fetches rose from 190 to 263 as more answers cited the archive's technique guides.

The archive's owner now treats the pair as a single line item in a quarterly policy review, alongside a stale check that diffs the deployed kit against the current database version.

Frequently asked questions

What is the anthropic-ai user-agent?

It is Anthropic's second training crawl token, listed as a verified entry in AgentBlock's database alongside ClaudeBot. Sites that block only ClaudeBot commonly leave anthropic-ai untouched, which is why AgentBlock's guidance always treats the two as a pair.

Does Claude cite sources the way ChatGPT does?

Yes — Claude's answers draw on Claude-SearchBot's retrieval index and Claude-User's live fetches, and cited pages are linked. Blocking the retrieval pair removes your pages from that citation pool; blocking only the training pair leaves it intact.

If I block ClaudeBot, is my content out of Claude's training data?

Future ClaudeBot crawls of your site stop. Copies already collected are not recalled by a crawler block, and anthropic-ai must be blocked too or the training crawl continues under the second token.

Why does AgentBlock's default block Claude's retrieval agents?

The default policy is the strict posture: block every verified agent except the utility link-preview fetcher. Allowing retrieval is a deliberate, per-agent flip in the builder for owners who value citations over strictness — the default makes that a choice, not an accident.

Can I test the Anthropic split before deploying?

Yes. The pre-deploy simulator runs a draft policy against every known agent and your key paths, and the impact preview replays the policy against your own server-log lines. Both are free, and neither changes anything live.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back