← All guides · Reviewed November 2026
Which AI Crawlers Should You Allow?
The short answer: allow the crawlers that send readers back — search, retrieval, and utility agents — and decide training crawlers on whether you want your pages in future models. The classes do different jobs: blocking a retrieval agent removes your pages as citable live sources, while blocking a training crawler changes nothing a visitor sees today. AgentBlock's database classes all 25 of its entries so the decision is per class, not per logo. Database v2026-10-18, reviewed November 2026.
What does each crawler class actually do?
| Class | What it does | Named examples (database notes) | If you block it |
|---|---|---|---|
| Training | Collects pages to train models | GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended | Your pages stay out of that training crawl; nothing visitor-facing changes |
| Search / indexing | Indexes pages so AI answers can find and cite them | OAI-SearchBot ("search/retrieval crawler feeding ChatGPT answers"), Claude-SearchBot, meta-webindexer ("allowing it helps Meta AI cite and link to your content") | AI answers lose a route to your pages |
| User-triggered fetch (agents) | Fetches a page when a person asks an AI about it | ChatGPT-User ("blocking removes your pages as citable live sources"), Claude-User, MistralAI-User | The assistant cannot read your page for the person asking |
| Utility | Link previews and platform plumbing | facebookexternalhit — the one agent AgentBlock's default policy allows, because blocking breaks link previews | Your links preview badly or not at all |
| Watch (no published name) | Real agents with no vendor-published user-agent | ChatGPT Atlas, Meta's Muse agent, Grok retrieval | Cannot be blocked by name at all — classified UNKNOWN, never guessed |
One entry deserves its own line: meta-externalfetcher. Meta's own documentation, as recorded in the database, says it may bypass robots.txt because the request comes from a person. A robots rule is a request; a user-agent rule at your edge or server is an enforcement — which is why the kit exists alongside robots.txt rather than instead of it. The full per-agent evidence is on the database page.
What posture does this platform take on its own pages?
Allow — and it is worth stating plainly, because a blocking tool preaching allow-all would be odd and allow-none would be dishonest. AgentBlock's own robots.txt serves User-agent: * / Allow: /: AI crawlers are allowed on its public surfaces, and the Ingenious AI fleet takes the same posture across its public pages (getingeniousai.com). The reasoning is the table above: for a platform that wants to be cited by AI answers, retrieval and search access is the point. Your site may weigh the training class differently — that is exactly the choice the builder puts in your hands, per agent. How the blocking itself works: how to block AI scrapers and bots.
How should you decide, class by class?
- Utility: allow. Blocking link-preview agents breaks how your pages look when shared. This is the default policy's one allow, for that reason.
- Search and user-triggered fetch: allow if you want citations. These are the agents that read your page for a person, or index it so an answer can cite it.
- Training: your call. Blocking is the common choice for sites that do not want their content in training sets; it costs nothing visitor-facing today.
- Ads and scraping classes: block unless you use them. They serve the vendor's pipeline, not your readers.
- Watch entries: accept UNKNOWN. No published name means no honest rule. Any tool that claims to block them by user-agent is guessing.
Frequently asked questions
Should I block GPTBot?
GPTBot is OpenAI's training crawler in AgentBlock's database. Blocking it stops that training crawl; it does not stop ChatGPT from fetching your pages when a user asks — that is ChatGPT-User, a separate retrieval agent. Decide the two separately: many sites block training and allow retrieval so AI answers can still cite them.
What is the difference between training, search, and user-triggered crawlers?
Training crawlers (GPTBot, ClaudeBot, CCBot) collect pages to train models. Search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot) index pages so AI answers can cite them. User-triggered fetchers (ChatGPT-User, Claude-User) fetch a page when a person asks about it. AgentBlock's database records the class of each of its 25 entries.
Which AI crawlers does AgentBlock allow by default?
Only one: facebookexternalhit, classed utility — link previews, where blocking breaks a feature. The default policy blocks the other 20 verified agents across training, indexing, retrieval, ads, and scraping classes. Every default is yours to flip in the builder.
Can every AI agent be blocked by name?
No. Three database entries are watch entries with no vendor-published user-agent — ChatGPT Atlas, Meta's Muse agent, and Grok retrieval. There is no name to bind a rule to, so AgentBlock classifies them UNKNOWN and never guesses a block rule for them.
What posture do AgentBlock's own public surfaces take?
Allow. AgentBlock's own robots.txt serves 'User-agent: * / Allow: /' — AI crawlers are allowed on its public surfaces, and the Ingenious AI fleet takes the same posture on its public pages. Blocking is a per-site choice the kit makes deployable; it is not the posture this platform takes for itself.
Set my policy — free builder · Deploy it: WordPress, Cloudflare, plain JS