← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

What Is an AI Crawler?

The short answer: an AI crawler is a program that requests pages from your website on behalf of an AI vendor — to train a model, to index your content for AI answers, to fetch a page a user just asked about, or to feed a scraping service. Unlike a traditional search crawler, each AI crawler is tied to a specific downstream use, so the useful question is never 'bots or no bots' but 'which named agents, for which job, on which pages'. AgentBlock's database names 25 of them.

Build my block kit — free · Test a policy in the simulator first

What is an AI crawler?

a software agent that fetches web pages for an AI system rather than for a human reader or a classic search index. AgentBlock's database (v2026-10-18) records 25 entries in six classes — training, retrieval, indexing, ads, scraping and utility — each labelled verified, reported or watch according to the evidence behind its user-agent name.

The names are the policy surface

Every AI crawler that identifies itself sends a user-agent string — GPTBot, ClaudeBot, PerplexityBot and so on. That name is the only handle a small site has: you cannot interview a crawler, but you can match its name at your edge or server and refuse the request. AgentBlock builds its whole kit on that handle, matching vendor-published user-agent substrings against each request before the page is served.

Names also carry evidence grades. In the database, 21 entries are verified — the vendor publishes the user-agent in its own crawler documentation. One entry is reported only, seen in bot directories but never published by its vendor, and three are watch entries with no published name at all. Only verified names are ever emitted into a block list, because a rule bound to a guessed name is a guess, not a policy.

Six classes, six different jobs

Training crawlers — nine entries, including GPTBot, ClaudeBot, CCBot and Bytespider — collect pages to build model training sets. Retrieval crawlers fetch or index pages so an AI answer can cite them: OAI-SearchBot and Claude-SearchBot index ahead of time, while ChatGPT-User and Claude-User fetch at the moment a person asks. The classes matter because blocking training and blocking retrieval have opposite visibility effects.

The remaining classes are narrower. Indexing has one entry, Meta's meta-webindexer, improving Meta AI search. Ads has one, meta-externalads, crawling to improve advertising products. Scraping has one, FirecrawlAgent, a scraping-API agent that identifies itself honestly. Utility has one, facebookexternalhit, which is not an AI bot at all — it fetches link previews, and blocking it breaks how your pages look when shared.

What an AI crawler is not

An AI crawler is not Googlebot. Classic search indexing is a separate system with separate tokens; Google's AI training opt-out, for example, is a distinct token called Google-Extended, and using it does not remove a site from Google Search. Conflating search crawlers with AI crawlers is how sites accidentally block the traffic they wanted to keep while leaving the training crawl they meant to stop untouched.

An AI crawler is also not a hacker. The named agents are, by and large, honest about who they are — that honesty is what makes name-based policy possible. The dishonest traffic, agents sending stock browser user-agents or rotating residential IPs, is a different problem, and AgentBlock states plainly on every kit that sophisticated spoofing can bypass any user-agent block.

Why sites started deciding per agent

For years the only choices were robots.txt politeness and doing nothing. Three things changed: vendors multiplied (five major vendors account for most of the 25 database entries), the uses diverged (training versus citation is a real business tradeoff), and vendors began publishing crawler documentation that made per-agent rules enforceable rather than aspirational. A site can now say 'train on nothing, cite anything' and actually implement it.

That is the decision AgentBlock's builder encodes. Each verified agent starts at its class default — the default policy blocks 20 agents and allows the one utility agent — and the owner flips any of them. The output is not advice but an artifact: a Cloudflare Worker, a WordPress plugin, or a plain-JS snippet generated from the dated database.

The three agents with no name

ChatGPT Atlas, Meta's Muse agent and Grok retrieval sit in the database as watch entries because their vendors publish no user-agent for them. Atlas browses with a standard Chrome user-agent; Grok retrieval has been reported fetching like an ordinary browser. There is no name to bind a rule to, so AgentBlock classifies them UNKNOWN and never guesses a block.

That honesty shapes expectations. A kit governs the named, identified majority of AI crawling. For the unnamed remainder, the available controls are contractual and behavioural — terms of use, rate behaviour, path protection — not name rules. Any tool that promises to block an agent it cannot name is selling the guess AgentBlock refuses to make.

Where to see the actual list

The full database is public on this site: every entry with its vendor, class, evidence grade, source and last-verified date. Reading it is the fastest way to make the abstract concrete — you will recognise some names from your own server logs and be surprised by others, and the evidence labels show exactly how much weight each name can bear before you build a policy on it.

Five actions to take today

  1. Pull your server log. List every user-agent containing Bot, crawler or a vendor name from the last 30 days — those are the agents actually visiting you.
  2. Match names to the database. Check each one against AgentBlock's database page and note its class and evidence grade before deciding anything.
  3. Decide by class first. Training, retrieval, indexing, ads, scraping and utility each get a posture; per-agent flips come second.
  4. Protect the utility agent. Keep facebookexternalhit allowed unless you accept broken link previews as the price.
  5. Build and simulate. Turn the decisions into a kit in the free builder and run the pre-deploy simulator before anything ships.

Worked example: a furniture maker's first crawler census

Harbor Joinery, a fictional 14-person furniture maker in Belize selling through a small WooCommerce site, pulled 30 days of logs before deciding anything. The census found 41,208 requests from named AI agents: GPTBot led with 18,940, followed by ClaudeBot at 9,311, CCBot at 6,204, and a long tail including 1,872 visits from facebookexternalhit fetching previews of shared product pages.

The class split changed the conversation. Training crawlers accounted for 36,510 of the 41,208 requests — the crawl the owners wanted to stop, because their product photography and joinery guides were the shop's only marketing asset. Retrieval agents accounted for just 2,655 requests, and those were the ones producing the AI-answer citations that had begun sending buyers.

Harbor Joinery's first kit therefore blocked the nine training entries plus ads and scraping, allowed the retrieval class and the utility agent, and left indexing blocked pending a review. The deployed Worker logged 1,204 blocked training requests in its first week and zero blocked link previews — the census, not a blanket ban, is what made that precision possible.

Frequently asked questions

Is an AI crawler the same as a search engine crawler?

Not exactly. Some vendors run both, but the tokens and purposes differ. Google-Extended is Google's AI training opt-out token and does not affect Google Search indexing, which is governed separately. AgentBlock's database covers AI agents; classic search crawlers are outside its 25 entries.

How many AI crawlers are there?

AgentBlock's dated database (v2026-10-18) records 25 entries: 21 vendor-verified, 1 reported-but-unverified, and 3 watch entries with no vendor-published user-agent. The real population is larger and changes, which is why the database is versioned and kits carry a stale check.

Can I block all AI crawlers at once?

You can block every verified name in the database with one default policy — AgentBlock's default blocks 20 and allows 1 utility agent. What you cannot do is block the three watch entries by name, because no name exists, or stop agents that spoof a browser user-agent.

Do AI crawlers obey robots.txt?

Many do, but robots.txt is a request, not an enforcement. Meta documents that its user-triggered fetcher may bypass robots.txt because the request originates with a person. AgentBlock generates robots rules as one layer and enforces at the edge or server as the real control.

Which AI crawler visits small sites most?

In most small-site logs the training class dominates, led by GPTBot and ClaudeBot, with CCBot's open-corpus crawl close behind. Your own log is the authority — the worked example above is fictional, and the proportions on your site will differ.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back