Evidence, dated
Agent database v2026-10-18
v2026-10-18: initial release. 21 verified vendor-documented agents, 1 reported-but-unverified directory listing, 3 watch entries from the Oct 2026 blocking wave (Meta Muse, xAI Grok, ChatGPT Atlas) with no vendor-published user-agent — classified UNKNOWN by design.
verified 21 reported 1 watch 3
Evidence labels: verified = the vendor publishes this user-agent in its own crawler docs — these are the only tokens emitted into block lists. reported = seen in reputable bot directories but not vendor-published — shown, never emitted. watch = the Oct 2026 blocking wave (Meta Muse, xAI Grok, ChatGPT Atlas): no vendor-published user-agent exists, so no rule can be bound — they always classify as UNKNOWN.
| Agent | Vendor | UA token(s) | Evidence | Added / verified | Why |
|---|---|---|---|---|---|
| GPTBot training |
OpenAI | GPTBot |
verified | 2026-10-18 / 2026-10-18 | Training crawler. JS support limited; SSR recommended if you allow it. Source: OpenAI GPTBot documentation; corroborated by AI user-agent reference, nohacks.co (Oct 2026). |
| ChatGPT-User retrieval |
OpenAI | ChatGPT-User |
verified | 2026-10-18 / 2026-10-18 | Fetches pages when a ChatGPT user asks about your content. Blocking removes your pages as citable live sources. Source: OpenAI documentation; corroborated by nohacks.co (Oct 2026). |
| OAI-SearchBot retrieval |
OpenAI | OAI-SearchBot |
verified | 2026-10-18 / 2026-10-18 | Search/retrieval crawler feeding ChatGPT answers. Source: OpenAI documentation; corroborated by nohacks.co (Oct 2026). |
| ChatGPT Atlas retrieval |
OpenAI | — none published — | watch | 2026-10-18 / 2026-10-18 | Atlas uses a standard Chrome UA — indistinguishable from a regular browser. CANNOT be blocked by user-agent rules. Always UNKNOWN. Source: Bot directory reporting, momenticmarketing.com (2025); OpenAI publishes no distinct Atlas user-agent. |
| ClaudeBot training |
Anthropic | ClaudeBot |
verified | 2026-10-18 / 2026-10-18 | Training crawler. Source: Anthropic documentation; corroborated by fixaeo.com (2026). |
| Claude-User retrieval |
Anthropic | Claude-User |
verified | 2026-10-18 / 2026-10-18 | User-triggered fetch for Claude answers. Source: Anthropic documentation; corroborated by fixaeo.com (2026). |
| Claude-SearchBot retrieval |
Anthropic | Claude-SearchBot |
verified | 2026-10-18 / 2026-10-18 | Search retrieval feeding Claude answers. Source: Anthropic documentation; corroborated by fixaeo.com (2026). |
| anthropic-ai training |
Anthropic | anthropic-ai |
verified | 2026-10-18 / 2026-10-18 | Anthropic's training crawl token. Source: Anthropic documentation. |
| Meta-ExternalAgent training |
Meta | meta-externalagent |
verified | 2026-10-18 / 2026-10-18 | Crawls 'to train foundation AI models or index content directly'. Block via robots.txt disallow; Meta says changes can take up to 24h. Source: Meta web crawlers documentation (updated 2026-05-21); reported by rrtechpulse.com, maccus.fr, aieranews.com (Oct 16-17 2026) re: Oct 30 Commercial Terms crawl clause. |
| meta-webindexer indexing |
Meta | meta-webindexer |
verified | 2026-10-18 / 2026-10-18 | Improves Meta AI search; Meta says allowing it helps Meta AI cite and link to your content. Source: Meta web crawlers documentation; reported Oct 2026. |
| meta-externalfetcher retrieval |
Meta | meta-externalfetcher |
verified | 2026-10-18 / 2026-10-18 | Fetches a page when a user asks Meta's AI to — INCLUDING for AI agents. Meta says it MAY BYPASS robots.txt because the request comes from a person. User-agent rules are unreliable here. Source: Meta web crawlers documentation; reported Oct 2026. |
| meta-externalads ads |
Meta | meta-externalads |
verified | 2026-10-18 / 2026-10-18 | Crawls to improve advertising and business products. Source: Meta web crawlers documentation; reported Oct 2026. |
| facebookexternalhit utility |
Meta | facebookexternalhit |
verified | 2026-10-18 / 2026-10-18 | NOT an AI bot — link-preview fetcher for shared URLs. Blocking it breaks link previews on Facebook/WhatsApp. Defaults to ALLOW. Source: Meta web crawlers documentation; nohacks.co (Oct 2026). |
| Muse (Meta AI agent) retrieval |
Meta | — none published — | watch | 2026-10-18 / 2026-10-18 | Meta publishes NO user-agent for Muse. Amazon blocked Muse through its Conditions of Use (Sept 2026) because robots.txt 'had no name to bind to'. Always UNKNOWN — policy/contract controls only, never UA rules. Source: nohacks.co, The AI User-Agent Landscape in 2026 (Oct 2026). |
| Grok (xAI) retrieval |
xAI | — none published — | watch | 2026-10-18 / 2026-10-18 | Grok retrieval uses residential IP rotation and spoofed Safari/Chrome/iPhone user-agents — functionally indistinguishable from human visitors at the UA layer. Directory-listed 'GrokBot'/'xAI-Bot' tokens are NOT vendor-published and are excluded from block lists. Always UNKNOWN. Source: nohacks.co (Oct 2026): 'xAI publishes no crawler documentation page'; fixaeo.com (2026): 'Grok's retrieval traffic often doesn't identify itself and fetches like an ordinary browser'. |
| Google-Extended training |
Google-Extended |
verified | 2026-10-18 / 2026-10-18 | Training opt-out token for Google's AI models. Does not affect Google Search indexing. Source: Google crawler documentation; nexterwp.com (Oct 2026). |
|
| Applebot-Extended training |
Apple | Applebot-Extended |
verified | 2026-10-18 / 2026-10-18 | Training opt-out token for Apple Intelligence. Source: Apple documentation; corroborated by natebal.com (2026). |
| PerplexityBot retrieval |
Perplexity | PerplexityBot |
verified | 2026-10-18 / 2026-10-18 | Real-time search grounding; drives citations. Source: Perplexity documentation; corroborated by natebal.com (2026). |
| CCBot training |
Common Crawl | CCBot |
verified | 2026-10-18 / 2026-10-18 | Open web-corpus crawler — upstream for most open LLM training sets. Common Crawl warns others falsely identify as CCBot; real CCBot uses dedicated IP ranges. Source: Common Crawl CCBot documentation; nexterwp.com (Oct 2026). |
| Bytespider training |
ByteDance | Bytespider |
verified | 2026-10-18 / 2026-10-18 | Data collection for ByteDance AI products; reputation for aggressive crawling. Source: Reputable bot directories, nexterwp.com (Oct 2026) — no official ByteDance docs page found as of Oct 7 2026. |
| MistralAI-User retrieval |
Mistral | MistralAI-User |
verified | 2026-10-18 / 2026-10-18 | Fetches content for citations in Mistral's Le Chat. Source: Bot directory reporting, momenticmarketing.com (2025). |
| YouBot retrieval |
You.com | YouBot |
verified | 2026-10-18 / 2026-10-18 | Powers You.com AI search and browser assistant. Source: Bot directory reporting, momenticmarketing.com (2025). |
| cohere-ai training |
Cohere | cohere-ai |
verified | 2026-10-18 / 2026-10-18 | Cohere training crawl token. Source: Cohere documentation. |
| FirecrawlAgent scraping |
Firecrawl | FirecrawlAgent |
verified | 2026-10-18 / 2026-10-18 | Scraping-API agent; identifies itself honestly. Source: Firecrawl documentation. |
| GrokBot (directory-reported) retrieval |
xAI | GrokBot xAI-Bot |
reported | 2026-10-18 / 2026-10-18 | Listed for transparency only. Because xAI publishes no crawler documentation, these tokens are EXCLUDED from generated block lists — a rule bound to an unverified token is a guess, and we don't guess. Source: Bot directories (texta.ai, menra.ai, fixaeo.com — 2026); NOT published by xAI. fixaeo.com: 'independent reporting, including from Cloudflare, says Grok's retrieval traffic often doesn't identify itself'. |
Honest scope: HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.