{"version":"2026-10-18","summary":{"version":"2026-10-18","notes":"v2026-10-18: initial release. 21 verified vendor-documented agents, 1 reported-but-unverified directory listing, 3 watch entries from the Oct 2026 blocking wave (Meta Muse, xAI Grok, ChatGPT Atlas) with no vendor-published user-agent — classified UNKNOWN by design.","total":25,"by_evidence":{"verified":21,"watch":3,"reported":1},"blockable":21},"entries":[{"id":"gptbot","name":"GPTBot","vendor":"OpenAI","category":"training","evidence":"verified","ua_substrings":["GPTBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"OpenAI GPTBot documentation; corroborated by AI user-agent reference, nohacks.co (Oct 2026).","note":"Training crawler. JS support limited; SSR recommended if you allow it."},{"id":"chatgpt-user","name":"ChatGPT-User","vendor":"OpenAI","category":"retrieval","evidence":"verified","ua_substrings":["ChatGPT-User"],"added":"2026-10-18","last_verified":"2026-10-18","source":"OpenAI documentation; corroborated by nohacks.co (Oct 2026).","note":"Fetches pages when a ChatGPT user asks about your content. Blocking removes your pages as citable live sources."},{"id":"oai-searchbot","name":"OAI-SearchBot","vendor":"OpenAI","category":"retrieval","evidence":"verified","ua_substrings":["OAI-SearchBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"OpenAI documentation; corroborated by nohacks.co (Oct 2026).","note":"Search/retrieval crawler feeding ChatGPT answers."},{"id":"chatgpt-atlas","name":"ChatGPT Atlas","vendor":"OpenAI","category":"retrieval","evidence":"watch","ua_substrings":[],"added":"2026-10-18","last_verified":"2026-10-18","source":"Bot directory reporting, momenticmarketing.com (2025); OpenAI publishes no distinct Atlas user-agent.","note":"Atlas uses a standard Chrome UA — indistinguishable from a regular browser. CANNOT be blocked by user-agent rules. Always UNKNOWN."},{"id":"claudebot","name":"ClaudeBot","vendor":"Anthropic","category":"training","evidence":"verified","ua_substrings":["ClaudeBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Anthropic documentation; corroborated by fixaeo.com (2026).","note":"Training crawler."},{"id":"claude-user","name":"Claude-User","vendor":"Anthropic","category":"retrieval","evidence":"verified","ua_substrings":["Claude-User"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Anthropic documentation; corroborated by fixaeo.com (2026).","note":"User-triggered fetch for Claude answers."},{"id":"claude-searchbot","name":"Claude-SearchBot","vendor":"Anthropic","category":"retrieval","evidence":"verified","ua_substrings":["Claude-SearchBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Anthropic documentation; corroborated by fixaeo.com (2026).","note":"Search retrieval feeding Claude answers."},{"id":"anthropic-ai","name":"anthropic-ai","vendor":"Anthropic","category":"training","evidence":"verified","ua_substrings":["anthropic-ai"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Anthropic documentation.","note":"Anthropic's training crawl token."},{"id":"meta-externalagent","name":"Meta-ExternalAgent","vendor":"Meta","category":"training","evidence":"verified","ua_substrings":["meta-externalagent"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Meta web crawlers documentation (updated 2026-05-21); reported by rrtechpulse.com, maccus.fr, aieranews.com (Oct 16-17 2026) re: Oct 30 Commercial Terms crawl clause.","note":"Crawls 'to train foundation AI models or index content directly'. Block via robots.txt disallow; Meta says changes can take up to 24h."},{"id":"meta-webindexer","name":"meta-webindexer","vendor":"Meta","category":"indexing","evidence":"verified","ua_substrings":["meta-webindexer"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Meta web crawlers documentation; reported Oct 2026.","note":"Improves Meta AI search; Meta says allowing it helps Meta AI cite and link to your content."},{"id":"meta-externalfetcher","name":"meta-externalfetcher","vendor":"Meta","category":"retrieval","evidence":"verified","ua_substrings":["meta-externalfetcher"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Meta web crawlers documentation; reported Oct 2026.","note":"Fetches a page when a user asks Meta's AI to — INCLUDING for AI agents. Meta says it MAY BYPASS robots.txt because the request comes from a person. User-agent rules are unreliable here."},{"id":"meta-externalads","name":"meta-externalads","vendor":"Meta","category":"ads","evidence":"verified","ua_substrings":["meta-externalads"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Meta web crawlers documentation; reported Oct 2026.","note":"Crawls to improve advertising and business products."},{"id":"facebookexternalhit","name":"facebookexternalhit","vendor":"Meta","category":"utility","evidence":"verified","ua_substrings":["facebookexternalhit"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Meta web crawlers documentation; nohacks.co (Oct 2026).","note":"NOT an AI bot — link-preview fetcher for shared URLs. Blocking it breaks link previews on Facebook/WhatsApp. Defaults to ALLOW."},{"id":"meta-muse","name":"Muse (Meta AI agent)","vendor":"Meta","category":"retrieval","evidence":"watch","ua_substrings":[],"added":"2026-10-18","last_verified":"2026-10-18","source":"nohacks.co, The AI User-Agent Landscape in 2026 (Oct 2026).","note":"Meta publishes NO user-agent for Muse. Amazon blocked Muse through its Conditions of Use (Sept 2026) because robots.txt 'had no name to bind to'. Always UNKNOWN — policy/contract controls only, never UA rules."},{"id":"xai-grok","name":"Grok (xAI)","vendor":"xAI","category":"retrieval","evidence":"watch","ua_substrings":[],"added":"2026-10-18","last_verified":"2026-10-18","source":"nohacks.co (Oct 2026): 'xAI publishes no crawler documentation page'; fixaeo.com (2026): 'Grok's retrieval traffic often doesn't identify itself and fetches like an ordinary browser'.","note":"Grok retrieval uses residential IP rotation and spoofed Safari/Chrome/iPhone user-agents — functionally indistinguishable from human visitors at the UA layer. Directory-listed 'GrokBot'/'xAI-Bot' tokens are NOT vendor-published and are excluded from block lists. Always UNKNOWN."},{"id":"google-extended","name":"Google-Extended","vendor":"Google","category":"training","evidence":"verified","ua_substrings":["Google-Extended"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Google crawler documentation; nexterwp.com (Oct 2026).","note":"Training opt-out token for Google's AI models. Does not affect Google Search indexing."},{"id":"applebot-extended","name":"Applebot-Extended","vendor":"Apple","category":"training","evidence":"verified","ua_substrings":["Applebot-Extended"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Apple documentation; corroborated by natebal.com (2026).","note":"Training opt-out token for Apple Intelligence."},{"id":"perplexitybot","name":"PerplexityBot","vendor":"Perplexity","category":"retrieval","evidence":"verified","ua_substrings":["PerplexityBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Perplexity documentation; corroborated by natebal.com (2026).","note":"Real-time search grounding; drives citations."},{"id":"ccbot","name":"CCBot","vendor":"Common Crawl","category":"training","evidence":"verified","ua_substrings":["CCBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Common Crawl CCBot documentation; nexterwp.com (Oct 2026).","note":"Open web-corpus crawler — upstream for most open LLM training sets. Common Crawl warns others falsely identify as CCBot; real CCBot uses dedicated IP ranges."},{"id":"bytespider","name":"Bytespider","vendor":"ByteDance","category":"training","evidence":"verified","ua_substrings":["Bytespider"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Reputable bot directories, nexterwp.com (Oct 2026) — no official ByteDance docs page found as of Oct 7 2026.","note":"Data collection for ByteDance AI products; reputation for aggressive crawling."},{"id":"mistralai-user","name":"MistralAI-User","vendor":"Mistral","category":"retrieval","evidence":"verified","ua_substrings":["MistralAI-User"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Bot directory reporting, momenticmarketing.com (2025).","note":"Fetches content for citations in Mistral's Le Chat."},{"id":"youbot","name":"YouBot","vendor":"You.com","category":"retrieval","evidence":"verified","ua_substrings":["YouBot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Bot directory reporting, momenticmarketing.com (2025).","note":"Powers You.com AI search and browser assistant."},{"id":"cohere-ai","name":"cohere-ai","vendor":"Cohere","category":"training","evidence":"verified","ua_substrings":["cohere-ai"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Cohere documentation.","note":"Cohere training crawl token."},{"id":"firecrawl","name":"FirecrawlAgent","vendor":"Firecrawl","category":"scraping","evidence":"verified","ua_substrings":["FirecrawlAgent"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Firecrawl documentation.","note":"Scraping-API agent; identifies itself honestly."},{"id":"grokbot-reported","name":"GrokBot (directory-reported)","vendor":"xAI","category":"retrieval","evidence":"reported","ua_substrings":["GrokBot","xAI-Bot"],"added":"2026-10-18","last_verified":"2026-10-18","source":"Bot directories (texta.ai, menra.ai, fixaeo.com — 2026); NOT published by xAI. fixaeo.com: 'independent reporting, including from Cloudflare, says Grok's retrieval traffic often doesn't identify itself'.","note":"Listed for transparency only. Because xAI publishes no crawler documentation, these tokens are EXCLUDED from generated block lists — a rule bound to an unverified token is a guess, and we don't guess."}]}