Policy builder

Build your block kit

Pick allow/block per known agent. Agents with no vendor-published user-agent (the Oct 2026 watchlist) can't be bound to a rule — they're listed on the database page, never guessed.

AgentVendorCategoryEvidencePolicy
GPTBot
Training crawler. JS support limited; SSR recommended if you allow it.		 OpenAI training verified
ChatGPT-User
Fetches pages when a ChatGPT user asks about your content. Blocking removes your pages as citable live sources.		 OpenAI retrieval verified
OAI-SearchBot
Search/retrieval crawler feeding ChatGPT answers.		 OpenAI retrieval verified
ChatGPT Atlas
Atlas uses a standard Chrome UA — indistinguishable from a regular browser. CANNOT be blocked by user-agent rules. Always UNKNOWN.		 OpenAI retrieval watch not rule-bindable
ClaudeBot
Training crawler.		 Anthropic training verified
Claude-User
User-triggered fetch for Claude answers.		 Anthropic retrieval verified
Claude-SearchBot
Search retrieval feeding Claude answers.		 Anthropic retrieval verified
anthropic-ai
Anthropic's training crawl token.		 Anthropic training verified
Meta-ExternalAgent
Crawls 'to train foundation AI models or index content directly'. Block via robots.txt disallow; Meta says changes can take up to 24h.		 Meta training verified
meta-webindexer
Improves Meta AI search; Meta says allowing it helps Meta AI cite and link to your content.		 Meta indexing verified
meta-externalfetcher
Fetches a page when a user asks Meta's AI to — INCLUDING for AI agents. Meta says it MAY BYPASS robots.txt because the request comes from a person. User-agent rules are unreliable here.		 Meta retrieval verified
meta-externalads
Crawls to improve advertising and business products.		 Meta ads verified
facebookexternalhit
NOT an AI bot — link-preview fetcher for shared URLs. Blocking it breaks link previews on Facebook/WhatsApp. Defaults to ALLOW.		 Meta utility verified
Muse (Meta AI agent)
Meta publishes NO user-agent for Muse. Amazon blocked Muse through its Conditions of Use (Sept 2026) because robots.txt 'had no name to bind to'. Always UNKNOWN — policy/contract controls only, never UA rules.		 Meta retrieval watch not rule-bindable
Grok (xAI)
Grok retrieval uses residential IP rotation and spoofed Safari/Chrome/iPhone user-agents — functionally indistinguishable from human visitors at the UA layer. Directory-listed 'GrokBot'/'xAI-Bot' tokens are NOT vendor-published and are excluded from block lists. Always UNKNOWN.		 xAI retrieval watch not rule-bindable
Google-Extended
Training opt-out token for Google's AI models. Does not affect Google Search indexing.		 Google training verified
Applebot-Extended
Training opt-out token for Apple Intelligence.		 Apple training verified
PerplexityBot
Real-time search grounding; drives citations.		 Perplexity retrieval verified
CCBot
Open web-corpus crawler — upstream for most open LLM training sets. Common Crawl warns others falsely identify as CCBot; real CCBot uses dedicated IP ranges.		 Common Crawl training verified
Bytespider
Data collection for ByteDance AI products; reputation for aggressive crawling.		 ByteDance training verified
MistralAI-User
Fetches content for citations in Mistral's Le Chat.		 Mistral retrieval verified
YouBot
Powers You.com AI search and browser assistant.		 You.com retrieval verified
cohere-ai
Cohere training crawl token.		 Cohere training verified
FirecrawlAgent
Scraping-API agent; identifies itself honestly.		 Firecrawl scraping verified
GrokBot (directory-reported)
Listed for transparency only. Because xAI publishes no crawler documentation, these tokens are EXCLUDED from generated block lists — a rule bound to an unverified token is a guess, and we don't guess.		 xAI retrieval reported not rule-bindable

Allowlist

User-agent substrings that always bypass the block — your uptime monitor, partners, scrapers you hired. One per line or comma-separated.

Honest scope: HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.