← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

What Can Robots.txt Actually Do About AI Crawlers?

The short answer: robots.txt can declare your per-agent posture in the one file crawlers are expected to consult, and it is the required vehicle for token opt-outs like Google-Extended and Applebot-Extended. What it cannot do is enforce: it is a request, honoured voluntarily, which one documented fetcher may bypass when a user asks, and which CMS regenerations silently erase. AgentBlock generates the robots layer from your policy and enforces the same decisions at the edge or server — the file publishes, the kit enforces, and the audit keeps them agreeing.

Build my block kit — free · Test a policy in the simulator first

What is robots.txt for AI crawlers?

the site's robots exclusion file used to declare per-agent rules for AI crawlers — disallowing training agents by name and expressing token opt-outs such as Google-Extended. It is an advisory standard: compliance is voluntary, so AgentBlock treats it as the published layer of a policy whose enforcement lives in the generated Worker or plugin.

The file's real powers

Robots.txt has three genuine strengths. It is universal — every serious crawler vendor documents their agent's robots behaviour, so a disallow is a communication the counterparty has agreed to read. It is expressive per agent: separate user-agent groups let you disallow GPTBot while saying nothing about retrieval agents, exactly the split a modern policy needs. And it is the designated vehicle for the token opt-outs: Google-Extended and Applebot-Extended live in this layer, with no other mechanism offered by their vendors.

It is also cheap evidence of posture. A dated robots file that names the training class tells partners, acquirers and regulators what your position was — a declaration, publicly filed, that other records can corroborate.

The advisory limit, with a named exception

Nothing in the protocol makes a crawler comply. Well-run vendors honour robots because reputation is their licence to crawl; the long tail of scraping does not consult it at all. And the exception is not only among the disreputable: Meta documents that meta-externalfetcher may bypass robots because the fetch is user-initiated. A robots-only policy therefore has, by design, a hole on one of the fastest-growing traffic classes.

The mature reading: robots compliance is a property of the crawler, not of your file. Your file sorts counterparties into those who accept your terms at the door and those who must be refused at it — both outcomes need machinery, and only one of them is the file.

Drift: the failure mode nobody notices

Robots files are regenerated constantly — by SEO plugins, CMS migrations, marketing contractors adding a sitemap line. Each regeneration can drop hand-added AI rules without a trace, and because a missing disallow produces no error anywhere, sites routinely believe they have a robots posture they lost months ago. This is the single most common AI-policy failure AgentBlock's audit finds.

The structural fix is to make the robots layer a generated artifact of the policy, not a hand-edited file: your AgentBlock policy generates its robots rules alongside the Worker or plugin, and the enforcement-gap audit compares the live file against the policy and reports drift as a finding. Hand edits become unnecessary, so regeneration stops being a threat.

Layering robots inside a real policy

The composed stack, from declaration to enforcement: robots.txt publishes per-agent rules and the token opt-outs. The edge Worker or server plugin refuses blocked agents regardless of their robots manners. Path rules fence private subtrees even for allowed agents. Rate caps bound volume. The canary detects structural mapping. Each layer covers a failure the others structurally cannot, and the whole is generated from one policy so the layers cannot quietly disagree about the decisions.

Owners sometimes ask which single layer to keep if budgets force a choice. The enforcing layer, every time: a refusal that happens beats a declaration that hopes. Robots earns its place as the second layer — the public record and the token vehicle — not as the wall.

Reading your current file critically

Fetch your robots.txt today and interrogate it. Which agents are named? Are the two Extended tokens present if your posture requires them? Does any blanket rule — a wildcard disallow written for an old SEO scare — contradict the policy you think you run? Is a plugin regenerating the file, and would your AI rules survive its next update? Most owners find at least one answer they did not expect, which is the audit's entire value proposition in manual form.

Then move the answers into a policy that generates the file. A decision recorded once, in the builder, outlives every regeneration; a decision typed into a text file is one plugin update from folklore.

Five actions to take today

  1. Inventory your live robots.txt. List every named agent, check the two Extended tokens, and hunt for blanket rules that contradict your real policy.
  2. Find out who regenerates it. If a plugin or CMS owns the file, hand-added AI rules are living on borrowed time.
  3. Generate robots from policy. Let the kit produce the robots layer from the same decisions as the enforcing artifact.
  4. Enforce the decisions elsewhere. Worker or plugin refusals cover the crawlers — documented and otherwise — that robots cannot bind.
  5. Audit for drift on a schedule. The enforcement-gap audit compares live robots and meta tags against policy; run it after every stack change.

Worked example: the opt-out that a plugin update erased

Juniper & Vale, a fictional stationery brand, added careful AI rules to its robots.txt in January — training class disallowed, both Extended tokens present. In April its SEO plugin shipped an update that regenerated the file for a new sitemap format. Nobody noticed; nothing errored. The training disallow and both opt-outs were simply gone.

The loss surfaced in July, when AgentBlock's enforcement-gap audit — run as part of a kit deployment — compared the live robots file against the intended policy and reported all three rules missing, with the drift dated by the plugin's changelog to the April update. Six months of assumed posture had been three months of reality and three months of folklore.

The brand's fix was structural: the robots layer is now generated from its AgentBlock policy and re-audited after every plugin update. The audit takes under a minute; the lesson it encodes is permanent.

Frequently asked questions

Do all AI crawlers honour robots.txt?

The major vendors document robots controls for their agents and generally honour them — it is the basis of their licence to crawl. But compliance is voluntary, Meta documents a possible bypass for its user-triggered fetcher, and the long tail of scraping ignores the file entirely. Enforcement at the edge or server is what makes a decision unconditional.

Where do Google-Extended and Applebot-Extended go?

In the robots layer — they are policy tokens expressed through robots rules rather than fetching crawlers. AgentBlock includes them in the generated robots rules whenever your policy blocks them, and the audit verifies they are present in the live file.

Can robots.txt block AI crawlers from specific folders only?

Yes, disallow rules are path-scoped per agent group, and that granularity is useful for declarations. Remember it inherits the same advisory limit: for private paths you actually care about, the enforcing layer's path rules are the control that refuses requests.

How often should I check my robots.txt?

After every CMS, plugin or stack change, and quarterly otherwise. Better: make the check mechanical. AgentBlock's enforcement-gap audit diffs the live file against policy on demand, which converts a forgettable chore into a one-click verification.

Is a missing robots rule a security incident?

It is a posture incident: decisions you believed in force were not. Treat it as drift to correct and learn from — generate the layer from policy, audit on schedule — rather than as a breach; no system failed except the assumption that a text file maintains itself.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back