← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read
How Do Rate Limits Work for AI Crawlers?
The short answer: a rate cap gives a named agent a budget — so many requests per minute, so many per day — and refuses the excess. In an AgentBlock policy, caps are set per agent in the builder; the Cloudflare Worker enforces them and logs over-budget refusals as RATE-CAP soft-blocks. Caps are the middle control between block and allow: they suit agents you accept in principle but not at unlimited volume, and they double as a load backstop when a crawler's behaviour changes faster than your policy.
Build my block kit — free · Test a policy in the simulator first
What is an AI crawler rate cap?
a per-agent request budget in an AgentBlock policy, expressed as a per-minute and a per-day allowance. Requests within budget are served under the agent's normal decision; requests past it are soft-blocked by the Worker with a RATE-CAP verdict recorded in the log.
Why a third control exists
Block and allow are verdicts about identity. But some problems are about quantity: a crawler you welcome at 200 requests a day is a load incident at 200,000; a retrieval agent whose fetches track real questions behaves nothing like one stuck in a retry loop hammering your search endpoint. A policy vocabulary with only yes and no forces those situations into the wrong box — usually a resentful total block of an agent you actually valued.
Caps are how grown-up traffic management has always handled this, and AgentBlock's policy format builds them in as first-class fields: rate_caps per agent, each with a per-minute and a per-day figure, carried into the generated Worker. Setting zero or leaving caps unset means no behavioural limit — the agent lives under its identity decision alone.
Anatomy of a soft-block
When an agent crosses its budget, the Worker refuses the request and logs a structured event: the agent's name, the path, the IP, and the verdict RATE-CAP — visibly different from a BLOCK verdict in your evidence. That distinction is operationally useful. A rising RATE-CAP line for an allowed agent is the earliest warning that its behaviour changed — a new crawl pattern, a customer of a scraping API aiming it at you, a buggy integration retrying.
Soft is the right word in one specific sense: the refusal applies to the over-budget excess, not the agent's identity. Tomorrow the daily budget resets; the next minute the per-minute budget does. An agent in temporary excess is throttled, not banned, and your policy does not need an emergency edit at 2 a.m. to restore a partner's access.
Setting numbers that mean something
Start from your own log, not from folklore. The impact preview counts each agent's recent requests; healthy baselines for most small sites run from dozens to a few thousand fetches a day per agent. Set the daily cap at roughly three to five times the observed baseline — high enough that a busy news day or a genuine surge passes untouched, low enough that a runaway is stopped before it matters — and the per-minute cap at a level no legitimate crawl pattern needs, often a tenth of the daily figure.
Then read the first month's RATE-CAP events as calibration data. Caps that never bind on agents you distrust may be decoration; caps that bind weekly on a retrieval agent are quietly taxing your citations, and the count of refused fetches is the tax receipt. Adjust with the numbers in front of you.
Caps as the spoofing backstop
Name-based policy has an honest limit — a request under a stock browser user-agent is not matched by any agent rule. Caps do not fully close that hole either, since they key on identified agents. But the Worker pairs caps with the sender gate for Pro policies: requests claiming a known agent's name from outside its vendor's IP ranges draw a SPOOF-SUSPECT verdict, and path rules plus caps fence the highest-value paths behaviourally. Defence in depth is not a slogan here; it is the specific stack of named rules, sender checks, budgets and fences, each covering a failure the others miss.
This is also the honest way to talk to clients about crawler control. 'We block the named training crawlers, cap everything allowed, fence the private paths, and log the verdicts' is a defensible sentence. 'We stop all scraping' is not, and AgentBlock's kits refuse to print it — the bypass disclaimer ships in every policy.
Caps for agents you have blocked anyway
It sounds redundant and is not. If a stack change ever drops your block rule — a Worker route misconfigured during a migration, a plugin deactivated — caps are the second line that keeps a blocked-class crawler from feasting during the gap. The stale check and kit verifier exist to find such gaps, but gaps are found on a schedule; caps bind in the meantime. Controls that assume every other control works are not controls; they are hopes.
Set the caps while you build the policy, in the same sitting, when the baselines are on screen. Retrofitting them later requires the log archaeology you will otherwise have already done once.
Five actions to take today
- Baseline from your log first. Run the impact preview to get each agent's real daily and peak-minute volumes before choosing numbers.
- Set daily caps at 3–5× baseline. Room for genuine surges, a hard stop for runaways — per agent, not one global figure.
- Set a sane per-minute cap. No legitimate crawl pattern needs burst rates anywhere near a load incident; the minute cap is the flood gate.
- Read RATE-CAP events monthly. They are calibration data: which agents changed behaviour, and whether allowed agents are being taxed.
- Cap even blocked agents. Budgets are the second line when a block rule is ever lost in a stack change.
Worked example: a news site caps a retrieval surge
The Harbour Gazette, a fictional regional news site, allows retrieval agents — citations in answers send it readers. Its baseline was calm: PerplexityBot averaged 640 fetches a day, Claude-SearchBot 410. Caps were set at 3,000 a day for each, with a 120-per-minute burst limit, and forgotten about for a quarter.
During a storm story that the Gazette broke first, PerplexityBot fetches hit the daily cap at 3,000 by mid-afternoon — real reader questions, arriving faster than the baseline ever suggested. The RATE-CAP line in the log made the event visible the same evening rather than at month-end, and the editor raised the cap to 8,000 for news agents with the story still live.
The quarter's lesson, recorded in the policy note: caps did not fail that day; they reported. The refusal count was the signal that demand had changed, priced in a unit — 2,900 refused fetches — that the next cap decision could use directly.
Frequently asked questions
Do rate caps apply to human visitors?
No. Caps in an AgentBlock policy are keyed to identified agent traffic under the policy. Ordinary browser visitors are not agents in the database and are not budgeted by the kit.
What happens when an agent hits its cap mid-crawl?
Further requests from that agent are refused with a RATE-CAP verdict until the window resets — the next minute for the per-minute budget, the next day for the daily one. The agent is throttled, not banned, and every refusal is logged with agent, path and time.
How should a tiny site set caps without much log history?
Start generous: daily caps in the low thousands per agent and a per-minute cap around a hundred will never trouble a legitimate crawl on a small site, but will stop a runaway cold. Tighten later from the RATE-CAP evidence, not from anxiety.
Can caps replace blocking a training crawler?
They answer different objections. A cap still serves the crawler your content up to the budget — pointless if your objection is training use itself. Block for principle, cap for volume, and use both where both objections are real.
Where do I see cap events?
In the same structured evidence as blocks: the Worker's logged events and the WordPress plugin's block log both record the verdict per request, and the evidence tooling aggregates them per agent for review.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- How Can You Tell a Spoofed AI Crawler from a Real One?
- Which AI Agents Cannot Be Blocked by Name?
- Why Doesn't AgentBlock Block the Reported GrokBot Token?
- What Can Robots.txt Actually Do About AI Crawlers?
- How Do You Design an Allowlist for AI Agents?
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded