← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read

How Can You Tell a Spoofed AI Crawler from a Real One?

The short answer: by checking the sender, not the string. A user-agent is a claim; the request's IP range is evidence. AgentBlock policies carry dated snapshots of known vendor IP ranges, and the Pro Worker's sender gate compares a claiming agent's origin against them: a request that wears a known agent's name from an implausible network is logged SPOOF-SUSPECT instead of being treated as the genuine agent. That one distinction keeps your block evidence honest and your allow decisions safe from impersonators.

Build my block kit — free · Test a policy in the simulator first

What is a SPOOF-SUSPECT verdict?

an AgentBlock Worker verdict for a request whose user-agent claims a known agent's identity but whose sender does not match the vendor's known IP ranges in the policy's dated snapshot. It marks the claim as unverified — neither credited as the real agent nor silently treated as one.

The name is a claim anyone can type

User-agent strings are self-declared. Setting a script's agent name to GPTBot takes one line of code, and the population with a motive to do it is real: scrapers borrowing a respected crawler's name to slip past allowlists, and — Common Crawl's own warning — impostors wearing CCBot's name for purposes that are not Common Crawl's. Any policy that treats the name alone as identity will be lied to at exactly the moments it matters most.

This is why AgentBlock's database distinguishes the token from the traffic. The database governs names; the sender gate governs claims. When the two disagree, the honest output is not BLOCK or ALLOW — it is a third verdict that says the identity is in question.

How the sender gate works

Each generated policy embeds a snapshot of known vendor IP ranges, stamped with its snapshot date — the policy literally carries vendor_ip_ranges and ip_snapshot_date fields, so the evidence always states which snapshot judged it. In the Worker, a request claiming a blocked-or-known agent's name is checked against the ranges for that vendor; a mismatch is logged SPOOF-SUSPECT with the claimed agent, path and IP, and refused on the policy's terms for unverified claims.

The snapshot dating matters. Vendor ranges change as clouds reallocate, and a verdict is only as good as the snapshot behind it — which is one more reason kits carry a stale check and policies get regenerated: sender intelligence ages like the agent database does, and the tooling treats both as perishable.

What spoofing is usually for

Read the pattern, not just the verdict. Spoofed claims cluster into a few shapes. Allowlist riding: claiming a utility or retrieval name you allow, to extract catalogue pages under its cover — the canary path often corroborates these, since the mapper underneath is structural. Block-testing: small numbers of claims probing whether your refusals are real. And noise: misconfigured tools and research scripts wearing famous names carelessly, which your log should forgive but still record.

The response differs per shape — fences and caps for riders, nothing much for noise — and none of the responses is available if your evidence flattened every claim into the genuine agent's counts. Spoof-suspect accounting is what lets a review distinguish 'GPTBot visited 40,000 times' from '40,000 requests claimed GPTBot; 6% failed sender verification'.

The limits of sender verification

Honesty section, as always. IP ranges are evidence, not proof: vendors expand ranges, resellers and cloud tenants complicate attribution, and a determined actor on rented infrastructure inside a plausible range can pass a sender check. The verdict name is deliberately modest — SUSPECT, not FRAUD — because the gate measures consistency, not intent. Policies should treat sender verification as one layer among name rules, caps, fences and canaries, which is how the Worker composes them.

What the gate does eliminate is the cheap lie — the datacenter script wearing a crawler's name from an unrelated network, which is the overwhelming majority of spoofed agent traffic small sites actually receive. Controls should be judged against the traffic that exists, not the adversary that can be imagined.

Keeping the snapshot fresh

Sender intelligence belongs on the same review cadence as the agent list itself. When the stale check flags your kit — because the database version moved — regenerate, and the new policy carries the current ranges with a new snapshot date. For agencies running the multi-site pack, one regeneration refreshes every site's snapshot together, which is the operational argument for the pack over hand-maintained per-site files.

File the snapshot date with your evidence exports. A SPOOF-SUSPECT count quoted without its snapshot date is an unfinished sentence; with it, a reviewer can judge exactly how much weight the verdicts bear.

Five actions to take today

  1. Treat names as claims. Build reviews on the three-verdict view — BLOCK, SPOOF-SUSPECT, CANARY — never on raw user-agent counts alone.
  2. Run a Pro Worker for sender checks. The sender gate ships in the policy-carrying Worker tier, using the vendor ranges embedded in your policy.
  3. Note the snapshot date. Every evidence export that cites SPOOF-SUSPECT counts should carry the ip_snapshot_date that produced them.
  4. Correlate suspects with the canary. Spoofed riders usually map structure; canary hits from the same senders confirm the pattern.
  5. Regenerate on stale flags. A moved database version means fresher ranges as well as fresher agent names — refresh both together.

Worked example: the GPTBot that was not

Alto Audio, a fictional retailer of studio equipment, allowed ChatGPT-User and OAI-SearchBot for citations and blocked the training class. Its June evidence review showed an anomaly: 3,150 requests claiming ChatGPT-User had drawn SPOOF-SUSPECT verdicts in May, fetching catalogue and specification paths in alphabetical order — a mapper's pattern, not a question-answering pattern.

Genuine retrieval traffic in the same month — sender-verified — was 480 fetches concentrated on review and comparison pages. The spoofed stream had been riding the allowed retrieval name to extract the catalogue for a price-comparison scraper, and only the verdict split made it visible as a separate fact.

Alto fenced its specification paths for unverified claims and tightened caps. Spoof-suspect fetches fell to 210 in June, genuine citations were untouched, and the retailer's policy note now states the rule its review adopted: an agent's name earns its policy treatment only when its sender agrees.

Frequently asked questions

Why would anyone spoof a crawler's user-agent?

To inherit its treatment. Sites allow retrieval and utility agents they would never allow generically, block some names and not others, and keep allowlists — every distinction is an incentive to wear the favoured name. Common Crawl warns of the same behaviour against CCBot specifically.

Does SPOOF-SUSPECT mean the request was malicious?

It means the claim was inconsistent: the name said one vendor, the sender's network said otherwise. Misconfiguration causes some of it, deliberate riding causes more. The verdict records the inconsistency; your review of its pattern decides what it meant.

How current are the vendor IP ranges?

Each policy embeds a dated snapshot — the ip_snapshot_date travels with the kit — and regenerating a kit on a stale-check flag refreshes it. Verdicts should always be read against their snapshot date, which is why the date is part of the policy record.

Can a spoofer pass the sender check?

A sufficiently resourced one can — rented infrastructure inside plausible ranges defeats range checks. The gate eliminates the cheap, common lie, and the remaining layers (caps, path fences, canary) address what passes. AgentBlock's bypass disclaimer, printed on every kit, covers the residual honestly.

Do spoofed requests count toward my per-agent statistics?

They are logged under their own verdict precisely so they do not pollute the genuine agent's counts. Your evidence can therefore show both numbers — genuine visits and suspect claims — which is the only honest way to report either.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back