← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 6 min read

Incident: When the JS Snippet Met an Agent That Disables JavaScript

The short answer: nothing was blocked, because the enforcement was happening — if at all — inside the visitor's browser, and crawlers do not bring a browser's obedience with them. AgentBlock's plain-JS snippet is labelled, in its own header, as a deterrent and notice layer: any agent that disables JavaScript bypasses it completely, which is to say nearly every serious crawler by default. Sites that need refusals must deploy at the edge (Worker) or the server (plugin). The snippet's honest jobs are notice, deterrence of casual copying, and stacks with no other deployable layer.

Build my block kit — free · Test a policy in the simulator first

What is a snippet-only incident?

a crawler-policy failure in which a browser-side JavaScript snippet is relied on as enforcement, while the targeted agents — which fetch pages without executing JavaScript, or with it disabled — are unaffected, leaving the owner with a policy that exists only for visitors who were never the problem.

The physics of the mistake

A JavaScript snippet is code delivered with the page, executed by the visitor's client, on the visitor's terms. For a human's browser those terms are mostly cooperative. For a crawler they are a menu it may decline entirely: fetch the HTML, parse it locally, never run a script. GPTBot's database entry notes its JavaScript support is limited; many retrieval and scraping clients run none; and a scraper that did run scripts would simply disable the one that inconveniences it. Enforcement that the enforcee can switch off is not enforcement — it is a request rendered in code, standing one layer below robots.txt in actual authority while looking, in a dashboard, considerably more impressive.

That last part is the trap. A snippet produces visible artifacts — a console notice, a client-side log entry for cooperative visits — that feel like a working control. The feeling is the incident's engine: five months of confidence, in the anchoring example, built on signals generated exclusively by visitors who were never the target.

What the log should have shown

The giveaway is an absence with a shape: named crawler traffic in the server log, serenely continuing at full volume on pages the snippet supposedly refused, month after month. Server logs do not know about client-side theatre; they record requests and responses, and a crawler served a 200 with full content was, by definition, not blocked. The incident resolves the first time someone compares the snippet's claims against the server's record — the evidence habit again, arriving late but decisively.

Owners inheriting a stack should run that comparison for every control, not just this one: pick the claim, find the server-side record, count. Controls that cannot produce a refusal in the server's own record are decorations, whatever their dashboards say.

What the snippet is actually for

The snippet has three honest jobs, and AgentBlock prices it as a Pro download for stacks that need them. Notice: it presents your policy, in-page, to cooperative agents and to the humans building them — a real function in a ecosystem that runs partly on norms. Deterrence: it raises the cost of the laziest copying, the right-click-and-scrape tier that does run a browser. And last resort: on locked website builders that permit no edge config and no plugins, a labelled deterrent is better than nothing — provided the policy document states, in writing, that enforcement is absent and migration is the fix.

Notice what is absent from the list: stopping crawlers. The kit's snippet header says enforcement belongs at the edge or the server. A vendor whose snippet claims otherwise is telling you, in advance, how their other claims should be weighted.

The migration that ends the class

Sites graduate from snippet-only in one of two moves. Add Cloudflare in front and deploy the free Worker — the smallest step to real refusals, and the one most locked-stack sites can take without changing hosts. Or move the enforcing surface into the stack: the WordPress plugin for WordPress sites, the synced robots plus gateway rules elsewhere. The policy does not change in either move — the same graded decisions compile to the stronger artifact — which is the payoff of the one-policy architecture: upgrading enforcement is a regeneration, not a rethink.

File the incident with the comparison that exposed it (server-side counts versus the snippet's claims) so the next stack decision starts from evidence. The sentence to write in the policy document is the kit's own: deterrent layer in place; enforcement layer named and dated as the goal.

Five actions to take today

  1. Test every control against the server log. If named crawlers are still being served full pages, the control is theatre regardless of its dashboard.
  2. Label the snippet honestly in your policy. Deterrent and notice layer; enforcement absent; migration named as the fix.
  3. Deploy one real layer. Free Worker at the edge or the plugin on the server — one afternoon converts requests into refusals.
  4. Keep the same policy across the move. Regenerate the stronger artifact from the existing record; do not re-derive decisions.
  5. File the exposure honestly. The months of snippet-only operation belong in the incident record, quantified from the server log.

Worked example: five months of confidence, one log query

A fictional course creator, selling from a hosted website builder, installed a JS blocking snippet in March and reported 'AI crawlers blocked' in her launch retrospective. The snippet's console notice displayed beautifully — to her, in her browser, on every visit she made to test it.

In August, preparing a licensing conversation, she pulled her server log for crawler counts and found the truth in one query: GPTBot had been served 31,000 full pages across the five months, ClaudeBot 17,000, neither diminished in any week. The snippet had executed only for visitors running browsers cooperatively — her own tests prominent among them.

She put Cloudflare in front of the same site that week and deployed the free Worker generated from a policy built in an hour. September's log showed the first refusals of her site's life: 5,900 training-class requests refused, counted in the server's own record. The licensing conversation, when it happened, used those numbers.

Frequently asked questions

Why do crawlers not run my JavaScript?

Because fetching and parsing HTML is cheaper, faster and sufficient for their purposes — and because executing arbitrary site code is a cost and a risk a crawler operator declines by default. GPTBot's own entry notes limited JS support; scraping clients typically run none. A control that requires the crawler's cooperation is a request, not a refusal.

Is the snippet useless, then?

No — it is mislabelled when called a blocker. As notice to cooperative agents, as deterrence to casual browser-based copying, and as the only deployable layer on locked builders, it does real work. AgentBlock sells it with its limits printed in its own header, which is the standard to hold every vendor to.

How do I check if my current 'blocker' is real?

One query: in your server log, are the named agents you believe blocked still receiving 200 responses with full content? If yes, nothing is being refused at any layer that matters. Then identify which layer your control actually executes in — browser, server or edge — and you will know what you own.

What is the cheapest real enforcement?

The free Cloudflare Worker: put Cloudflare in front of the site if it is not already, paste the generated template, add the route. Policy building and the Worker download are free, and the refusals begin appearing in the server-side record immediately.

Can the snippet and a real layer coexist?

Yes — owners keep the snippet for its notice function after deploying a Worker or plugin, generated from the same policy. The rule is about reliance, not presence: never let the snippet be the only thing standing in front of a decision you care about.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back