← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 7 min read
How Do You Block Bytespider, ByteDance's Training Crawler?
The short answer: you block the Bytespider user-agent like any training crawler — it is a verified training-class entry in AgentBlock's database — and, because Bytespider's reputation is about volume as much as purpose, you pair the block with the kit's rate caps so that even lookalike or follow-up traffic is soft-blocked past a per-minute and per-day budget. The name block answers the training objection; the caps answer the load objection. They are different problems and the kit treats them separately.
Build my block kit — free · Test a policy in the simulator first
What is Bytespider?
ByteDance's data-collection crawler for its AI products, a verified training-class entry in AgentBlock's database (v2026-10-18), listed from reputable bot directories after no official ByteDance documentation page was found as of October 2026. It is widely reported for aggressive crawl rates.
Two objections, often confused
Site owners raise two distinct complaints about Bytespider. The first is principled: its fetches collect data for ByteDance's AI products, and owners who decline training use decline this too. The second is operational: Bytespider has a long-standing reputation for crawling hard — high request rates, deep systematic coverage — which on a small site shows up as log noise, bandwidth and occasionally real load on shared hosting.
Keeping the objections separate matters because their remedies differ. A name block fully answers the first for every request that honestly carries the name. The second is answered by rate caps, which govern behaviour regardless of what a request calls itself: past the budget, the Worker returns a RATE-CAP soft-block instead of the page.
The evidence grade, stated plainly
AgentBlock's database is candid about this entry: Bytespider's listing rests on reputable bot directories, with no official ByteDance crawler documentation page found when the database was researched in October 2026. It still carries the verified grade because the user-agent token is consistently documented across those directories and matches observed traffic — but the note stays on the entry, and owners deserve to see it before building on the name.
That candour is the database's design, not an embarrassment. Entries carry sources and last-verified dates precisely so that a policy can be defended: 'we block the token Bytespider, on this evidence, since this date' is a sentence an owner can say to a client, a partner or an auditor without bluffing.
Rate caps as the volume answer
Rate caps in an AgentBlock policy are per-agent budgets: a per-minute and a per-day allowance, after which further requests are soft-blocked. For a crawler you have decided to allow but not to feed unlimitedly — or for the tail of an aggressive crawler's behaviour under variant names — caps convert an argument about intent into an arithmetic rule your edge enforces without judgement calls.
Caps also protect against your own mistakes. If a block rule is mis-keyed during a stack change, a Bytespider-class crawler hits the cap rather than your origin's patience, and the soft-block events in the log tell you something is fetching at volume while you find the rule error. Load control and policy control back each other up.
Robots manners and the enforcement layer
Bytespider's reputation includes loose adherence to robots.txt on some sites' experience, which is the general lesson in miniature: robots rules are requests. For any agent whose manners you doubt, the decision must live where refusals are real — the Cloudflare Worker at the edge or the WordPress plugin on your server. The plain-JS snippet is a deterrent layer and should never be the only thing standing in front of a crawler you actually object to.
Deployed at the edge, the block also produces clean evidence: structured events with agent, path, verdict and IP. When the question 'is the Bytespider block even working?' arises — and with this crawler it always arises — the log, not the intention, answers it.
Bytespider inside the training set decision
Bytespider rarely deserves a solo policy. If training use is your objection, it is one of the nine training entries and belongs in the class decision alongside GPTBot, ClaudeBot, the tokens and CCBot. Owners sometimes block Bytespider alone for load reasons while allowing other training crawlers — that is coherent too, as long as the policy note says the decision was about volume, not principle, so a future reviewer does not misread the posture.
Either way, re-run the impact preview before shipping a change: it replays your draft policy against your own recent log lines and shows, per agent, what would have been blocked, capped or allowed. For a high-volume crawler, that preview is the difference between a policy you understand and one you merely hope about.
Five actions to take today
- Name the real objection. Training use, crawl volume, or both — write it down; the remedies are different controls.
- Block the Bytespider token. Set it in the builder as part of the training-class decision, on edge or server enforcement.
- Add rate caps. Give aggressive agents a per-minute and per-day budget so behaviour past the budget is soft-blocked regardless of name variants.
- Skip the JS-only layer. Never let the plain-JS snippet be the sole control in front of a crawler whose robots manners you doubt.
- Read the first week's log. Confirm BLOCK and RATE-CAP events appear for Bytespider-class traffic, and that legitimate agents are untouched.
Worked example: a shared-hosting shop tames a crawl flood
Petal & Post, a fictional florist on shared hosting, found its site timing out on Saturday mornings — its busiest trading window. The logs told the story: in one 30-day month, Bytespider-named requests totalled 96,300, peaking at 4,100 requests in a single hour, against the shop's total human traffic of about 19,000 visits in the same month.
The shop blocked the training class including Bytespider and set conservative rate caps across the remaining agents. In the first full month under the kit, Bytespider-name requests fell to 7,840 refused attempts with zero pages served, RATE-CAP soft-blocks absorbed another 1,260 over-budget requests from agent-class traffic, and Saturday timeout incidents went from five in the baseline month to none.
The owner's policy note records both halves honestly: the block was for training use, the caps were for load, and the shared-hosting bill — not any principle — is what would reopen the question.
Frequently asked questions
Does Bytespider ignore robots.txt?
It has that reputation in many site owners' logs, which is why AgentBlock's guidance is to enforce any decision you care about at the edge or server rather than relying on robots.txt alone. Robots rules remain useful as a published statement of your policy; enforcement makes it real.
Why is Bytespider in the database if ByteDance publishes no docs page?
The entry records its own evidence: reputable bot directories consistently document the token and it matches observed traffic, while the database notes plainly that no official ByteDance documentation page was found as of October 2026. Owners see the note on the entry before deciding.
What is a RATE-CAP soft-block?
When an agent exceeds its policy budget — a per-minute or per-day cap in your kit — the Worker refuses further requests with a RATE-CAP verdict instead of serving pages. It is a behavioural limit that applies even when you have not blocked the agent outright.
Should I block Bytespider if I allow other training crawlers?
Only if your objection is its crawl behaviour rather than training in principle. The two postures are both coherent; what matters is recording which one you chose, because a future reviewer will otherwise assume the training-class position is inconsistent.
Can rate caps hurt real customers?
Caps in AgentBlock policies apply to agent-class traffic identified by the policy, not to ordinary browser visitors. Set them generously for agents you allow, and use the impact preview against your real log to see the effect before anything ships.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- How Do You Block GPTBot Without Losing ChatGPT Citations?
- How Do You Block ClaudeBot and Keep Claude Citations?
- How Do You Block CCBot and the Common Crawl Training Feed?
- Google-Extended vs Googlebot: How Do You Opt Out of AI Training Only?
- How Does Applebot-Extended Opt You Out of Apple Intelligence Training?
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded