← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 6 min read
Incident: How a Stale Kit Missed Meta's New Crawler Wave
The short answer: a kit built before October 2026 contains no Meta-ExternalAgent decision at all — the entry's documentation wave, the Muse watch entry and the restructured Meta family arrived with database v2026-10-18. An un-refreshed kit does not malfunction; it faithfully enforces a world that no longer exists, while its owner believes 'Meta' is handled. The stale check exists to close exactly this gap: it diffs your kit's database version against the current one and lists what your policy has never seen.
Build my block kit — free · Test a policy in the simulator first
What is a staleness incident?
a crawler-policy failure in which the agent population changes — new documented agents, new watch entries, revised evidence — while a deployed kit continues enforcing its older database snapshot, leaving new agents without decisions and owners with a false sense of coverage.
The month the family changed
October 2026 was the crawler world's busiest month on record in this database's short life. Meta's crawler documentation — updated that year — laid out a five-agent family where many owners' mental model held a single 'Facebook crawler'. Reporting around the October 30 Commercial Terms clause pushed Meta-ExternalAgent into the news. Meta's Muse agent and xAI's Grok were exercising the industry's no-name problem in public, Amazon's contract-based Muse block among them. ChatGPT Atlas arrived in the same wave. One database version, v2026-10-18, absorbed all of it: new verified entries, three watch entries, revised notes throughout the Meta section.
Every kit generated before that version is a photograph of a smaller world. The photograph is not wrong about what it shows. It simply does not contain the new arrivals — and a policy cannot refuse an agent it has never heard of, except by class defaults it only applies at generation time.
What 'no decision' looks like in production
Nothing — which is the incident's shape. A new training agent on an un-refreshed site is fetched, served and logged as an unknown or unmatched name, indistinguishable in a casual log read from any other new bot. The owner's documents, meanwhile, say 'AI training blocked', a sentence that was true when written. Staleness is the only failure class in this series where the policy, the artifact and the logs all agree with each other and are jointly wrong about the world.
The tell, for owners who read closely, is the UNKNOWN share: new agents arrive inside it. A rising UNKNOWN trend in the monthly evidence — the signal the watch-entries guide teaches — is often staleness announcing itself behaviourally before any version check is run.
Nine days versus a quarter
The recovery is the stale workflow from the stale-kit guide: run the check, read the diff as decisions (each new verified entry inherits a class default at regeneration — confirm the default is your choice for that entry), export the signed policy, regenerate, verify, file the diff. The elapsed exposure is bounded by one variable only: how quickly the check is run after the version lands. Fern & Field's nine days, from the worked example there, came from running the check the week of release. Sites without the habit discovered the October wave in the next quarter's review — or in a journalist's question.
The WordPress plugin's stale-kit cron exists to compress that window for the sites least able to compress it themselves: the flag waits in the admin for the next login, turning 'we forgot' into 'we were told'.
Making versions boring
The mature posture treats database versions the way software teams treat dependency updates: expected, scheduled, unremarkable. A quarterly line in the calendar — stale check, verifier, gap audit, evidence read — absorbs most versions without ceremony, and a major wave like October's justifies an off-cycle run of the same line. What the posture refuses is the alternative biography, in which a crawler policy is installed once, with pride, and then slowly becomes a historical document while continuing to be cited as a control.
File the diffs. A run of dated database diffs beside the kit hashes is the complete answer to the diligence question 'how do you keep this current?' — asked, eventually, by every serious counterparty a growing business meets.
Five actions to take today
- Run the stale check on a calendar, not on a rumour. Quarterly minimum, plus an off-cycle run whenever the crawler world has a loud month.
- Read the diff as decisions. Every new verified entry will inherit a class default; confirm each default is a choice you would defend.
- Watch UNKNOWN as the behavioural tell. A rising unidentified share often means the world added agents your kit predates.
- Signed-export before regenerating. The refresh stays reversible, and the old-versus-new diff becomes the audit record.
- File version history with kit hashes. Dated diffs are the standing answer to 'how is this kept current?'
Worked example: two sites meet the same October
Two fictional retailers, one database wave. Fern & Field ran its stale check the week v2026-10-18 landed: the diff showed 6 new or materially revised entries, two of which changed live decisions, and the refreshed kit shipped nine days after the version — exposure window, nine days, documented to the day.
Harbor & Pine, running a kit from the previous version with no check on its calendar, met October in January, when a trade article prompted the question 'are we blocking that Meta crawler?'. Its logs then showed Meta-ExternalAgent served 7,300 times across the quarter — every fetch lawful under a kit that had simply never heard the name.
Same wave, same tooling available, a quarter's difference in outcome — produced entirely by whether a one-minute check had a date in a calendar. That is the whole argument of this incident, and it is not really about Meta.
Frequently asked questions
How do I know which database version my kit uses?
It is recorded in the policy itself — database_version travels with every generated kit. The stale check compares it against the current version automatically; you do not need to find it by hand, only to run the check.
A new agent appeared — am I unprotected until I regenerate?
For that agent, your kit holds no decision, so its treatment is whatever your stack does with unmatched names — usually, service. Class-level protection for new arrivals only enters at regeneration, which is why the check-to-refresh interval is the exposure metric that matters.
Does the stale check change anything by itself?
No. It reports the version comparison and the database diff. Regeneration remains a human decision — the diff may contain entries you want to decide differently from their class defaults, and that glance is the owner's job the check exists to schedule.
Can the check run automatically?
On WordPress, the generated plugin's stale-kit cron flags staleness in the admin on a schedule. Worker deployments use the hosted check. Either way the flag is a prompt, not an action — the refresh still passes through the signed-export and verify steps.
What if I skip a version or two?
The diff accumulates: the stale check compares your version against the current one across however many versions separate them, and the review takes proportionally longer. Skipping is recoverable; never checking is the incident.
Related guides
- How to Block AI Scrapers and Bots from Your Website
- Which AI Crawlers Should You Allow?
- How Do You Block AI Bots on WordPress, Cloudflare, or Plain JS?
- Pre-deploy breakage simulator — test a policy before it ships
- Impact preview — your server log against a draft policy
- What Are the Most Common AI Blocklist Design Mistakes?
- Per-Agent Policy or Block All AI Bots: Which Is Right?
- How Do You Test a Block Policy Before Deploying It?
- How Do You Preview a Block Policy Against Your Real Server Log?
- What Is an Enforcement Gap Audit for AI Crawler Policy?
- Block evidence — parse a log, build an evidence pack
- The dated agent database — 25 entries, graded