← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 6 min read

How Should a Creator or Portfolio Site Handle AI Crawlers?

The short answer: protect the work, stay bookable. Block the training class — a portfolio's images and writing are the creator's style in ingestible form, and consent to train on them should never be a default. Allow the major retrieval agents — 'who should I hire for…' is now an answer-engine question, and citations are how an unknown creator gets into it. Keep facebookexternalhit sacred, because a creator's shares are their livelihood's circulatory system, and fence nothing you cannot name: a one-person site's policy should fit on an index card and still be running in a year.

Build my block kit — free · Test a policy in the simulator first

What is a creator crawler policy?

an AgentBlock policy for a portfolio or creator site: the training class blocked to keep the body of work out of datasets without consent, retrieval agents allowed so answer engines can cite and recommend the creator, and the link-preview fetcher protected as the site's primary distribution mechanism.

The creator's version of the trade

A portfolio is a strange asset: it must be maximally visible to be worth anything, and its visibility is exactly what makes it ingestible. Training crawlers do not experience the distinction between 'look at my work' and 'learn my style' — a systematic crawl takes both. The creator's objection to training is therefore not the publisher's licensing abstraction; it is closer to a musician's objection to a free sample library of their own playing. Blocking the class states the industry-standard answer in the only language crawlers honour: a refusal at the door.

Retrieval is the opposite relationship. When someone asks an AI for an illustrator in a style, a photographer in a city, a designer for a niche, the answer is assembled from fetchable portfolios. The creator who blocks retrieval has not protected the work; they have resigned from the recommendation.

Small site, simple policy, no apology

A one-person operation should run the kit's strengths at their simplest: the default policy — 20 blocked, the preview fetcher allowed — with retrieval flipped on for the major agents after a first month of reading the impact preview, the free Worker deployed, and caps at the generous starter settings. The whole estate is one generated artifact; there is no plugin to babysit unless WordPress is home, in which case the plugin's own stale-kit cron becomes the maintenance department.

Resist imported complexity. Lists copied from creator forums carry the full mistake catalogue — phantom tokens, family substrings, the preview fetcher blocked by someone whose audience was not built on shares. The graded database exists so a busy creator does not have to become a crawler scholar to be correctly configured.

Images, text and honest limits

Most of a visual portfolio's value sits in images, and images are what training crawlers enumerate most thoroughly — every image URL is a fetchable asset with the page's context attached. The name block governs the honest crawlers' access to all of it, page and image alike, because the Worker refuses the request before either is served.

The limits deserve the same plain speech the kit uses. A determined scraper under a browser name is a behavioural problem, not a policy one; hotlinking and reposting by humans are different wrongs with different remedies; and nothing about crawler policy recalls work already ingested. What the policy does — completely, verifiably — is close the named, honest, industrial-scale door, which is where the volume was.

Shares are the business

Creators distribute by being shared, and the share card is the unit of that economy. Every control decision must be audited against one question: does the link still render beautifully when a stranger pastes it into a chat? That is facebookexternalhit's daily work, and the creator policy treats it with unusual ceremony — allowed by default, allowlisted in writing, share-tested on the newest piece after every change, because a creator's launch day is precisely when a silent preview failure costs the most and is noticed last.

The same lens catches path-rule ambitions: a creator site needs few fences — a client-proofing area, an admin path — and each one should be checked against the share test before it ships, in the simulator, where embarrassment is free.

Five actions to take today

  1. Block the training class, on principle, in writing. Your style is not a default dataset; consent, if ever given, will be a deliberate act.
  2. Allow the major retrieval agents. Recommendations are assembled from fetchable portfolios; be in the answer when the question is your niche.
  3. Run the simplest enforceable artifact. The free Worker, generated once and verified — a policy a one-person business can actually maintain.
  4. Treat previews as revenue. facebookexternalhit allowlisted, and a share test on your newest piece after every policy change.
  5. Ignore forum lists. Graded, vendor-published tokens only — the database's discipline is the time-saving feature for a busy creator.

Worked example: an illustrator's first year under the kit

Mara Voss, a fictional illustrator with a 220-piece portfolio site, deployed the default policy on the free Worker in an afternoon, then flipped four retrieval agents to allow after her first impact preview showed 310 retrieval fetches against 9,800 training fetches in her first month.

Across the year, the kit logged 118,000 training refusals. Enquiries attributed to AI answers — 'the assistant described your editorial work' is a sentence her inbox now contains — produced 6 commissions she counts as the channel's first-year yield, against total crawler-policy effort she estimates at three hours, including two stale-check refreshes.

Her preview discipline survived its own test in October: a regenerated kit after a database update, a share test on her newest series, cards rendering perfectly in the group chat where half her commissions begin. The policy's index-card summary, in her words: 'Training: no. Answers: yes. Shares: always.'

Frequently asked questions

I am a photographer — does blocking training protect my images?

It refuses the named training crawlers' requests for your pages and the images they serve, at the edge, before anything is delivered — the industrial-scale ingestion door, closed and logged. It cannot recall already-ingested work or govern human reposting, and the kit says so rather than promising a wider magic.

Won't blocking crawlers make me invisible?

Blocking training does not touch search or answers. Retrieval agents — the citation channel — are a separate decision you are keeping open, and classic search crawlers are not in the database at all. The policy removes you from datasets, not from discovery.

I cannot afford tools — is the free tier enough?

For a portfolio site, usually yes: policy building, the database and the Cloudflare Worker download are free, and that combination is a complete, enforced, verifiable policy. Pro adds the WordPress plugin and JS snippet downloads for sites on those stacks.

A client sends me work through my site — what do I fence?

Client proofing, download and admin areas: the paths where unpublished client work lives. Fence them for all agents, keep them authenticated as you already do, and let the public portfolio stay wide open beside them.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back