← All guides · Published November 14, 2026 · Reviewed November 14, 2026 · By the Ingenious AI team · 6 min read

What Crawler Policy Should a SaaS Documentation Site Run?

The short answer: open the documentation, fence the application. A SaaS site's docs are its best answer-engine content — retrieval agents citing your docs are demonstrating your product inside someone else's question — so allow the retrieval class on docs and marketing paths. Training is a genuine choice: many SaaS vendors allow it on public docs for discoverability inside models, others block the class; decide it as a stated posture. The non-negotiables are the fences: /app, /billing, /login, /reset and API-console paths refused to every agent, and the preview fetcher protected for the links your users share in support chats.

Build my block kit — free · Test a policy in the simulator first

What is a SaaS crawler policy?

an AgentBlock policy for a software product site: retrieval agents allowed across documentation and marketing sections, a stated training posture applied to the nine training entries, and the application surface — app, billing, auth and console paths — fenced for all agents by path rule.

Docs are a citation asset, not a leak

Ask where a SaaS product gets discovered now: a developer asks an AI how to solve a problem, and the answer — built from fetched documentation — names the tools that solve it. Docs that retrieval agents can fetch are eligible for that answer; docs they cannot are not, and no marketing budget buys the placement back. The docs section's retrieval posture is therefore a growth decision wearing a crawler costume, and should be made by whoever owns growth, with the referral numbers in the room.

The content fits the channel unusually well. Documentation is factual, structured and versioned — exactly what answer engines quote accurately — and a citation sends a reader who arrives pre-sold on the approach, mid-task, with the problem live. SaaS analytics teams in these guides consistently find answer-referred trial starts converting at or above site average.

Training: the one real judgement call

Public docs in training data cut both ways, and honest SaaS owners land on both sides. For allowing: models that have absorbed your docs answer questions about your category with your concepts in their vocabulary — a diffuse, unmeasurable, real advantage. For blocking: your docs are a moat competitors' models should not ingest free, and the citation channel works without training consent since retrieval is a separate decision. The database supports either posture cleanly; what it does not support is drifting into one unexamined. Write the choice and its reason into the policy note, because the next review — after a competitor's launch, say — will want to know what was decided and why.

Whichever way the docs go, non-public material is not part of the question. Internal runbooks, staging docs and preview branches are fenced or authenticated, never merely unlinked.

Fencing the application surface

The app surface is where SaaS crawler policy earns its keep. /app and its subtrees, billing and subscription paths, login, signup-abuse-sensitive flows, password reset, API key and console pages: all refused to all agents by path rule. The reasoning is the booking-site reasoning with higher stakes — these paths carry session states, tokens and customer data, and a user-triggered fetcher acting 'on a user's behalf' is precisely the visitor whose delegation should stop at the application's door unless your own product design says otherwise.

API documentation is public and open; the API itself is a different surface with its own authentication and rate story, outside crawler policy. Do not let the docs decision blur into the API's — different layers, different controls, different owners.

Support chats, previews and the sharing layer

SaaS products are shared in support chats, community threads and incident channels — links pasted at the moment of need. facebookexternalhit renders those cards; keep it allowed and allowlisted, and share-test the docs pages users actually paste, not just the homepage. A broken preview on a troubleshooting link is a support cost dressed as a crawler detail.

Status and changelog pages deserve a line in the review too: they are heavily fetched during incidents by every agent class, legitimately. Caps sized for quiet weeks will bind exactly when the status page matters most; the Gazette's storm lesson — caps as reporting instruments — applies unchanged.

Five actions to take today

  1. Allow retrieval on docs and marketing. Citations inside answers are the discovery channel; measure their trial conversions before ever reconsidering.
  2. Decide training as a stated posture. Either answer is defensible for public docs; an unexamined default is not. Record the reasoning.
  3. Fence the application surface completely. App, billing, login, reset and console paths refused to all agents by path rule.
  4. Keep API policy separate. Public API docs are content; the API itself is authentication and rate limiting — different layer, different owner.
  5. Protect the support-sharing layer. Preview fetcher allowlisted; share-test the docs and status pages your users paste under pressure.

Worked example: a SaaS that treats docs as the demo

LoopLedger, the fictional bookkeeping SaaS, allows retrieval across its 480 documentation pages and marketing site, blocks the training class as a recorded posture ('consent is a licensing conversation'), and fences /app, /billing, /login, /reset and /console for all agents. Caps sit at five times baseline with headroom on /status.

In the two quarters after the policy shipped, answer-referred trial starts rose 61% — the figure from its policy review — with documentation citations identified as the mechanism in its own attribution notes: 9,700 retrieval fetches a quarter, concentrated on integration guides.

The fences fired 190 times in the same period, all user-triggered fetches refused at the application boundary — the policy working exactly at the seam it was built for: everything public is citable, everything operational is closed.

Frequently asked questions

Will allowing retrieval agents expose my API?

No. Retrieval agents fetch public web pages — your documentation and marketing content. Your API is governed by its own authentication and rate controls, a separate layer. What crawler policy adds is the fence that keeps agent fetches out of the application's web paths.

Our docs are our moat — shouldn't we block training and retrieval both?

Training, coherently, yes if that is your posture. Retrieval is the harder call: blocking it protects the docs from summary while removing them from the answers where buyers discover tools. Most SaaS vendors in these guides separate the two — block training, open retrieval — and let conversion data arbitrate.

What about our staging docs and preview deploys?

Fence them by path or put them behind authentication — never rely on unlinked obscurity. Preview and staging subtrees are a standard fence line in the path-rules guide precisely because agents traverse structure humans never see.

Can competitors use retrieval access to monitor our pricing?

Your public pricing is public; a person can read it too. What policy controls is bulk, named automation — and the scraping class plus rate caps address the monitoring-at-scale version of the concern without closing the citation channel your growth depends on.

Build my block kit — free · Browse the agent database

Related guides

HONEST SCOPE: this policy blocks KNOWN agent user-agents and behaviors listed below. Sophisticated spoofing can bypass any user-agent block (e.g. an agent sending a stock browser UA, or rotating residential IPs). This is a control, not a guarantee.
← Back