Which AI bots should I allow?

Group pages before choosing crawler rules

A company may publish product guides for anyone to read while keeping customer contracts and account exports behind sign-in. A blanket crawler rule can quietly hide useful public pages, while a disallow line can create a false sense that a private-looking URL is protected.

Start with the page group and the result you want. Public discovery, future model use, user-requested fetching, and confidential access are separate decisions. The map shows which control belongs to each content group before a technical owner edits a rule.

Use the reading guide to jump to the publisher controls, decision matrix, worked example, security boundary, reusable policy brief, or next decision.

Separate search, model use, and user requests

The label AI crawler does not identify one universal purpose. Some bots help a service discover public pages for search. Other tokens express a publisher’s preference about future model training or grounding. A user-requested fetch may follow a different path again.

OpenAI documents OAI-SearchBot for ChatGPT search separately from GPTBot, which may collect content for model training. The settings are independent, so a publisher can allow search while disallowing GPTBot. This is a documented control, not a promise that a page will be cited.

Google uses Googlebot controls for access to Search, including its AI features. Google-Extended is a separate robots.txt token for certain training and grounding uses; Google says it does not change Search inclusion or ranking. Do not block Google-Extended if your actual decision concerns Google Search access.

Anthropic and Perplexity also document separate search and user-requested agents. Their guidance shows why you should inspect each provider’s current policy instead of copying one generic list. Perplexity says its Perplexity-User fetcher is initiated by a user and generally ignores robots.txt rules.

Use a policy matrix for each URL group

Use the table to record the desired outcome before writing robots.txt rules. The examples are current publisher tokens checked on October 9, 2026. The names do not imply identical behavior across providers.

Apply the rule to a hypothetical publisher

Consider a hypothetical software company with public setup guides, licensed partner research, and a private customer portal. Its leaders want the setup guides to remain discoverable, but they have not granted every model-use right for partner research. Customer exports must stay available only to the signed-in account that owns them. This example is a policy exercise, not a client result.

The team can allow Googlebot and selected AI search agents on the public guides, then choose training and grounding preferences per provider and URL group. It should check partner terms before exposing research publicly or permitting model use. Customer exports stay behind authentication and an authorization check at the application or origin. A robots.txt disallow line may reduce compliant crawling, but it cannot replace those checks.

Before applying the policy, the site owner should compare the intended rules with the effective robots.txt on each hostname and the CDN or firewall behavior. If a site-wide default or edge rule overrides the intended exception, the file can look right while the actual request still fails.

Robots.txt does not protect private URLs

The Robots Exclusion Protocol asks compatible crawlers to follow published rules. The standard explicitly says those rules are not a form of access authorization. A URL that returns customer data without checking the requester remains exposed even when robots.txt says not to crawl it.

For restricted content, use sign-in, server-side permissions, and the protections already available at the origin or CDN. For public pages, test the exact URL paths and the response delivered to each intended crawler. Then inspect request logs or the provider’s webmaster tools to confirm that the rule reached the layer doing the blocking or allowing.

Keep visibility expectations modest. Google says a page that meets its technical requirements still is not guaranteed to be crawled, indexed, or served. A crawler allow rule enables a path; it does not promise inclusion or citations.

Write a crawler policy brief

Make the decision reviewable by the content owner, site operator, and anyone responsible for licensing or privacy. Keep the brief next to the robots.txt or edge-rule change so a future edit does not erase the reason for it.

• URL group: list the public, licensed, and restricted routes covered by the decision.
• Search goal: identify which public pages should be discoverable and which publisher search agents are in scope.
• Model-use preference: record the allowed or disallowed training and grounding uses by publisher and content owner.
• User fetch: note whether user-directed page requests are acceptable and how each provider treats its agent.
• Protection and verification: name the owner for robots.txt, CDN or WAF rules, application access checks, request logs, and rollback.

Choose the next policy decision

Start with one URL group and test the live response before expanding the policy. If ownership, publisher behavior, or edge rules are unclear, I can help review the current setup and define the smallest safe change.