Define the business decision before the robots rule
An AI crawler policy should reflect what the organization wants to permit: search discovery, user-requested retrieval, model training, or none of these. Blocking or allowing every agent as one group can create an outcome the business did not intend.
OpenAI documents separate user agents for search, training, and user-triggered access. Anthropic also documents crawler controls. Policies and agent names can change, so verify the current primary documentation before deployment.
Build an agent register
Maintain a reviewed register with:
| Field | Example decision | | --- | --- | | Declared user agent | Exact token used in robots.txt | | Operator | Company responsible for the crawler | | Documented purpose | Search, training, user request, or another use | | Desired policy | Allow or disallow | | Official source | Current operator documentation | | Technical control | robots.txt, authentication, firewall, or rate limit | | Last reviewed | Date the policy was revalidated | | Owner | Legal, security, content, or engineering approver |
Do not copy an unverified list from another site and assume every name is current.
Understand the control boundaries
robots.txt is a public crawling instruction, not an access-control system. Compliant crawlers may follow it, but protected or confidential information still requires authentication and server-side authorization.
Use the right control:
- Public page discovery: robots policy and page-level index controls
- Private content: authentication and authorization
- Staging: access control plus noindex defense
- Abuse or excessive traffic: rate limiting and firewall controls
- Licensed content: contractual and legal review in addition to technical policy
Do not place secrets or private file paths in robots.txt. The file is publicly readable.
Separate search from training where supported
OpenAI's documentation distinguishes OAI-SearchBot from GPTBot. This allows a site owner to make separate decisions about search inclusion and model training. User-triggered agents can have different behavior and controls.
The same principle applies across operators: base the policy on the documented agent and purpose, not the word "AI."
A policy might allow search discovery while disallowing training, but the organization should approve that choice. Technical teams should not infer legal or content licensing policy alone.
Test the deployed result
After changing robots.txt:
- Fetch the production file from the canonical host.
- Confirm it returns 200 as plain text.
- Check that specific agent groups do not conflict with broader groups.
- Verify sitemap references use the production canonical origin.
- Test important public URLs without cookies.
- Check response headers, page-level robots directives, and authentication.
- Monitor server logs for representative user agents and errors.
Crawler access does not guarantee inclusion, indexing, citation, or a favorable answer. It only removes or applies one access condition.
Keep important content server rendered
Allowing a crawler is not useful if the important answer appears only after an unsupported interaction, authenticated API request, or failing JavaScript dependency. Public content intended for discovery should have stable response status, canonical metadata, crawlable links, and meaningful server-rendered HTML.
Use structured data only when it matches visible content and a supported vocabulary. There is no universal markup that forces an AI system to cite a page.
Review changes over time
Set a recurring review for:
- New or retired user agents
- Changed operator documentation
- Business policy changes
- Security and legal requirements
- Unexpected server load
- Important content moved behind authentication
- Search or AI visibility changes
Store the policy in version control so changes have an owner, rationale, and review date.
Limitations
Agent strings can be spoofed, not every system follows robots instructions, and operator behavior can change. Network controls may also block legitimate crawlers accidentally.
The responsible approach is layered: public crawling instructions, real access control for private data, tested rendering for public pages, current documentation, log review, and clear organizational ownership.
Primary sources
- OpenAI crawler and user agent documentation OpenAI Developers accessed
- Anthropic web crawler controls Anthropic Help Center accessed