AI Crawler Robots.txt: Search Access vs Training
October 24, 2026 · Sameer Sabir
Review AI crawler rules without mixing search discovery with training collection. Check robots.txt, noindex, firewall access, and what an AEO audit can verify.
An AEO audit should distinguish whether an AI search crawler can reach a public page from whether content may be collected for model training. These are different decisions. Treating every AI user agent as one bot can produce rules that do not match your publishing policy.
This guide focuses on reviewing crawler access for public documentation and product pages. It does not promise citations in AI answers. Keep private dashboards, account records, and customer data behind real access controls regardless of your crawler policy.
Separate search, training, and user-requested visits
OpenAI’s current crawler documentation describes OAI-SearchBot for search and GPTBot for content that may be used for model training. Their robots.txt settings are independent. A publisher can choose to allow search crawling while disallowing GPTBot. ChatGPT-User describes certain user-requested visits; it is not the control for automatic search crawling.
Other providers use their own agents and policies. Review each provider’s official documentation rather than assuming one rule controls all AI services. Record which public pages you want discovered and which uses your organization permits.
Review the deployed robots.txt file
Open robots.txt at the root of the production host, including the correct www or non-www version. Check the actual deployed response rather than a local source file. Identify broad disallow rules, provider-specific groups, and paths that might accidentally match useful public pages.
Do not paste a blanket “Allow: /” recipe into every agent group. First list your public routes and routes that should remain private. A specific crawler group must express the intended path restrictions for that crawler; do not assume rules in another group will automatically be combined with it.
For each public route, write down the intended outcome: Google search access, AI search access, and training collection policy. Review the resulting rules against those decisions. Save the old file so you can compare or roll back an accidental change.
Keep crawl access separate from index instructions
Google documents robots.txt as a crawl-control mechanism. Blocking a URL there is not a dependable way to remove it from search results. A linked URL can still be discovered without Google fetching its page content. Google also needs crawl access to read a noindex instruction on a page.
This distinction matters when reviewing a useful public page that is missing from search. Check its index instructions and declared preferred URL as well as robots.txt. Do not remove authentication from private content to make a crawler test pass. Private content needs server-enforced permissions, not a promise that a robot will stay away.
Check the hosting layer too
An allowed path may still return a bot challenge, a login screen, a redirect loop, or an error from a firewall. Inspect the response body as well as the status. A successful status that serves a challenge page does not establish access to the article.
Use the provider’s published verification information and your hosting platform’s verified-bot controls where appropriate. A user-agent string alone can be imitated. Prefer a scoped rule for the relevant host and public paths instead of disabling protection across the site.
Verify changes without overreading the result
Confirm the deployed file contains the intended rules.
Check that a public article opens without an account and has the expected content.
Use Search Console’s live inspection for Google access and its indexed result for Google’s stored status. These answer different questions.
Review relevant server or CDN logs using the crawler provider’s verification guidance.
Allow time for crawler policy changes to be picked up. One successful fetch is evidence of access, not a citation or ranking guarantee.
A sitemap can help discovery. Neither a sitemap submission nor a crawl-access check guarantees indexing, retrieval, or inclusion in an AI answer. An llms.txt file also does not replace robots.txt, access control, or useful content.
Where an automated AEO check fits
VibeShield’s SEO and AEO audit checks supported technical readiness signals and reports evidence and coverage gaps. It does not measure brand mentions or AI citations. Use those findings alongside your publisher policy, hosting logs, and the broader AEO readiness guide.
Official references
Provider documentation checked October 24, 2026. Prepared with AI assistance. Recheck the official references before changing production crawler policy.