Educational guide
The short answer
AI products do not all “read” a website through one process. A crawler may fetch a URL, a search system may index its content, an answer product may retrieve that index for a live response, and a model provider may use a different crawler for training. Treat each layer separately. Allowing one bot creates eligibility for its documented purpose; it does not guarantee indexing, citation or recommendation.1. Start with the HTTP response
Request the exact canonical URL and inspect its final status, redirect chain, content type and rendered HTML. A useful public page should normally return 200 in one hop. A 3xx points elsewhere, a 4xx says the resource is unavailable, and a 5xx signals a server failure. Verify desktop and relevant bot requests without disguising one response to search engines and another to users.2. Robots.txt controls crawling, not every form of use
Robots.txt provides crawler directives for compliant user agents. It does not remove a URL already known to a search system, protect confidential information or replace authentication. Google documents that blocking a page in robots.txt can prevent crawling while the URL may still appear without a useful snippet. See Google's robots.txt introduction.3. Noindex requires crawl access
A robots meta tag or X-Robots-Tag can tell supporting search systems not to index a response. The crawler must be able to fetch the page to see that directive. Google documents the supported indexing rules and warns against blocking a noindex page in robots.txt. See Google's robots meta documentation.4. Rendering and content extraction
A fetched HTML response may contain the main content immediately or rely on JavaScript. Test what a crawler receives before client-side execution and what a supported renderer can produce. Keep essential titles, headings, canonical links and primary copy available in robust HTML. This is good accessibility and reliability practice; it is not a special citation trick.5. OpenAI uses purpose-specific agents
OpenAI documents OAI-SearchBot for surfacing websites in ChatGPT search and GPTBot for potential model-training use. It also documents user-initiated agents separately. A publisher can make different decisions for search and training. Review OpenAI's publisher and developer FAQ and verify the current published IP information before firewall changes.6. Perplexity also distinguishes agents
Perplexity documents PerplexityBot for search indexing and Perplexity-User for user-requested fetching. It states that robots directives for Perplexity-User may be ignored because the request is initiated by a user, while network controls can still apply. Check the current Perplexity crawler documentation before configuring access.7. Google crawling and AI features
Google publishes its common crawler identifiers and says established SEO requirements apply to its AI features. A page must be indexed and eligible to appear, but eligibility does not guarantee it will be shown. Use Google's crawler list and AI-search guidance as the primary references.8. Crawlability is not training
A page fetched for current search retrieval is not necessarily used to update a model's weights. Conversely, information learned during model training is not a live copy of the page. Pretraining, fine-tuning, indexing and retrieval are different processes with different update paths. The LLM training guide explains the distinction.9. Diagnostic checklist
Check the canonical URL, one-hop status, response headers, robots.txt for the exact agent, meta and X-Robots directives, server-rendered content, canonical tag, sitemap inclusion, internal links, log evidence and firewall rules. Then test the product behavior separately. Record agent name, timestamp, result and any exposed source. Never infer a crawler visit solely from an AI answer.10. What success means
Technical success means the intended public agent can retrieve the intended response and the page sends consistent indexing signals. Search success means the URL is indexed where applicable. Answer visibility means a defined prompt run exposed the brand or source. These are separate acceptance criteria, and none guarantees traffic or conversion.Interpretation note
AI and search behavior changes over time. Guidance does not promise discovery, ranking, citations, traffic or commercial outcomes; confirm platform-specific details against primary documentation.