AI discovery guide ·
What do ChatGPT, Gemini, Claude and Perplexity actually see when they visit a website?
There is no single view of your website shared by every AI service. What a system receives depends on the retrieval path, request, access rules and response at that moment. Start with the service’s documented behavior, then inspect your own delivered content.
By Optiview · Sources checked
Download this article (.md)Separate search, user requests and training
OpenAI documents OAI-SearchBot for search, GPTBot for potential training use, and ChatGPT-User for user-triggered actions. These roles are different; a training crawl is not evidence that your page appeared in a search answer. OpenAI crawler documentation.
Anthropic separately documents ClaudeBot, Claude-SearchBot and Claude-User. They cover training, search and user-directed retrieval respectively. Treat the observed requester and purpose separately rather than labeling every request “Claude read our site.” Anthropic crawler documentation.
Google and Perplexity have their own controls
Google-Extended is a control token for specified Gemini training and grounding uses, not a separate crawler user-agent. Google Search AI features use Search infrastructure and its applicable controls. Google says special AI files are not required for AI Overviews or AI Mode. Google crawler controls · Search AI features.
Perplexity distinguishes PerplexityBot for search from Perplexity-User for user-requested visits. Its documentation says user-triggered fetches generally ignore robots.txt. This is a documented distinction, not a reason to weaken your access controls. Perplexity crawlers.
Inspect the response before interpreting the answer
For a product page, check whether its description, price qualifications and supporting links arrive in the initial HTML. Then compare a rendered browser view. A script file being downloaded does not establish that an AI system executed it. Regional redirects, authentication and consent flows can change the response.
Record the final URL, status, Content-Type, timestamp and source limitations. A user-agent string alone is not authenticated identity. An answer about your brand may draw on another page or an index; it is not proof of a fresh visit to your homepage.
Where a clearer source helps
Optiview prepares received content and combines it with reviewed, published additions for supported delivery. It does not make all four platforms request Markdown, crawl links or cite your brand. Compare a real page in the website preview, review connection options and distinguish response evidence from outcomes in our methodology.