Sources & References

Google AI Overviews and web content

support.google.com/websearch/answer/13376098

Official guidance on how Google generates AI Overviews and what content characteristics improve visibility.

GPTBot user-agent documentation

platform.openai.com/docs/gptbot

OpenAI's specification for their web crawler, including robots.txt directives and crawl behavior.

Google-Extended bot documentation

developers.cloudflare.com/fundamentals/reference/bots/google-extended/

Cloudflare's guide to the Google-Extended user-agent for AI training opt-out control.

ClaudeBot documentation

support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web

Anthropic's documentation on ClaudeBot web crawler behavior and robots.txt directives.

PerplexityBot documentation

docs.perplexity.ai/docs/perplexitybot

Perplexity's specification for their web crawler and how it respects robots.txt.

Cloudflare AI Bot Tags

developers.cloudflare.com/bots/concepts/cloudflare-bot-tags/

Reference for identifying and managing AI crawler traffic including GPTBot, ClaudeBot, and others.

Generative Engines and the Web

arxiv.org/abs/2404.16366

Academic research on how generative AI systems interact with and cite web content. Key findings on GEO optimization strategies.

How Search Generative Experience (SGE) Works

blog.google/products/search/generative-ai-search/

Google's blog post explaining how AI-powered search results are generated and what content signals matter.

OpenAI ChatGPT API

platform.openai.com/docs/api-reference

OpenAI API documentation for ChatGPT models and web browsing capabilities used in our live citation testing.

Anthropic Claude API

docs.anthropic.com/en/api

Anthropic API documentation for Claude models with web search integration.

Perplexity API

docs.perplexity.ai/reference/post_chat_completions

Perplexity API for real-time web search and citation testing.

Brave Search API

brave.com/search/api/

Brave Search API documentation for AI-powered search results.

Schema.org Article specification

schema.org/Article

Canonical reference for Article structured data properties including citation, isBasedOn, and license.

Schema.org Organization specification

schema.org/Organization

Specification for Organization entity markup used in authority signals.

Schema.org Person specification

schema.org/Person

Specification for author and contributor markup.

Robots.txt Specification

rfc-editor.org/rfc/rfc9309.html

Official RFC 9309 specification for robots.txt file format and crawler directives.

Meta Robots Tag Specification

developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Google's documentation on meta robots tags including noindex, nofollow, and noai.

Canonical Link Element

developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Guidance on using canonical tags to consolidate duplicate URLs and improve crawlability.

Optiview AEO/GEO Test Corpus v1.0

optiview.ai/datasets/aeo-geo-test-v1.csv

Our reference dataset of test pages used to validate scoring methodology across 18+ industry verticals.

OptiviewAuditBot Documentation

api.optiview.ai/bot

Our crawler's user-agent specification, crawl behavior, and how to control it via robots.txt.