Four website formats your web team may mention

Learn what robots.txt, sitemaps, llms.txt and Markdown are for—and which problems they do not solve.

AI discovery guide

By OptiviewFor business owners, marketers and agenciesAbout 2 minPublished October 10, 2026Sources checked October 10, 2026Updated October 31, 2026

Download guide (.md)

Editorial illustration · Get set up

A few terms in plain English

Markdown A simple text format that keeps headings, lists and links.

HTML The code used to describe a webpage.

robots.txt tells cooperative crawlers where not to go

The Robots Exclusion Protocol specifies how crawlers interpret rules for paths. It is not authentication or authorization. Protect private information with real access controls rather than a disallow rule. Different user-triggered fetchers may also document behavior that differs from automatic crawlers. RFC 9309.

sitemap.xml lists pages you want crawlers to discover

A sitemap provides a structured URL inventory, with optional metadata such as modification time. It helps discovery but does not supply the full content of each page or require a search engine to index it. Keep canonical URLs accurate and modification dates meaningful. Sitemap protocol.

llms.txt is a proposed guide, not a requirement

The llms.txt proposal describes a concise Markdown file with context, guidance and links to more detailed content. Its current proposal also describes Markdown URL conventions. A convention is not proof that any particular website implements those routes, or that every AI service uses the file. The llms.txt proposal.

If your guide links to a Markdown resource, request that exact URL and verify its content. Do not append .md to arbitrary pages and assume it works. A broken discovery link is still broken when an AI agent follows it.

Markdown: a readable format for page content

A page’s Markdown representation carries headings, facts and links. It can be delivered as a real file or through configured content negotiation. It needs the same factual ownership and freshness discipline as the visible website. A short index does not replace the underlying answers.

For example, a fictional supplier might list its support page in a sitemap and llms.txt. The support page still needs a clear coverage statement and limitations; neither index invents those details.

Choose the right tool for the problem

Audit discovery links, access controls and page content separately. Optiview adds source preparation and a reviewed publishing workflow; it does not replace your robots policy or promise universal llms.txt adoption. See delivery options and HTML versus Markdown.

Go to the source

Keep exploring.

References linked in this guide. Open the original source for its full scope and latest details.

rfc-editor.orgRFC 9309Opens in a new tab sitemaps.orgSitemap protocolOpens in a new tab llmstxt.orgThe llms.txt proposalOpens in a new tab

Keep going

A useful next read.

Technical guide

Can AI read the same page your customers see?

Developer guide

Convert HTML into Markdown with the Parse API

Developer guide

Fetch a public webpage as Markdown with Retrieve

Back to the resource library

Additional source context

Ask what problem you need to solve before adding another file. A sitemap helps discovery; it cannot promise that AI will read or recommend you.

Your website was built for people. Optiview makes it ready for AI, too.

Try Optiview

See my site with Optiview · Run your free AI Audit

Explore

Publish · Parse · Retrieve · Pricing

Learn

Guides · Developer docs · Research & thinking · Experiments & methods · AI Audit · AI agent directory · FAQ

Company

About Optiview · Agencies · Contact · Privacy · Billing terms

Additional page links


Published brand narrative

Site-wide brand information

Who we are

Optiview — Parse, Retrieve and Publish for AI

Optiview helps businesses and developers make website information easier for AI to use. Parse converts supplied HTML into clean Markdown and source metadata. Retrieve fetches an eligible public HTTPS webpage and prepares its content. Publish helps a verified business prepare its own website content and make approved Narrative Graph context available through a supported connection.

Parse and Retrieve use application keys and a prepaid developer wallet. Publish has separate website setup, ownership checks and a subscription. Optiview does not control whether an outside AI service discovers, uses or cites the resulting content.

Applies to: Optiview

Why we matter

A clearer source, from website content to business context

Webpages mix useful information with navigation, layout and application code. Businesses may also have important qualifications, answers and context that are not clearly published.

Optiview gives developers a way to prepare source content for their applications, and gives businesses a way to publish the context their teams approve. Your application chooses how to use Parse and Retrieve results. Your business controls what it publishes through Publish.

Clearer publishing makes information available; it does not determine an AI system’s answer. Each outside service chooses its own retrieval, recommendations and citations.

What we offer

Three products, with separate jobs and clear pricing

Developer credits do not purchase Publish subscriptions. Parse and Retrieve do not add a customer’s private Narrative Graph to arbitrary third-party pages. See https://optiview.ai/pricing/ and https://optiview.ai/developers/ for the current offer and limits.

Why trust us

Inspect the output and understand what the evidence establishes

Developers can inspect the prepared Markdown, job status, source metadata, processing warnings and credit ledger. Retrieve records the public source response it observed; Parse identifies supplied HTML rather than claiming to verify the current origin.

Publish administrators can review parser output, the combined published-output preview, source checks and publication history. A preview is separate from a recorded live delivery. Saved edits remain separate from the active publication until explicitly published.

Our experiment pages contain selected first-party AI conversations and source comparisons. Delivery records establish what was served, not downstream ingestion, recommendations, citations or revenue. Observed language matches do not establish causation or a consistent visibility improvement. Explore the evidence and limitations at https://optiview.ai/proof/.

Helpful knowledge

Common questions about the three products

Which product should I use? Use Parse when you already have HTML. Use Retrieve when you need Optiview to fetch one eligible public page first. Use Publish when your business wants to prepare its verified website and publish approved context.

Do Parse and Retrieve require website ownership or DNS changes? No website connection is needed for these developer products. You remain responsible for source rights and permitted downstream use. Public availability does not grant a reuse license.

How do developer accounts and keys work? Create a developer workspace and save the personal sign-in key and offline recovery code. Create scoped service keys for server-side applications. Email is an unverified contact identifier, not account-ownership proof or an email-only recovery method. Keep all credentials private.

Can my agent use Optiview? The authenticated developer API returns prepared content for your application or model. A downloadable JavaScript client and local MCP adapter are available. Your host must authorize sources and any credit expenditure. Compatibility does not establish adoption or a partnership with an AI provider.

Does the API include my private Narrative Graph? Parse and Retrieve are source-only. Verified website publishing and approved publisher context belong to the separate Publish workflow.

Does Publish replace our website or CMS? No. Your website stays in its existing system. Your web team establishes and tests a supported connection. Your team reviews and explicitly publishes any Narrative Graph additions. Ordinary visitors retain the normal website experience.

Does every AI tool get the prepared version? No. Delivery depends on the website connection, source eligibility and the request. Some tools explicitly request Markdown; supported AI requests may also receive prepared Markdown or HTML through eligible delivery paths. Traditional search crawlers retain the original website response. A claimed tool name does not prove the requester’s identity.

Does the demo publish or save to our workspace? No. The public demo is a separate preview. Its AI-generated examples require review and do not transfer automatically into an approved publication.

Will Optiview guarantee citations or recommendations? No. Optiview prepares and publishes sources. Outside AI services choose what to retrieve, use and cite.

What to do next

Choose your starting point

Already using Publish? Review your Narrative Graph, publish approved changes explicitly, inspect the combined source in Markdown and check recorded delivery in Activity. Keep your existing visibility tools to study downstream AI answers.