Four website formats your web team may mention
Learn what robots.txt, sitemaps, llms.txt and Markdown are for—and which problems they do not solve.
AI discovery guide
By OptiviewFor business owners, marketers and agenciesAbout 2 minPublished October 10, 2026Sources checked October 10, 2026Updated October 31, 2026
Editorial illustration · Get set up
A few terms in plain English
Markdown A simple text format that keeps headings, lists and links.
HTML The code used to describe a webpage.
robots.txt tells cooperative crawlers where not to go
The Robots Exclusion Protocol specifies how crawlers interpret rules for paths. It is not authentication or authorization. Protect private information with real access controls rather than a disallow rule. Different user-triggered fetchers may also document behavior that differs from automatic crawlers. RFC 9309.
sitemap.xml lists pages you want crawlers to discover
A sitemap provides a structured URL inventory, with optional metadata such as modification time. It helps discovery but does not supply the full content of each page or require a search engine to index it. Keep canonical URLs accurate and modification dates meaningful. Sitemap protocol.
llms.txt is a proposed guide, not a requirement
The llms.txt proposal describes a concise Markdown file with context, guidance and links to more detailed content. Its current proposal also describes Markdown URL conventions. A convention is not proof that any particular website implements those routes, or that every AI service uses the file. The llms.txt proposal.
If your guide links to a Markdown resource, request that exact URL and verify its content. Do not append .md to arbitrary pages and assume it works. A broken discovery link is still broken when an AI agent follows it.
Markdown: a readable format for page content
A page’s Markdown representation carries headings, facts and links. It can be delivered as a real file or through configured content negotiation. It needs the same factual ownership and freshness discipline as the visible website. A short index does not replace the underlying answers.
For example, a fictional supplier might list its support page in a sitemap and llms.txt. The support page still needs a clear coverage statement and limitations; neither index invents those details.
Choose the right tool for the problem
Audit discovery links, access controls and page content separately. Optiview adds source preparation and a reviewed publishing workflow; it does not replace your robots policy or promise universal llms.txt adoption. See delivery options and HTML versus Markdown.
Go to the source
Keep exploring.
References linked in this guide. Open the original source for its full scope and latest details.
rfc-editor.orgRFC 9309Opens in a new tab sitemaps.orgSitemap protocolOpens in a new tab llmstxt.orgThe llms.txt proposalOpens in a new tab
Keep going
A useful next read.
Technical guide
Can AI read the same page your customers see?
Developer guide
Convert HTML into Markdown with the Parse API
Developer guide
Fetch a public webpage as Markdown with Retrieve
Additional source context
Ask what problem you need to solve before adding another file. A sitemap helps discovery; it cannot promise that AI will read or recommend you.
Your website was built for people. Optiview makes it ready for AI, too.
Try Optiview
See my site with Optiview · Run your free AI Audit
Explore
Publish · Parse · Retrieve · Pricing
Learn
Guides · Developer docs · Research & thinking · Experiments & methods · AI Audit · AI agent directory · FAQ
Company
About Optiview · Agencies · Contact · Privacy · Billing terms
Additional page links
- More in this collection
- Technical guide · Can AI read the same page your customers see?
- Developer guide · Convert HTML into Markdown with the Parse API
- Developer guide · Fetch a public webpage as Markdown with Retrieve
Published brand narrative
Site-wide brand information
Who we are
Optiview — Parse, Retrieve and Publish for AI
Optiview helps businesses and developers make website information easier for AI to use. Parse converts supplied HTML into clean Markdown and source metadata. Retrieve fetches an eligible public HTTPS webpage and prepares its content. Publish helps a verified business prepare its own website content and make approved Narrative Graph context available through a supported connection.
Parse and Retrieve use application keys and a prepaid developer wallet. Publish has separate website setup, ownership checks and a subscription. Optiview does not control whether an outside AI service discovers, uses or cites the resulting content.
Applies to: Optiview
Why we matter
A clearer source, from website content to business context
Webpages mix useful information with navigation, layout and application code. Businesses may also have important qualifications, answers and context that are not clearly published.
Optiview gives developers a way to prepare source content for their applications, and gives businesses a way to publish the context their teams approve. Your application chooses how to use Parse and Retrieve results. Your business controls what it publishes through Publish.
Clearer publishing makes information available; it does not determine an AI system’s answer. Each outside service chooses its own retrieval, recommendations and citations.
What we offer
Three products, with separate jobs and clear pricing
- Optiview Parse: Supply complete HTML and a source URL. Receive clean Markdown, source and output hashes, parser details and processing warnings. Parse does not fetch the URL or execute JavaScript. A successful page uses 1 credit.
- Optiview Retrieve: Supply one eligible public HTTPS URL. Optiview checks supported access and publisher policies, fetches anonymous static HTML and parses it. It does not search the web, execute JavaScript, bypass a login or access private networks. A successful page uses 5 credits total, including parsing.
- Developer credits: USD $20 buys 20,000 credits shared by Parse and Retrieve. There is no developer subscription, automatic recharge or scheduled credit expiration. Failed jobs consume no credits. Applicable taxes and the total appear at checkout. Source size, quotas and retention limits apply.
- Optiview Publish: USD $99 per website per month plus applicable tax. Connect a verified website, review the prepared source and optionally maintain approved Narrative Graph context in Identity, Story, Offerings, Trust, Knowledge and Actions. Choose site-wide or exact-page scope. Saving drafts does not publish them. Eligible AI tools can receive prepared Markdown or HTML through supported delivery, while ordinary visitors keep the normal website.
Developer credits do not purchase Publish subscriptions. Parse and Retrieve do not add a customer’s private Narrative Graph to arbitrary third-party pages. See https://optiview.ai/pricing/ and https://optiview.ai/developers/ for the current offer and limits.
Why trust us
Inspect the output and understand what the evidence establishes
Developers can inspect the prepared Markdown, job status, source metadata, processing warnings and credit ledger. Retrieve records the public source response it observed; Parse identifies supplied HTML rather than claiming to verify the current origin.
Publish administrators can review parser output, the combined published-output preview, source checks and publication history. A preview is separate from a recorded live delivery. Saved edits remain separate from the active publication until explicitly published.
Our experiment pages contain selected first-party AI conversations and source comparisons. Delivery records establish what was served, not downstream ingestion, recommendations, citations or revenue. Observed language matches do not establish causation or a consistent visibility improvement. Explore the evidence and limitations at https://optiview.ai/proof/.
Helpful knowledge
Common questions about the three products
Which product should I use? Use Parse when you already have HTML. Use Retrieve when you need Optiview to fetch one eligible public page first. Use Publish when your business wants to prepare its verified website and publish approved context.
Do Parse and Retrieve require website ownership or DNS changes? No website connection is needed for these developer products. You remain responsible for source rights and permitted downstream use. Public availability does not grant a reuse license.
How do developer accounts and keys work? Create a developer workspace and save the personal sign-in key and offline recovery code. Create scoped service keys for server-side applications. Email is an unverified contact identifier, not account-ownership proof or an email-only recovery method. Keep all credentials private.
Can my agent use Optiview? The authenticated developer API returns prepared content for your application or model. A downloadable JavaScript client and local MCP adapter are available. Your host must authorize sources and any credit expenditure. Compatibility does not establish adoption or a partnership with an AI provider.
Does the API include my private Narrative Graph? Parse and Retrieve are source-only. Verified website publishing and approved publisher context belong to the separate Publish workflow.
Does Publish replace our website or CMS? No. Your website stays in its existing system. Your web team establishes and tests a supported connection. Your team reviews and explicitly publishes any Narrative Graph additions. Ordinary visitors retain the normal website experience.
Does every AI tool get the prepared version? No. Delivery depends on the website connection, source eligibility and the request. Some tools explicitly request Markdown; supported AI requests may also receive prepared Markdown or HTML through eligible delivery paths. Traditional search crawlers retain the original website response. A claimed tool name does not prove the requester’s identity.
Does the demo publish or save to our workspace? No. The public demo is a separate preview. Its AI-generated examples require review and do not transfer automatically into an approved publication.
Will Optiview guarantee citations or recommendations? No. Optiview prepares and publishes sources. Outside AI services choose what to retrieve, use and cite.
What to do next
Choose your starting point
- Compare Parse, Retrieve and Publish to choose the right workflow.
- Create a developer workspace for self-service Parse and Retrieve, then add credits and create an application key.
- Read the API quickstart and agent-tool setup for request examples, supported sources, errors, quotas and downloads.
- Review pricing and terms for prepaid credits and the separate Publish plan.
- Preview your website to inspect prepared content and explore unapproved narrative examples without connecting or publishing.
- Plan Publish setup with your web team. Publishing requires ownership checks, review and a supported connection; contact hello@optiview.ai for assisted setup.
Already using Publish? Review your Narrative Graph, publish approved changes explicitly, inspect the combined source in Markdown and check recorded delivery in Activity. Keep your existing visibility tools to study downstream AI answers.