# llms.txt, robots.txt, sitemap.xml and Markdown: four different jobs.

AI discovery guide · By Optiview · October 10, 2026

Canonical article: https://optiview.ai/resources/four-discovery-files/

Sources checked October 10, 2026.

These mechanisms solve different problems. robots.txt communicates crawler rules, a sitemap lists URLs, llms.txt proposes agent-oriented context and links, and Markdown expresses content. None guarantees that a service will retrieve or cite a page.

## robots.txt: crawler rules, not access control

The Robots Exclusion Protocol specifies how crawlers interpret rules for paths. It is not authentication or authorization. Protect private information with real access controls rather than a disallow rule. Different user-triggered fetchers may also document behavior that differs from automatic crawlers. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).

## sitemap.xml: a map of URLs

A sitemap provides a structured URL inventory, with optional metadata such as modification time. It helps discovery but does not supply the full content of each page or require a search engine to index it. Keep canonical URLs accurate and modification dates meaningful. [Sitemap protocol](https://www.sitemaps.org/protocol.html).

## llms.txt: a proposed guide for agents

The llms.txt proposal describes a concise Markdown file with context, guidance and links to more detailed content. Its current proposal also describes Markdown URL conventions. A convention is not proof that any particular website implements those routes, or that every AI service uses the file. [The llms.txt proposal](https://llmstxt.org/).

If your guide links to a Markdown resource, request that exact URL and verify its content. Do not append .md to arbitrary pages and assume it works. A broken discovery link is still broken when an AI agent follows it.

## Markdown: the information itself

A page’s Markdown representation carries headings, facts and links. It can be delivered as a real file or through configured content negotiation. It needs the same factual ownership and freshness discipline as the visible website. A short index does not replace the underlying answers.

For example, a fictional supplier might list its support page in a sitemap and llms.txt. The support page still needs a clear coverage statement and limitations; neither index invents those details.

## Use the mechanism that matches the job

Audit discovery links, access controls and page content separately. Optiview adds source preparation and a reviewed publishing workflow; it does not replace your robots policy or promise universal llms.txt adoption. See [delivery options](https://optiview.ai/web-teams/) and [HTML versus Markdown](https://optiview.ai/resources/html-versus-markdown-for-ai/).