What Is llms.txt and Do AI Engines Use It?
A proposed standard for a markdown file at /llms.txt that gives AI tools a short summary of a site and links to its most useful pages, so they skip the clutter.
llms.txt is a proposal, first published by Jeremy Howard in 2024 at llmstxt.org, to help AI agents use a website. Web pages wrap their content in navigation and scripts, which wastes an agent's limited context. An llms.txt file gives agents a curated starting point: the site's name, a short summary, and links to the pages that matter, ideally in clean markdown.
The format is plain markdown. An H1 with the site's name is the only required part, followed by a blockquote summary, optional notes, and H2 sections that list links with short descriptions. A section named "Optional" marks links an agent can skip. Many sites also publish an llms-full.txt with their full text, a common convention the proposal doesn't define.
Support varies by AI engine, so treat llms.txt as a cheap extra, not a ranking lever. Coding agents and docs tools use it most. Google has said Google Search doesn't use it, including for AI Overviews, though Chrome's Lighthouse added an experimental check for it in 2026. Content still decides whether AI engines cite your docs; see GEO for documentation.
How Kelu uses this
kelu.dev publishes its own llms.txt, generated from the site's pages at every build, plus an llms-full.txt with their text. When Kelu ingests your docs site, its crawler doesn't depend on llms.txt: it finds pages through your sitemap and by following links, then turns each page into markdown with headings before chunking.
Related terms
MCP (Model Context Protocol)
An open protocol that lets AI applications (like IDE assistants) connect to external tools and data sources through a standard JSON-RPC interface.
Read definitionKnowledge Base
A curated, indexed collection of an organization's knowledge — docs, wikis, tickets, repos — that an AI assistant retrieves from when answering questions.
Read definitionChunking
Splitting documents into smaller passages before embedding, so retrieval can return the precise section that answers a question rather than a whole page.
Read definitionRAG (Retrieval-Augmented Generation)
An AI architecture that retrieves relevant documents from a trusted knowledge source and passes them to a language model, so answers are grounded in real content instead of the model's training data.
Read definition
From definition to deployment
Connect a knowledge source and see cited AI answers on your own content in minutes — free, no credit card required.
Free to start · No credit card required