llms.txt Generator Setup, File Specs, and Search Reality

Site owners who want to provide structured context for artificial intelligence tools can use an llms.txt generator or assemble the file by hand. Online utilities scrape pages to build these markdown files automatically, but site owners need to understand what belongs in the file, which crawlers read it, and how the specification evolved in August 2026.

An llms.txt generator is a software tool that scans website pages, extracts titles and descriptions, and formats them into a curated markdown index for language models. It creates an llms.txt file to guide AI agents to clean documentation, though search engines still determine crawling access through robots.txt files.

How does an llms.txt generator work?

An llms.txt generator crawls a website or parses an XML sitemap to compile page titles, URLs, and summaries into a plain markdown file. The resulting index helps automated systems find clean documentation without parsing navigation bars, scripts, or advertising markup.

Online utilities provide form-based generation for existing domains. The generator at llmstxtgenerate.com accepts a sitemap URL, a website homepage address to follow internal links, or a list of pasted URLs. It extracts titles and descriptions to produce a standard llms.txt file, while offering an llms-full.txt file for a one-time fee of $1 to bundle complete page text.

Developer tools automate this process through code. Firecrawl created an open-source project named llmstxt-generator, which used Firecrawl for crawling and GPT-4-mini for text processing. The tool provided a web interface at llmstxt.firecrawl.dev and a direct API endpoint. On 17 June 2025, Firecrawl committed a deprecation notice stating the standalone API endpoint would remain unmaintained after 30 June 2025, directing users toward an example repository at create-llmstxt-py.

Modern documentation platforms generate these files automatically. According to Mintlify's documentation, the service creates and hosts an llms.txt file at the project root and at /.well-known/llms.txt with zero maintenance. Mintlify also serves llms-full.txt, advertises these endpoints through HTTP Link headers, truncates page descriptions to 300 characters, and splits large documentation sets into sub-indexes under /_llms/ when a file exceeds 100,000 characters.

What does an llms.txt file contain?

A standard llms.txt file organizes links to clean markdown pages under a strict heading structure defined by llmstxt.org. The specification sets an exact hierarchy so programmatic parsers and language models can read the document.

The file begins with an optional byte-order mark and a mandatory H1 heading naming the project or site. A blockquote follows immediately below the title to provide a brief project summary. Authors can include plain markdown paragraphs or lists after the blockquote to explain how to interpret the files, provided these sections contain no headings. The document then presents file lists organized under H2 headings, where each entry is a markdown link formatted as name with optional descriptive text after a colon.

Pages linked inside the file should serve clean markdown. The llmstxt.org specification recommends serving markdown versions of web pages by appending .md to the full URL (page.html.md) or by replacing the file extension (page.md). For URLs that end without a file name, servers can supply index.html.md or index.md.

The specification provides an example based on the FastHTML documentation:

# FastHTML
> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's FT "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:
- Although parts of its API are inspired by FastAPI, it is not compatible with FastAPI syntax and is not targeted at creating API services.
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.

## Docs
- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options

## Examples
- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.

## Optional
- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development.

In this layout, the Optional section holds secondary links that an agent can skip when operating with limited context.

When should a site use an llms.txt generator?

Publishing an llms.txt file helps coding agents find API documentation, but it does not influence Google Search rankings or generative AI answers. Website owners should evaluate the technical audience for their content before adding one.

Search engines do not use these files. According to Google's AI optimization guide, last updated on 10 July 2026, site owners can ignore LLMS.txt files and other special markup for Google Search, AI Overviews, and AI Mode. Google notes that its systems crawl and index many file types, but crawling an llms.txt file does not mean the search engine treats it in any special way.

Access permissions for AI search engines depend on robots.txt. According to OpenAI's crawler documentation, OpenAI uses OAI-SearchBot to surface websites in ChatGPT search features, while GPTBot crawls content for foundation model training. If a site owner blocks OAI-SearchBot in robots.txt, ChatGPT search answers will not display the site. OpenAI references llms.txt only as the index for its own developer documentation at platform.openai.com/llms.txt, not as a file it collects from third-party websites.

Public adoption of the standard remains low across general websites. The HTTP Archive 2025 SEO report, published on 15 January 2026 and updated on 9 June 2026, identified valid llms.txt files on 2.13% of desktop sites and 2.10% of mobile sites, totaling 324,184 valid files. Most files originated from automated CMS add-ons: 39.6% carried comments from All in One SEO, and 3.6% contained comments from Yoast SEO.

Technical documentation sites benefit most from the format. The llmstxt.org specification notes that AI labs maintain llms.txt files for their own developer guides, including OpenAI, Anthropic, and Gemini. For a library or software API, an llms.txt file helps coding assistants retrieve precise code examples.

General startup discovery requires broader distribution channels. If you are launching a product and looking for early distribution, you can submit to directories like Product Hunt or bunny.directory. As a disclosure, you are reading this guide on our launch directory blog. We review submissions manually, reset daily rankings at midnight UTC, and keep listings visible with daily streaks. Product pages on bunny.directory receive permanent dofollow backlinks, which free tiers keep as long as a verification badge remains on the product site.

What's new (as of 2026-09-23)

The specification for llms.txt and browser auditing standards changed significantly in mid-2026. These updates clarified how agents discover files and how developers scope directory paths.

In August 2026, llmstxt.org published v2 of the specification. The update introduced standard link relations for discoverability: rel="alternate" with type="text/markdown" points to a page's markdown version, and rel="describedby" points to the llms.txt file that covers it. Webmasters can serve these links in HTML link elements or in HTTP Link response headers.

The v2 specification also defined path scoping and file conventions. A file at /docs/llms.txt covers all pages located beneath that path, and agents must use the most specific file when multiple files exist. The specification permits both page.html.md and page.md extensions. Furthermore, v2 dropped the context-expansion tool llms_txt2ctx and removed the mechanical parsing semantics of the Optional section, preserving it solely as an informal convention for secondary material.

Browser developer tools introduced automated checks for the format earlier in the year. According to Chrome Lighthouse documentation, last updated on 5 May 2026, Lighthouse audits include an agentic browsing check for llms.txt. A server error during retrieval triggers an audit flag, while a 404 response is marked as Not Applicable because the file remains optional.

Google updated its official position on AI search requirements on 10 July 2026. In Google's AI optimization guide, the search engine stated that webmasters do not need to create AI-specific text files, markdown variants, or content chunks to succeed in generative search features.

Key takeaways

• Google's AI optimization guide stated on 10 July 2026 that Google Search and its generative AI features do not use llms.txt files.
• Website inclusion in ChatGPT search results is controlled by OAI-SearchBot rules in robots.txt rather than llms.txt.
• The HTTP Archive reported in June 2026 that only 2.10% of mobile sites host an llms.txt file, with 39.6% generated automatically by All in One SEO.
• Version 2 of the llms.txt specification, published in August 2026, introduced rel="describedby" discovery links and defined path scoping for subdirectories.

FAQ

Does Google use llms.txt for AI Overviews?

Google does not use llms.txt files for AI Overviews, AI Mode, or organic search rankings. According to Google's AI optimization guide, last updated on 10 July 2026, site owners can ignore special markdown files and machine-readable text files for Google Search. The search engine relies on its standard crawling systems and retrieval-augmented generation to surface web pages.

Can you place an llms.txt file in a subdirectory?

You can place an llms.txt file in a subpath such as /docs/llms.txt to describe all pages located under that directory. The llmstxt.org v2 changes, published in August 2026, state that a file covers the URLs beneath its own path, and automated agents should choose the most specific file available on a domain.

Does an llms.txt file prevent AI bots from scraping your site?

An llms.txt file does not block scrapers or restrict automated access to website content. Site owners must configure robots.txt directives to manage crawler permissions, such as disallowing GPTBot for training data or disallowing OAI-SearchBot for search indexation, as detailed in OpenAI's crawler documentation.

To generate a clean markdown index directly in your browser, you can use our free llms.txt generator.