AI Search
llms.txt Explained: What It Is and Whether Your Site Needs One
By Shayan Ahmad · · 6 min read

llms.txt is a proposed convention for a plain-text Markdown file that lives at the root of a website, usually at /llms.txt, and tells AI systems what the site is and which pages matter most. It has been widely discussed since it was put forward in 2024, and some sites have published one, including agencies and software companies that want a clean summary of their official pages.
It is not a formal web standard in the way robots.txt is, and it is not a switch that makes ChatGPT or Google cite you. This article explains what the file is for, what it does not do, and how to write one if you decide it is worth having.
What llms.txt Is Meant to Do
The idea is simple. A large language model that can fetch web pages often has a hard time seeing a site the way a human does. Homepages can be heavy with navigation and marketing copy. Important facts may be spread across dozens of URLs. An llms.txt file is a short, structured overview, written in Markdown, that introduces the organisation and lists the key pages with a one-line description of each.
In the original proposal, the file sits at https://yourdomain.com/llms.txt. Some sites also publish a longer companion file, sometimes called llms-full.txt, that includes more of the underlying content. The short file is a map. The long file, if you use one, is closer to a curated export.
That is useful in principle for any system that is willing to look at the file when it is trying to understand your brand. It is also useful internally: writing one forces you to decide what the official facts are, and which URLs you consider canonical for each topic.
What llms.txt Does Not Do
It is important not to overstate the effect. A few limits are clear:
- It is a proposed convention, not a required standard that every major AI company has committed to honoring in a documented, uniform way.
- Publishing the file does not, by itself, get you into Google AI Overviews, ChatGPT answers or Perplexity citations.
- It does not replace robots.txt, which is still the established place to allow or block crawlers.
- It does not replace a sitemap, which remains the usual way to help search engines discover URLs.
- It cannot correct inaccurate third-party pages. If a directory or an old article has the wrong address, that problem still needs to be fixed at the source.
Think of llms.txt as a helpful label on your official content, not as a ranking factor you can count on. The work that actually improves your chances of being mentioned, as we describe in our GEO and AEO guide, is still the quality, clarity and consistency of the pages the file points to. AI search optimisation starts there, whether or not you add this extra file.
Should Your Site Have One?
A file is worth considering if you have a clear set of official pages and you want a single, maintained summary that both humans and machines can read. That often includes companies with many services, product docs, location pages or a large help centre, where a crawler might otherwise land on a thin or marketing-heavy URL first.
It is less urgent if your site is a handful of pages that already state the facts clearly, and you do not have time to keep a second document in date. An outdated llms.txt that still lists old products, a previous address or broken links is worse than no file, because it looks official.
If you do add one, treat it as a living document. Whenever you launch, retire or move an important page, update the file in the same change. Someone should own that task, just as someone should own the sitemap and the Business Profile.
How to Write a Useful llms.txt
Keep the structure simple
Markdown is the expected format. A practical file usually includes:
- The organisation name and a short, factual description of what you do and who you serve.
- A small set of key facts that you want quoted accurately, such as contact routes, service areas or how a project starts.
- Headed sections that group important pages: company, services, tools, case studies, locations, documentation.
- For each page, a full URL and a one-sentence description of what a reader will find there.
Write in plain language. Avoid slogans, and do not load the descriptions with keyword lists. The file should read like a careful contents page, not like an advert. If you would not be happy seeing a sentence lifted into an answer, do not put it in the file.
Point to real, stable URLs
Link only to pages that return a normal success response, that you intend to keep, and that represent the official version of that topic. Do not list every blog post you have ever published. Do not list tracker or parameter URLs. If two pages cover the same subject, pick the one you want an AI system to treat as canonical and leave the other out.
Use the same names and facts as the pages themselves and as your Google Business Profile or About page. The file is not the place to trial a new tagline that does not appear anywhere else. Consistency is part of making your brand understandable as an entity, which is a wider on-page SEO concern as well as an AI one.
Publish it where it can be fetched
Place the file at the root of the domain you care about, at /llms.txt, using a text or Markdown content type. Make sure it is not blocked in robots.txt, and that it is available over HTTPS without a login. You can link to it from a footer or a documentation page if you want humans to find it, but the main requirement is that a simple request to that address returns the file.
You do not need a special plugin for a small site. A static file in the website root is enough. On larger stacks, add it to the same release process as other root files, so it cannot drift out of date unnoticed. If you later change platform, include /llms.txt in the migration checklist alongside redirects and sitemaps, which is ordinary technical SEO hygiene.
How This Fits With the Rest of Your AI Search Work
An llms.txt file is a small piece of a larger picture. It can help a system that looks for an official summary. It will not compensate for thin service pages, conflicting NAP details, or content that never answers the buyer's question. If your time is limited, fix those pages first, then add the file as a map to the work you have already done.
When you brief an agency, ask whether they will maintain the file and, more importantly, how they will improve the pages it lists. A file without that work is a tidy gesture. We discuss that wider scope in our GEO and AEO comparison.
FAQs
Is llms.txt required for AI search visibility?
No. Many sites appear in AI answers without one, and having one is not a recognised requirement. It can still be a useful official summary if you keep it accurate.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which parts of a site they may request. llms.txt is a proposed, human-readable overview of the site. You may want both, but they do different jobs and should not be mixed into one file.
Should I put all of my website content into llms-full.txt?
Only if you are prepared to keep that long file current, and only for content you are happy for a model to reuse. For most businesses, a short llms.txt that points to the best pages is enough. A huge dump of every URL is hard to maintain and not always more helpful.
How often should I update the file?
Update it whenever an important page is added, removed or moved, and review it on a fixed schedule, such as quarterly. A stale file can spread outdated facts, which is the opposite of what you want.
If you would like a second opinion on whether an llms.txt file would help your site, and which pages it should list, start with a free SEO audit and we will look at the official pages an AI system ought to find first.
- llms.txt
- ai search
- technical seo
- robots.txt
- site documentation
- ai crawlers
Have a question about this for your own site?
A 15-minute call is usually enough to tell whether it applies to you and what to do first.


