Is llms.txt worth implementing?
It is cheap, low risk, and not guaranteed to be read by anyone, so treat it as an option rather than a project. A plain-text map of your most useful pages costs about an hour and cannot hurt you. What it will not do is substitute for content that answers questions, a site that serves its substance in HTML, or accurate third-party records. Do those three first, then add the file if you want to.
What the proposal actually is
A plain-text file at a known path listing your canonical URLs with a short description of each, so a machine can find the useful parts of a site without crawling every filter permutation and tag archive.
It is a convention proposed by the community, not a standard any vendor has committed to honoring. That is the whole basis for the hedged answer — the idea is sensible, and adoption is not something you or anyone selling it to you can promise.
Why the honest answer is a shrug
Be careful with anyone presenting this as a ranking lever or a visibility unlock. The most common version of that pitch bundles an hour of work into a recurring engagement and reports the file's existence as a result.
The correct framing is cheap insurance. If it gets adopted, you already have one. If it does not, you spent an hour. Neither outcome deserves a line item on a strategy deck.
If you do it, do it properly
- List only pages that stand alone. A page that needs the rest of the site to make sense does not belong on a map for machines.
- One line saying what each page answers. Not a keyword phrase — an actual description of the question it settles.
- Keep it current or delete it. A file full of dead links is worse than no file, because it is a public statement that you do not maintain your own site.
- Do not duplicate your sitemap. A sitemap lists everything; this should list what is worth reading.
- Leave out gated, thin and templated pages. Pointing a machine at your weakest content is an odd thing to volunteer.
Where the same hour is better spent
Confirm your substance is in the served HTML rather than injected by script. Confirm the AI fetchers are not being blocked by your security layer. Rewrite your ten most important paragraphs so they survive being quoted alone. Reconcile your business records across the platforms that feed local answers.
Each of those has a known mechanism and a measurable outcome. That ordering is the difference between AI search work with an engineering basis and a checklist of fashionable files — and it is the same reason we start integration projects with what the systems actually do before adding anything new.
Topics: llms.txt · standards · tradeoffs · technical
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.