Key Takeaways
- Google currently says website owners do not need special AI text files and explicitly gives llms.txt as something they can ignore.
- OpenAI documents robots.txt controls for its crawlers; its published crawler guidance does not make llms.txt a requirement for ChatGPT search.
- An llms.txt file is a proposed convention, not a guaranteed route into an AI answer and not a replacement for crawling, indexing or useful pages.
- If you already have the file, review its purpose and accuracy calmly. There is no evidence-led reason for a rushed site-wide project.
- Spend most of the effort on technical access, clear source pages, internal relationships, first-hand evidence and consistent third-party information.
The honest answer is simple: an llms.txt file is not a proven shortcut into ChatGPT or Google AI Overviews.
It is a proposed way to give language-model tools a concise list of useful website resources. That may be convenient for a tool that deliberately supports the convention. It does not mean major search platforms use the file for ranking or citation.
Google has gone further than remaining silent. Its current AI-search guidance tells site owners they do not need additional machine-readable files and names llms.txt as an example of an unnecessary distraction.
What Is llms.txt?
The llms.txt proposal suggests placing a Markdown-formatted text file at the root of a website, normally at:
https://example.com/llms.txt
The file can introduce the site and link to selected documentation or content. The idea is to give an AI tool a compact route to material that may otherwise sit inside a large website.
That is different from several familiar files:
- robots.txt communicates crawl permissions to user agents that honour it;
- sitemap.xml lists canonical URLs and useful metadata for search crawlers;
- structured data describes visible page entities and properties using supported vocabularies;
- llms.txt is a proposed curated guide for tools that choose to read it.
Those roles should not be confused. Adding links to llms.txt does not force a crawler to visit them. It does not make a noindex page indexable, override robots.txt, establish a canonical or prove that a statement is true.
What Google Actually Says
Google's AI optimisation guide says there are no additional technical requirements or special optimisations for appearance in AI Overviews and AI Mode. It advises site owners to ignore unnecessary AI-specific text files, explicitly including llms.txt.
Google's related AI features and your website documentation explains that pages must be indexed and eligible to appear in Search with a snippet to be shown as supporting links. The familiar work still matters:
- allow Google to crawl the page;
- return a successful response;
- avoid unintended noindex and snippet restrictions;
- make the important content available;
- use accurate structured data where it genuinely applies;
- provide a good page experience;
- publish helpful, original information.
This does not mean Google will never change its systems. It means a current SEO plan should follow current, documented behaviour rather than a prediction.
If an agency says Google requires llms.txt, ask for the exact current Google source. A screenshot of a generated answer, a social-media thread or the existence of a plugin is not platform documentation.
What OpenAI Documents
OpenAI's web crawler documentation identifies different user agents. OAI-SearchBot relates to surfacing and linking websites in ChatGPT search. GPTBot relates to potential model training. ChatGPT-User supports actions initiated by a user.
The documented control is robots.txt. A business can choose which user agents to allow according to its content, legal and commercial policy.
The documentation does not state that ChatGPT search requires llms.txt. A crawler can find a well-linked public page without the file, and placing a page in the file does not guarantee that ChatGPT will cite it.
OpenAI's publisher and developer FAQ offers further information about search inclusion, crawler controls and referral tracking. Again, the practical themes are access, reliable pages and measurement.
Why the File Became Popular
The proposal arrived at a time when website owners wanted a clear action for AI search. It is attractive because it is:
- quick to create;
- easy to explain;
- visible at a public URL;
- supported by some site tools and plugins;
- similar in appearance to established web-control files.
Those qualities make it a useful experiment for particular documentation tools. They do not establish a broad search benefit.
The same pattern has happened with many SEO fashions. A small technical change becomes a checklist item, then a service, then a claim that every website is “missing” it. The original nuance disappears.
A useful generative engine optimisation strategy is much less dramatic. It makes the business and its evidence easier to retrieve and verify across the web.
When llms.txt May Still Have a Practical Use
There are limited cases where the convention may help:
Developer Documentation
A software company might publish a curated route to stable API and product documentation for tools that explicitly read the file. The value is convenience for those tools, not a guaranteed public-search ranking.
Internal AI Tools
An organisation can design its own assistant to read an approved llms.txt file as part of a controlled ingestion process. In that case, the organisation controls both ends and can test the behaviour.
A Low-Cost Experiment
If the main SEO foundations are healthy, a team may maintain a small file and observe server logs for known access. Define the question first: which user agent is expected to request it, which tool supports it and what outcome would count as useful?
Do not confuse “the file was requested” with “the file caused a citation”. A crawler can discover it and never use its contents in an answer.
Risks and Maintenance Costs
The file is small, but it is not entirely free.
It Can Become Stale
A curated list can retain redirected, deleted or superseded pages. That creates a poor guide for any tool that does use it.
It Can Expose the Wrong URLs
Do not include private, staging, administrative, paid or draft-looking routes. The file is public. It is not an access-control system.
It Can Create Conflicting Descriptions
If the summary in llms.txt differs from the live page, which one should a tool trust? Keep visible canonical pages as the source of truth.
It Can Distract the Team
The largest cost is opportunity. Time spent debating file syntax may be better used fixing blocked JavaScript content, weak service descriptions or inconsistent business profiles.
It Can Encourage False Claims
A provider should not report “AI search optimised” merely because it added a file. Ask for crawl evidence, page improvements, prompt records, citations, referrals and commercial results.
What to Check Before Adding Anything
Run a basic technical review:
- Is robots.txt reachable and intentional?
- Can Googlebot and any other permitted crawler access important pages?
- Do those pages return a useful 200 response?
- Is the main answer present in HTML or reliably rendered?
- Are canonical and noindex directives correct?
- Does the XML sitemap contain current canonical URLs?
- Can important pages be reached through normal internal links?
- Do server and CDN logs show blocked or challenged crawlers?
- Is structured data valid and consistent with visible content?
- Are old pages redirected individually rather than sent to the home page?
If these checks reveal a fault, fix it before creating another discovery file. Our SEO services cover technical access and page quality together. For JavaScript-heavy websites, a website audit can inspect what crawlers and users actually receive.
Improve the Information AI Search Needs
AI-search questions tend to be specific. A broad marketing page may not provide enough information to answer them.
Strengthen pages with:
- a direct explanation near the top;
- clear authorship and review dates;
- original examples and practical steps;
- assumptions behind costs or comparisons;
- limits and cases where the advice does not apply;
- links to primary evidence;
- relevant FAQs written from customer language;
- a clear relationship to the service or product offered.
Avoid creating many thin articles that restate the same answer. Consolidate overlapping pages and make the best source substantially useful.
Our guide to answer engine optimisation covers this source-page work. If AI visibility matters commercially, AI strategy consulting can help separate a sensible experiment from a fashionable distraction.
Build Evidence Beyond Your Domain
An AI answer may cite or rely on:
- official documentation;
- reviews;
- directories;
- forums;
- professional profiles;
- research;
- news and specialist publications;
- customer and supplier websites.
Keep important company facts consistent across genuine profiles. Earn detailed reviews through real service. Contribute useful expertise where the audience already asks questions. Publish case studies only with permission and enough context to be credible.
No root text file can replace that external evidence.
The same principle applies to brand clarity. Our guide to creating a brand AI search can recommend explains how consistent identity, specific services and corroboration work together.
If You Already Have llms.txt
There is no need to panic or delete it blindly.
Review:
- who created it and why;
- which tool or process uses it;
- whether the content is public and current;
- whether every URL returns the intended page;
- whether the summary matches visible content;
- whether logs show legitimate requests;
- how it will be maintained.
If it supports a documented internal or customer use, keep it simple. If nobody can identify a purpose, treat it as a minor optional file. Do not attach ranking claims to it.
Record the decision in the website documentation so the file does not become an unexplained artefact that survives several rebuilds.
A Better AI-Search Priority List
For most small businesses, the order should be:
- crawl and index access;
- useful HTML and stable URLs;
- strong service and product pages;
- evidence-led supporting articles;
- accurate identity and structured data;
- authoritative external mentions;
- measurement of AI citations, referrals and enquiries;
- optional format experiments with a defined user.
That order is less exciting than installing a plugin, but it addresses the conditions platforms actually document.
If a migration or rebuild is involved, preserve the foundations before adding new experiments. The digital marketing service can connect technical SEO, content and measurement, or you can contact MattDarm for a focused review.
Frequently Asked Questions
Does Google use llms.txt for AI Overviews or AI Mode?
Google's current guidance says website owners do not need new AI text files and specifically gives llms.txt as an example of something to ignore. Google says the normal Search technical requirements and useful content practices apply to AI Overviews and AI Mode.
Does ChatGPT require an llms.txt file?
OpenAI's published crawler documentation describes robots.txt controls for OAI-SearchBot and other user agents; it does not state that llms.txt is required for ChatGPT search. Make important pages accessible to the crawler you intend to allow, but do not treat the file as a guarantee of retrieval or citation.
Will adding llms.txt harm my SEO?
A small valid text file is unlikely to change ordinary rankings by itself, but it can waste attention, become stale or expose draft and private-looking URLs if maintained badly. The larger risk is believing it replaces crawl access, indexing, internal links, accurate content or independent evidence.
Should I delete an existing llms.txt file?
Not automatically. Check whether any genuine customer, tool or internal process uses it, whether its links are current and whether it exposes anything unintended. If it has no proven purpose, removal or simple maintenance is a business choice, not an urgent Google SEO fix.
What should I do instead of chasing AI SEO files?
Confirm crawl and index status, return useful server-rendered or rendered HTML, improve direct answers and evidence, connect articles to service pages, maintain accurate structured data where relevant and build trustworthy third-party mentions. Measure citations, referrals and enquiries before deciding what worked.
Where llms.txt Sits in the Priority List
llms.txt may be useful to a tool that deliberately supports it. That is a narrower claim than “this gets you into AI search”.
Follow the current platform documentation. Keep important pages accessible and genuinely helpful. Build evidence that can be checked. Then measure whether the business is cited, visited and contacted. If you test llms.txt after that, label it as an experiment rather than an SEO guarantee.
