The four files nobody links to

Files for machines

Every website has a few plain-text files at fixed addresses that no page links to. Visitors never see them. Search engines, security scanners, password managers, and now AI agents fetch them on every visit and act on what they find.

Four are worth having: robots.txt, sitemap.xml, security.txt, and llms.txt. Three are long-standing conventions and one is a proposal. All four together take about an hour, and they cost nothing.

We went through this list on our own site this week. Two of the four were missing and a third was thinner than it should have been, so we fixed our own gaps first. The free audit now checks all four on any site you point it at.

robots.txt: what crawlers may read

robots.txt lives at the root of your site, at yoursite.org/robots.txt. It tells crawlers which parts of the site they may read and which to leave alone, and it points them to your sitemap. Google, Bing, and every well-behaved crawler fetch it before anything else.

Crawlers don't have to obey it, and some don't. That leads to the one rule that matters for this file: never list a path you wouldn't want a stranger to find. A line that says Disallow: /board-documents/ doesn't hide your board documents. It publishes their address to anyone who opens the file, and people do open it. Anything sensitive belongs behind a login, and its path doesn't belong in this file at all.

The second most common mistake is the staging block. A site under construction gets a Disallow: / rule so search engines ignore it, and nobody removes the rule at launch. The site is live, the organization is waiting for traffic, and Google has been told to stay out. We see this on sites that have been live for a year.

For most organizations a good robots.txt is short. Allow everything, keep crawlers out of the search results page, the cart, the login, and any private area, and name the sitemap.

User-agent: *
Allow: /
Disallow: /portal/
Disallow: /admin/
Disallow: /search/

Sitemap: https://yoursite.org/sitemap.xml

On WordPress there is usually no physical file. WordPress answers the address itself with a default, and SEO plugins let you edit it from the dashboard. A plain text file uploaded to the site root overrides both.

sitemap.xml: the page list

The sitemap lists the pages you want indexed, with the date each one last changed. Search engines can find pages by following links, but the sitemap is how they learn about a new page in hours instead of weeks. It's also the first thing Google Search Console and Bing Webmaster Tools ask for.

WordPress has generated one automatically since version 5.5, at /wp-sitemap.xml. Most SEO plugins replace it with a better one at /sitemap.xml. Either way, put its address in robots.txt and submit the sitemap to both webmaster tools. If you've never submitted it, that's a ten-minute task that keeps paying off.

security.txt: who to tell

When a security researcher finds a problem in your website, they want to tell someone who can fix it. Most organization sites give them only the general contact form or a guess at an email address. A report sent through the contact form gets read by whoever happens to answer the contact form.

security.txt fixes this. It's an internet standard, RFC 9116, and it lives at /.well-known/security.txt. It names a contact for security reports and carries an expiry date, so an out-of-date contact doesn't stay on the web forever. Security reviewers, and the automated scanners that government and enterprise procurement teams run, check for it as a sign that someone is looking after the site.

The whole file is a few lines.

Contact: mailto:hello@yoursite.org
Contact: https://yoursite.org/contact/
Expires: 2027-09-01T00:00:00.000Z
Preferred-Languages: en
Canonical: https://yoursite.org/.well-known/security.txt
Policy: https://yoursite.org/security/

One detail trips people up. The date field is called Expires, and it must be no more than a year out. Anything that reads the file treats a wrong field name or a past date as if the file were missing. We found one this week on a well-run advocacy site, where a plugin had generated the file with the field named wrong.

The file is also more useful with a short page beside it, the Policy line above, that says how you handle reports and that a researcher acting in good faith has nothing to fear from you. Ours is at LFW.com/security/ and it's nine short paragraphs.

The .well-known folder is a reserved location for files like this, so tools know where to look. Password managers read a change-password file there, and the certificate system uses it to confirm you control the domain. Nothing on your site links there, because only software needs to read it.

llms.txt: a new proposal

llms.txt is the newest of the four, and the one to be careful with. It is a proposal, not a standard. The idea is a short plain-text page at /llms.txt that tells an AI assistant what your organization does and where the important pages are, in a form the assistant can read without working through the whole site.

Crawlers request /llms.txt whether or not the file exists. Ours was requested more than a dozen times in the weeks before we published it. It costs nothing to write or serve. And nobody has shown that it changes search rankings or what an AI assistant says about you, so anyone who promises that is guessing.

Treat it as a courtesy to machines. It can't hurt, as long as you never put anything in it you wouldn't put on your homepage. It's public, like every file in this article.

Ours lists what LFW does, who we serve, and a dozen reference pages. It took twenty minutes to write, and we check every link in it on every deploy.

What the free audit checks now

As of this week, the free audit fetches all four files on any site you point it at and rates each problem by how much it matters. A missing robots.txt or sitemap is minor. A robots.txt that blocks the whole site is serious, because search engines can't see the site until it's fixed. A robots.txt that names private-looking paths is flagged as a security finding, because it is one. A missing or expired security.txt is minor. A missing llms.txt is only a suggestion, and the finding's title says so.

One caution, which the audit repeats: none of these files is a security control. They show that someone is looking after the site, which is what a security reviewer, a search engine, and a procurement officer all want to know. That's worth an hour of your time.