Blog · What the free audit finds
The Four Files Nobody Links To
Every website has four plain-text files no visitor ever sees: robots.txt, sitemap.xml, security.txt, and llms.txt. Search engines, security scanners, and AI agents read them on every visit.
What each one is, who actually reads it, what to put in it, and the one rule that keeps all of them safe. We went through this list on our own site this week and found gaps, so the free audit now checks for them.
Files for machines
Every website carries a few plain-text files at fixed addresses that no page links to. A visitor never sees them. Search engines, security scanners, password managers, and now AI agents fetch them on every visit and act on what they find.
There are four worth having: robots.txt, sitemap.xml, security.txt, and llms.txt. Three are long-standing conventions. One is a proposal. All four take about an hour, and all four are free.
We went through this list on our own site this week, while writing the recommendations we give clients, and found two of the four missing and one of the others thinner than it should have been. So this article is also a record of fixing our own gaps first. The free audit now checks all four on any site you point it at.
robots.txt: what crawlers may read
robots.txt lives at the root of the site, at yoursite.org/robots.txt. It tells crawlers which parts of the site they may read and which they should leave alone, and it names the sitemap. Google, Bing, and every well-behaved crawler fetch it before anything else.
It is a request, not a lock. A crawler that chooses to ignore it can. That leads to the one rule that matters for this file: never list a path you would not want a stranger to find. A line that says Disallow: /board-documents/ does not hide the board documents. It publishes their address to anyone who reads the file, and people do read it. Anything sensitive belongs behind a login, and its path belongs nowhere in this file.
The second most common mistake is the staging block. A site under construction gets a Disallow: / rule so search engines ignore it, and the rule survives the launch. The site is live, the organization is waiting for traffic, and Google has been told to stay out. We see this on sites that have been live for a year.
A good robots.txt for most organizations is short. Allow everything, keep crawlers out of the search results page, the cart, the login, and any private area, and name the sitemap.
Show the code Hide the code Text, 7 lines
User-agent: *
Allow: /
Disallow: /portal/
Disallow: /admin/
Disallow: /search/
Sitemap: https://yoursite.org/sitemap.xmlOn WordPress there is usually no physical file. WordPress answers the address itself with a default, and SEO plugins let you edit it from the dashboard. A plain text file uploaded to the site root overrides both.
sitemap.xml: the page list
The sitemap is the list of pages you want indexed, with the date each one last changed. Search engines can find pages by following links, but the sitemap is how they learn about a new page in hours instead of weeks, and it is the first thing Google Search Console and Bing Webmaster Tools ask for.
WordPress has generated one automatically since version 5.5, at /wp-sitemap.xml. Most SEO plugins replace it with a better one at /sitemap.xml. Either way, the address belongs in robots.txt, and the sitemap belongs in both webmaster tools. If you have never submitted it, that is a ten-minute task with a lasting effect.
security.txt: who to tell
When a security researcher finds a problem in a website, they want to tell someone who can fix it. Most organization websites give them no way to do that except the general contact form or a guess at an email address, and reports that go through a contact form get answered by whoever answers the contact form.
security.txt solves this. It is an internet standard, RFC 9116, and it lives at /.well-known/security.txt. It names a contact for security reports and carries an expiry date so a stale contact does not sit on the web forever. Security reviewers and the automated scanners that government and enterprise procurement teams run check for its presence as a sign that someone is minding the shop.
The whole file is a few lines.
Show the code Hide the code Text, 6 lines
Contact: mailto:hello@yoursite.org
Contact: https://yoursite.org/contact/
Expires: 2027-09-01T00:00:00.000Z
Preferred-Languages: en
Canonical: https://yoursite.org/.well-known/security.txt
Policy: https://yoursite.org/security/Two details trip people up. The date field is called Expires, and it must be no more than a year out; a file with the wrong field name or a past date is treated as missing by everything that reads it.
We found one this week on a well-run advocacy site, generated by a plugin that named the field wrong. And the file is more useful with a short page beside it, the Policy line above, that says how you handle reports and that a good-faith researcher has nothing to fear from you. Ours is at LFW.com/security/ and it is nine short paragraphs.
The .well-known folder itself is a reserved place for files like this, defined so that tools know where to look. Password managers read a change-password file from it, and the certificate system uses it to prove you control the domain. Nothing on the site links there. Only software reads it, and that is the point.
llms.txt: a suggestion, honestly labeled
llms.txt is the new one, and the one to be careful about. It is a proposal, not a standard. The idea is a short plain-text page at /llms.txt that tells an AI assistant what the organization is and where the important pages are, in a form the assistant can read without parsing the whole site.
Here is what is true about it. Crawlers request the address whether or not the file exists; ours was requested more than a dozen times in the weeks before we published it. It costs nothing to write and nothing to serve. And nobody has shown that it changes search rankings or what an AI assistant says about you. Anyone who tells you otherwise is guessing.
So the honest position is that it is a courtesy to machines. It cannot hurt, on one condition: never put anything in it that you would not put on the homepage. It is public, like every file in this article.
Ours lists what LFW does, who we serve, and a dozen reference pages. It took twenty minutes, and every link in it is checked on every deploy.
What the free audit checks now
As of this week, the free audit fetches all four files on any site you point it at and reports what it finds, with the severity set honestly. A missing robots.txt or sitemap is minor. A robots.txt that blocks the whole site is serious, because the site is invisible to search while it stands. A robots.txt that names private-looking paths is flagged as a security finding, because it is one. A missing or expired security.txt is minor. A missing llms.txt is a suggestion, and the finding says so in its title.
One more caution, because the audit will say it too. None of these files is a security control. They are signs of a site that someone is looking after, which is what a security reviewer, a search engine, and a procurement officer are all trying to establish. That is worth an hour.