We Retired llm.json. The Crawlers Kept Asking.

Published September 22, 2026

We put llm.json back on every site in the portfolio this week.

Two months ago we took it away on purpose.

The crawler logs are why it came back.

The Theory Was Good

The Digital Karma Federation is the set of machine-readable files every site in this portfolio publishes about itself. A manifest, a catalog, a health file, a few others. The standard lives on Digital Karma Web.

In v8.0, on July 29, we retired /llm.json.

The reasoning was clean. The manifest already said who the site was, what it published, and where everything lived. A second file saying the same things is a second thing that can drift out of date.

One source of truth. Fewer files. Less to break.

Very tidy.

Also wrong.

Nobody Told the Crawlers

A spec is a document people read.

Crawlers read the URLs they have learned to request.

On September 9 we found portfolio sites still serving an llm.json that had not been regenerated in 51 days. Bots were still fetching it. We had not stopped anyone from asking. We had only made sure the answer went stale.

So we pulled the server logs from the Digital Karma Data Warehouse. Every GET request across the portfolio, August 18 through September 22. That is the full 35 days of raw logs the warehouse keeps.

Read the first two rows again.

The file we retired got asked for more often than the file we told everyone to use instead.

There is debate around theory. There is no debating receipts.

What We Did About It

We did not just put the old file back.

The old file was part of the problem. Hand-made, forgotten, stale.

v8.1 changed one rule. llm.json is never written by hand. It is generated every night from the manifest the site already publishes, so it cannot disagree with it.

That sounds small until you count the generators.

This portfolio runs about twenty different versions of the script that builds these files. Some of them were deleting llm.json every night, because v8.0 told them to.

Instead of teaching twenty scripts a new trick, there is now one builder. Each site's own generator runs first. Then the builder reads what it produced and writes llm.json from that.

155 sites. 1,085 files at the new version. Checked every night by an audit that also clicks through every link between sibling sites to make sure none of them redirect.

That audit found something too. Sites were linking eleven of their siblings through a www address those siblings do not use, so every one of those links took a redirect. Fixed the same day.

The Second Door

While we were reading the logs, a second request showed up that we were not answering at all.

/llms.txt. Plural. It is a markdown convention from llmstxt.org, a short page that tells a language model what a site is and links to the parts worth reading.

2,116 requests. Roughly 960 of them got a 404, because almost none of our sites had one.

Same builder. Same rule. Every site has one now, built from the same manifest and catalog.

A site that already wrote its own llms.txt by hand keeps it. The builder only replaces files it wrote itself.

What the Numbers Do Not Prove

A request proves a crawler asked for a URL, and what the server answered.

It does not prove the crawler used the file, quoted it, trained on it, or ranked anything because of it.

The llms.txt number is the soft one. Most of those 404s came from tools and agents we could not identify. The named AI crawlers mostly asked for llms.txt on the few sites that already had one.

We did not need proof of use to make this call.

A small, accurate, generated file costs nothing to serve. A 404 costs you the chance to be read at all.

The Part Worth Keeping

The mistake was not retiring a file.

The mistake was deciding what crawlers should want, instead of looking at what they asked for.

We build these sites to be read by machines. The machines vote with their requests. When the spec and the logs disagree, we go with the logs.

Then we fix the spec.

The full technical write-up, with the file formats and how to run the builder on your own site, is on Digital Karma Web: llm.json and llms.txt: Why v8.1 Brought Them Back. The specification is here.

And if you want to see the real thing, this site publishes its own at /llm.json and /llms.txt. Both are rebuilt every night.