Your documentation is lying to you. And so are your validations

This summer I opened one of our own documents describing, in full detail, an infrastructure that had been switched off for months. Addresses, diagrams, which service talked to which. It was not badly written: it was perfectly written. The only thing wrong with it was that it was no longer true, and nothing in the document said so.

We found it because we were migrating a client's documentation into a new system. And that is the point: nobody found it by reading. It had been sitting there for months, available for someone to make a decision with complete confidence.

Documentation does not age: it expires

Text that ages shows it. Text that expires does not. The underlying reason is that the format we write in has no way of expressing the difference between three things that have nothing in common: I checked this, here is how to check it and I am assuming this.

Six months later all three read the same. That is where the bad decisions come from, and not through carelessness: through badly calibrated confidence.

Three states that never blur

The tool we built forces every claim to declare how it is backed. Measured: someone ran the command and the output is stored. Verifiable: not captured, but there is the command to check it in a minute. Hypothesis: believed, not checked, and it says so.

What stops being true is not deleted

When something changes, the new information supersedes the old and the old one stays. It sounds counterintuitive, but knowing why the previous answer looked right is usually exactly what you need while diagnosing. Deleting it throws that away and leaves no trace that anyone believed it.

The third answer

Freshness probes re-run what was captured and answer three things: same, changed, or could not check. That third one has been the most useful. A tool that turns a network timeout into an "all fine" is worse than no tool at all: it manufactures confidence out of a connection failure.

Asking another model, and calling it an opinion

One of the probes does something different: it shows the claim to a different model and asks whether it still holds. It deliberately does not send our reasoning. Give a model your reasoning and it tends to agree with it, and then the check stops being independent without stopping looking like one, which is the worst possible combination.

The same principle, applied to the work

If you demand evidence from documentation, it is odd not to demand it from what you do with it. So the repository is not only the knowledge layer: it is 44 skills and eight agents built on the same criterion — a check that cannot fail is not a check.

Grouped by the sentence you would say out loud, not by category. Twenty-one are ours and each carries an incident that cost real time; the other twenty-three are adapted from excellent work by Matt Pocock, with his MIT licence and copyright intact.

Eight agents, and none of them validates itself

There is one agent per kind of work, each with its own checklist. The one that implements does not validate its own work. The tests one does not look at whether they pass, it looks at whether they prove anything. The front-end one never reviews CSS: it reviews the rendered page, with a screenshot. And there is a devil's advocate whose job is to stop the change; approving is not in its description.

Each agent also declares which model it runs on, because most review work does not need the largest one. The rule behind it fits in a line: save where being wrong is cheap, never where it is expensive.

Which gives the one hard rule of the assignment: nothing at the highest level runs on the small model.

And let a command decide it, not you at eleven at night

Deciding how much validation each change deserves, change by change and under pressure, is exactly how you end up validating nothing: the decision always gets made by whoever most wants the answer to be "not much". So a command makes it, against a table written when nothing was on fire.

$ validated-memory route "small tweak to the checkout flow" \
      --path src/billing/refund.py

Level 3 — 3 validations

  matched money: billing, checkout, refund
  matched touched path: billing

  model   : opus
  effort  : high
  agents  : security-reviewer, data-reviewer, devils-advocate

"Small tweak" describes an intent. The words and the paths describe risk, and only one of those two gets a vote.

How many validations a change needs

This is not argued change by change, because arguing it is exactly how you end up validating nothing. It is decided in advance, from the kind of work:

Independent means something specific: whoever validates does not receive the reasoning of whoever implemented. And reviewing yourself later does not count, because blind spots travel with the person.

What does NOT count as validation

This is probably the most useful part of the whole repository, and it is a list of phrases that sound like checks without being one. Each of them let something through once:

It applies it to itself

A tool that demands evidence has no authority whatsoever if it ships unverified itself. So its CI validates its own memory with its own CLI, checks that it still has no runtime dependencies, and passes 286 tests. If any of that fails, it does not merge.

It is published and you can use it

We released it under the Apache-2.0 licence. It was written by Juan Carlos Vázquez at everyWAN. It works as a Claude Code plugin, and the CLI runs on Python alone with no dependencies: if you have python3, you are done.

It does not solve getting documentation written. It solves making it say how it knows what it says, once written. Which was the missing part.