The study

What we promise, and where it is verified

Some promises are true by construction — there is no server, so there is nothing to measure — and others are claims about behaviour, which are only verified by measuring them. Confusing the two is the easiest way to believe you have verified everything.

The thesis

What is decidable is decided in code; the rules remove the model's typical errors; what is left for the model is only what requires understanding a language. With this division, a small model does work it could not do on its own.

No measurement in this study asks a model to judge another model: a judge costs as much as the work it judges, and it moves the problem instead of solving it. What is counted, is counted — a contradiction is expected or not by construction of the case, a number is either in the notes or it isn't, a quotation either survives a character-by-character comparison or it is invented.

Code alone never gets it wrong

Eighteen thousand cases, three thousand per language, no model loaded: only fact extraction, pairing and the suppression rules.

Seven thousand three hundred and thirty-three alarms, none false. And not because the system is cautious to the point of silence: alarms are 41% of the cases. What this means must be stated precisely: the cases are not independent observations — the generator has a few dozen combinations repeated many times — so the correct reading is zero errors on everything the generator can produce, in six languages, not an estimate of the chance of being wrong on real notes.

The same code, however, stops at 0.815 on recall: almost one conflict in five goes unseen, and that is not a matter of polish. Implicit conflicts — “we're taking the train back” against “I booked the flight home” — leave no measurable fact behind. There you have to understand what the sentences mean, and that is the model's job.

What it costs, in real resources

2.73 GB is less than a browser with a dozen tabs. Everything this study measures fits inside it — extraction, rules, judgement, summaries, search — and nothing leaves the Mac. There is also a resource that isn't measured in gigabytes: on this corpus almost half the cases don't produce a single pair to send to the model.

The rules are worth more than model size

The ablation: same cases, same model, same notes, with only the deterministic suppressions switched off. Precision drops from 0.986 to 0.818, and false alarms go from 1 to 16. A few dozen lines of linguistic markers remove fifteen false alarms out of sixteen — and they cost less than nothing, because without them the model receives 16% to 19% more calls. We also tried the opposite, teaching the hard distinctions in words inside the prompt: over a thousand cases, 0.703 against 0.705. Moving a threshold does not teach a distinction.

The line between code and model was drawn in the wrong place

This is the result the study did not expect to find, and it emerged from classifying the errors rather than counting them. The line was wrong at two opposite points.

In neither case is the model, the prompt or the size touched: what moves is the line of who decides what. And the difference between the two models is not that the small one understands less — it is that it trusts less, and its caution was eating the code's work.

A recalculation is not a measurement: the rule was written and the nine hundred cases were run again from scratch. The prediction hit both figures — but it also showed that the first version covered one branch out of two, because three conflicts were lost not to a denial by the model, but because its quotations did not survive the comparison with the notes.

Summaries and search

One hundred and forty-four archives, twelve per language per model. Half the questions have no answer in the context, and that is the half that counts: a model that answers anyway, with a figure that isn't there, produces something that looks like your own note and isn't.

On search the two models are indistinguishable, and it is the result that matters most for a personal archive: one hundred and forty-four chances to invent, none taken. On summaries the size shows, and that is the real price of the Essential profile — on a wiki page an invented number weighs more than a missing one, because the page is what you re-read instead of the notes.

The three profiles

A profile is not a model: it is an assignment of tasks to two models. Pages and contradictions go to the intense model, search to the fast one.

Choosing Essential does not buy you worse contradictions — it buys you less complete pages.

What this study does NOT prove

• It does not prove it works on your notes. The corpora are constructed: the cases are invented, the contradictions are planted. Real notes are sloppier, more elliptical and more ambiguous.
• The cases are not independent observations. Eighteen thousand cases are a few dozen combinations repeated many times.
• The corpus is easier than reality at one decisive point: the sentence carrying the fact always begins with the subject, which is the best possible condition for the extractor. The 1.0000 precision everything rests on is measured where extraction works best.
• The languages are not equivalent. Italian has a thousand cases behind it; the other five have this study and a hand-written gold set of sixteen cases.

A wrong measurement, and how it was found

The reading numbers were redone from scratch. The first version of the bench was calling the previous engine's prompts: the code compiled, the tests passed, and the numbers belonged to a different pipeline. The suspicion came from a result too clean to be true — the small model wrote in the wrong language twelve times out of twelve in English. It is worth saying because it is the kind of error a study should not contain and that no automatic check would have caught. The wrong reports stay in the repository next to the good ones: they serve to show the difference, not to support a conclusion.

What this study says to the references in The idea

• Bush — The value lies in the links, not the documents. In the Memex the trails are drawn by the user; here they are proposed by the code, and the question becomes how far you can trust a link you did not draw. The number: zero false conflicts proposed in eighteen thousand cases.
• Luhmann — Structure emerges from the links, it is not designed. But Luhmann made the links by hand and knew why; here the why has to be shown — which is why every contradiction carries the two literal sentences that ground it.
• Ahrens — Rephrasing is already thinking, so the permanent note must be written by the person. That is exactly why the wiki does not flow back into the notes: if the model's text came back in, the app would be rephrasing on your behalf.
• Park — Reflection consolidates, or volume smothers meaning. This study adds a detail that work does not address: consolidating is the moment when invention happens. Reflection without a check on the numbers consolidates the errors too, in a more authoritative form than the one they were born in.
• Karpathy — The closest proposal to what we do: a wiki the model maintains, where it “reads, distils, updates and flags conflicts”. On that last point our measurements say the opposite, and it is the result below.

Asking the model to flag conflicts while it writes the page is the first thing we tried, and it is the thing that works worst of all: four conflicts found out of forty. Not because of model size — a verification task slipped inside a writing task is done badly by anyone. Taking it away from the model and giving it to the code: recall 1.000 and precision 0.911.

In a wiki maintained by an LLM, conflict detection is not a job for the LLM.

And one promise this study does not touch

The idea says that “the system doesn't give you back what you know — it makes you write what you didn't know you thought”. It is the most ambitious claim, and there is no number here that supports it. You can measure whether a contradiction is real, whether a number was there, whether a quotation holds. You cannot measure whether a question made you think: only the person reading it knows that, on their own notes, a few months later. A study like this one can at most remove the reasons the promise would fail for certain.

Read the idea →