{"path":"research/session36-the-keeper-and-the-order-of-asking.md","content":"# Session 36: the keeper of the seven keys, and the order in which a machine is asked\n\n**Date**: 2026-09-22 · **Type**: session record · **What it was**: a specification that had been\nwritten down for nine days and never built, a defect in how every newer judgement pass asks its\nquestion, and a number on the claim page that read as a probability and was not one.\n\n---\n\n## 1. The seven-key check existed as a ruling and nowhere else\n\nThe founder asked *\"Who is the keeper of the seven keys? … Has the seven-keyed pass been\nimplemented?\"* Measured: the keys appeared in exactly two places in the code — `sameness.py`\ndefining them, and the API serving them to the sitting's page. **No machine pass assigned one.**\n\nThe first answer given was that they were *\"purely your answer sheet\"*, and that was **wrong about\nthe design while being right about the code**. The ruling of 2026-09-13 specifies them as a machine\ncheck:\n\n> the pairwise seven-key check runs **first** on every candidate pair … and scheme-bearing\n> relationship classification runs only on pairs the check has already found distinct and related.\n> One classifier, one order.\n\n`auto_connect` had shipped **one** of the seven on 2026-09-21 — *same*, as a yes/no at step zero —\nand asked none of the other six. **A specification can sit in a ratified doc for nine days, be\ncited approvingly, and still be unbuilt**, which is the uncollected-idea failure with a ruling\nattached rather than merely a good idea.\n\n`deliberus/keeper.py` is now that pass, named by the founder after the Helloween album.\n\n**Its first live run went at the hardest pair in the corpus** and behaved correctly in the way that\nmatters: on the two Wall Advisory Opinion renderings at 0.889 cosine — where the corpus's own\nanalysis records that *\"the stance lives only in the verb\"* — it returned `confident=False` with\nthe swap check **inconsistent**. It declined to be sure, and flagged that its answer moved when the\nclaims were exchanged, on the one pair the documentation predicts is hardest. It also did not\nanswer `same`, which is where the old four-way classifier sent that pair, into a support link.\n\n## 1b. The first real run: `overlaps` is a support signal filed as a decomposition proposal\n\n**Measured 2026-09-22**, the keeper's first run over a whole source (the AI-doom diagram, 75\ncandidate pairs against the live corpus): **64 pairs came back `overlaps`, 1 an attack, and exactly\n1 edge was created.** A source that arrived claim-rich and argument-poor came out of auto-connect\n*less* connected than a run without the keeper would have left it, which inverts the purpose of the\nrun.\n\n**⚠ The first diagnosis written here was wrong and is corrected below.** It read the 85% firing rate\nas a nearest-fit shrug — the model reasoning its way to a support relation and dumping it in the\nclosest key because evidence-to-hypothesis has no home in the seven keys. That was asserted from\nfive sampled reasons **without reading the prompt**, and the prompt refutes it:\n\n> `overlaps` — establishing one would partly advance the other, but neither contains the other\n>\n> **OVERLAPS is judged by EVIDENCE, not by topic.** Two claims about the same subject with no\n> evidential bearing on each other are unrelated. The question is whether establishing one would\n> partly establish the other.\n\nSo *\"B could serve as a concrete instance supporting A's general claim\"* is not a label that\ncontradicts its reasoning. It is the definition, met exactly. **The verdicts are correct, the\nreasoning is sound, and the key set has not broken on a new register.** The lesson is the ordinary\none: a label was judged against what it means in the literature the set was coarsened from\n(MacCartney & Manning's set-theoretic relations between sentences) rather than against what this\nsystem's own prompt says it means.\n\n**What is genuinely wrong is one line of routing.** `SEPARABILITY_ANSWERS` contains\n`(\"a_contains_b\", \"b_contains_a\", \"overlaps\")`, so an `overlaps` verdict returns as a separability\nproposal before `classify_pair` is ever called. But by the prompt's own definition, *overlap is\nevidential bearing short of containment* — which is what `SUPPORTS` means. **Containment is a\nseparability signal; overlap is a support signal, and it was put in the wrong bucket.**\n\nThat makes the 64 verdicts **good data lodged in the wrong place**: 64 identified evidential\nrelations sitting in the graph as decomposition proposals, none of them an edge.\n\n**Proposed, not done** (the ordering is a founder ruling and this changes live extraction\nbehaviour): **remove `overlaps` from `SEPARABILITY_ANSWERS`**, so the pair records its verdict and\nthen continues to the scheme classifier, which names the support and its scheme. Containment still\nstops, because a part-of relation is exactly what the machine must not write as an edge on its own.\nOne line, and the reason is now the right one.\n\n## 1c. The merged reading, and what its A/B found instead\n\nBuilt 2026-09-22 behind `DELIBERUS_MERGED_PAIR` (opt-in, default off): one call answering both the\nseven-key question and the relation, keeping the ruled order by field order, with the keeper's\ninstruction block embedded **verbatim**. Twelve real pairs, run through both paths.\n\n*⚠ 2026-10-07: measured again on 54 pairs of a real run. The merged reading saves nothing in money\n(6.4 cents a pair against 6.3) and gave the same relation as the graph on 39 of 54, where the\ntwo-call path read a second time gave 46. The switch stays off:\n[auto-connect-upgrade.md](auto-connect-upgrade.md) § 2c, What the step costs.*\n\n**The result was not the clean agreement the stored-verdict sample predicted: 2 agree, 6 disagree,\n4 stop.** And the shape of the disagreement is the finding.\n\n**Four of the six are `two-call = same` on pairs the merged reader judged `overlaps`.** The keeper\nhad already asked the sameness question and answered *distinct*. The standalone classifier then\nre-asked it — `same` is still option 0 on its menu — and answered *same*. In the live two-call path\n`auto_connect` acts on `result.relationship == \"same\"` by returning a merge candidate, **so the\nclassifier's answer overrides the keeper's on precisely the question the ordering ruling exists to\nsettle first.**\n\nThat is a defect in the path running in production, surfaced by building its replacement. Which of\nthe two is right is a separate question and not answered here; the prior favours the keeper, whose\nprompt is tuned for this question, over a one-line option on a relational menu. Either way the\ncurrent arrangement asks sameness twice and lets the less careful asker win.\n\n**Two corrections to figures I had quoted.** The stop rate here was **4 of 12 (33%)**, not the 13%\nmeasured on the earlier source — so the merge saves materially less than the ~46% I estimated, since\nthat arithmetic assumed almost every pair reached a second call. And the earlier agreement figure\n(128 of 140) was computed over *stored* verdicts, which only exist for the containment and overlap\nkeys; it could not see this disagreement class at all, because a pair the classifier calls `same`\nis written as a merge candidate and never as an edge.\n\n**Held, not flipped.** The flag stays off. The evidence changed what the question is: not *should we\nmerge to save calls*, but *why does the classifier get to re-answer sameness, and which answer is\nright*. The second is a ruling, and the parts-review machinery is the natural way to get it, since a\nstratified sample of these exact pairs would say which reader to believe.\n\n## 2. The order of asking, and why the defect was invisible\n\n**Field order is generation order.** A structured response that emits its verdict before its\nreasoning has already committed by the time it writes the justification, so the reason can only\nrationalise. The judge-design literature is unanimous; one published case study moved accuracy\n96% → 98% on that change alone.\n\n**The audit corrected the alarm in a useful direction.** The extraction pipeline had led with\n`reasoning` since March — `auto_connect`, `decompose_claim`, `models.py` all do it right. What had\ndrifted were six newer judgement passes, which simply did not follow the house convention:\n\n| Pass | What it decided before reasoning |\n|---|---|\n| `terminus_llm` | the horizon kind |\n| `attribution` | the role |\n| `reported_speech` | **everything** — reasoning was the last field |\n| `concept_backfill` | the sense |\n| `decomposition_judgement` | the kind and the relevance, **through the full 197-edge run** |\n| `temporal_extraction` | the date, before what the date rested on |\n\n**The convention existed and nothing enforced it**, which is the whole finding. Prose in a module\nheader is not a guard. `tests/test_reason_before_verdict.py` is, and it **earned its place on its\nfirst run by catching the sixth instance a manual audit had missed** — the date one, where it\nmatters more than most, since the honest answer is usually null and a model that has already\nwritten a date will not retract it.\n\nThe guard carves out API request bodies, where a human decided before they typed, and names the\ncorrected passes explicitly so a refactor that reverts one fails on its own line.\n\n## 3. A test must never reach a model, and the only symptom was the clock\n\nWiring the keeper into `auto_connect` left four tests patching only `classify_pair`. Every one of\nthem began calling a real model. **Nothing failed in a way that named the cause** — the suite went\nfrom 6 seconds to 3.5 minutes and that was the entire signal.\n\n`llm_call` now refuses when `DELIBERUS_NO_MODEL_CALLS` is set, which `tests/conftest.py` sets for\nevery run, with the message naming what to patch. The two test files that drive `llm_call` itself\nlift it explicitly and say why. Suite back to 6.3 seconds.\n\nThis is the same discipline as the database isolation that conftest has carried since March, and it\narrived the same way: a real incident whose only tell was something looking slightly off.\n\n## 4. A cosine shown as a percentage reads as a probability\n\nThe founder, on finding *\"85% match\"* on the claim page: *\"Risking showing something as certain in\nthe UI when it's not very certain at all is bad generally.\"*\n\nThree things were wrong and only the first is obvious.\n\n- **A percentage reads as a probability of sameness.** It was a cosine distance multiplied by a\n  hundred, and **nobody has measured any relationship between the two quantities.**\n- **The number runs highest exactly where it is least trustworthy.** A claim and its flat\n  contradiction sit almost on top of each other in embedding space, because the models barely\n  register the word *not*.\n- **The range made it worse.** All 56 of those links fell between 0.80 and 0.889, so they rendered\n  as a narrow 80–89% band that looked like a meaningful gradient and was not one.\n\nThe card stays, because clicking it travels and that is a real move. The arithmetic and the word\n*match* are gone.\n\n## 5. The sitting, reshaped by the founder's own question\n\n*\"Do we really need the sameness sitting, or could we perhaps make do with some stickprov from the\njudgments of the keeper?\"* — mostly yes, and the reshape is better than either alternative. The\nband design was built when no machine verdict existed, so it showed the crudest honest signal.\n\n**Three things a plain sample would have lost**, each of which is what makes the sitting a\nmeasurement rather than a vibe: the **blind subset** (reviewing a verdict anchors you on it — and\nthe blind pairs now withhold the keeper's *reasoning and confidence* too, since a reader told *the\nmachine was unsure* is anchored just as surely as one told the answer), the **seeded wrongs**, and\nthe **shared salt**.\n\n**And the stratum easiest to leave out is the control.** Reviewing only the pairs the keeper found\nhard would say nothing about the confident majority — which is exactly the population an auto-merge\nwould act on without asking.\n\n---\n\n## 6. The founder's words, and what each one changed\n\n**The question that found the unbuilt specification.** After two of my answers had framed the seven\nkeys as a human answer sheet:\n\n> *\"Sigh. I guess I'm confused since IF these seven keys have been identified, couldn't an LLM\n> simply use them in a pass to judge similarity very exactly? Is this not reliable or what? Have we\n> had this idea and discarded it? What gives?\"*\n\nThe instinct was right and my framing was wrong. **The corpus had ruled it nine days earlier and it\nwas simply unbuilt** — which no amount of reading my own summary would have surfaced, because the\nsummary was the thing that was wrong.\n\n**The correction that stopped the architecture being described as worse than it is.**\n\n> *\"Ok but about this sameness checking - what is planned long-term thus far is not only\n> embedding-similarity, right? What has been discussed as the path forward in the corpus?\"*\n\nMy sentence had made cosine sound like the plan of record. It never was: `auto_connect` has been\ntwo-stage since March, with embeddings only as the candidate net. **A true statement about one\ncomponent can misdescribe the system**, and the tell is that it made the design sound naive.\n\n**The display principle, stated generally rather than about this widget.**\n\n> *\"What is this link being drawn? If it's not very certain, then what is the purpose of such a\n> rendered link? Risking showing something as certain in the UI when it's not very certain at all\n> is bad generally.\"*\n\n**The naming.** *\"Should be named after the Helloween song ;P\"* — hence `keeper.py`. The project's\nown rule prefers self-documenting names over cleverness, and this one survives it: in a codebase\nwhere *the seven keys* is established vocabulary, *the keeper* of them is the plainest possible\ndescription of what the module does.\n\n**The autonomy grant, and its scope.** *\"Fix all bugs, wire it in everywhere it's relevant, rerun\nstuff, do what you think is wise 🙂 Proceed.\"* — which is what licensed wiring the keeper into the\nlive extraction path, a change I had explicitly flagged the turn before as his to make.\n\n## 7. What was considered and rejected\n\n**Two sittings rather than one.** A keeper stickprov *beside* the existing sameness sitting.\nRejected because the blind subset, the seeded wrongs and the shared salt would each have to be\nduplicated or lost, and they are what make the sitting a measurement rather than an impression.\nReshaping what the one sitting draws from keeps all three for free.\n\n**Jev as the sameness judge, in a seven-way pick.** The founder's hypothesis was that a decision\nmodel would fix the negation blindness. Researched and **refused on the vendor's own evidence**:\nnegation is on TypeSafe's published jaggedness page (*\"double negatives and indirection degrade\nit\"*, *\"scoping words, negations and implied conditions are read literally\"*). A seven-way semantic\npick is also its worst measured regime — nearest to Banking77 at 76.0, the one public set where\nboth comparison models beat it. What survives is a **narrow yes/no contradiction check** in front\nof the cosine, which is its best regime, and it is filed in\n[jev-and-the-judgement-layer.md](jev-and-the-judgement-layer.md) § 3d rather than discarded.\n\n**Reordering every model the field-order audit flagged.** Fifteen were flagged; **nine were false\npositives found by reading rather than by the script**. Four are API request bodies, where a human\ndecided before they typed and there is no generation order at all; two are synthesis models whose\nleading fields are retrieved state rather than a verdict; three were flagged on a secondary field\npair inside a model that already led with `reasoning`. The guard encodes those carve-outs\nexplicitly, so the exemptions are a claim someone can dispute rather than a silence.\n\n**Writing decomposition edges from the keeper's containment verdicts.** The obvious wiring, and\nrefused: a machine writing `DECOMPOSES_INTO` from a pair judgement is exactly what put 84\nnon-relations in the graph. The verdict is recorded on the resemblance link; the edge stays a\nperson's to make.\n\n---\n\n## Cross-references\n\n`deliberus/keeper.py` · [auto-connect-upgrade.md](auto-connect-upgrade.md) § 2c (the ruling and\nwhat shipped) · `tests/test_reason_before_verdict.py` ·\n[the-residual-error-taxonomy.md](the-residual-error-taxonomy.md) (where the order-of-asking defect\nbelongs as an error class) ·\n[soft-canonical-clustering-and-reversible-merge-semantics.md](soft-canonical-clustering-and-reversible-merge-semantics.md)\n(the seven keys and the review protocol) ·\n[session35-the-verification-night.md](session35-the-verification-night.md) (the night before)\n"}