{"path":"research/pass-quality-and-the-choice-of-model.md","content":"# Pass quality and the choice of model\n\n**Date**: 2026-10-05 · **Type**: research on founder instruction · **Status**: the reading and the\nproposals are Claude's and unruled (ledger Arc Y), except one clause ruled the same day: the live\nsplitter does not hold its cuts, and his labelling is the check (§ 6; what that labelling runs on is\n§ 6b). The hold in § 6 is the founder's, in his words; he lifted its deploying half the same day,\nso the engine is in production, switched off.\nThe measurements in § 4 were made the same day: read-only counts on the live stores, and 18 model\ncalls through the router engine that cost 14 cents and stored nothing.\n\nThe founder, 2026-10-05:\n\n> *\"Read the microtasks-in-broad-daylight doc and the docs that were created around the same days as\n> that was and think deeply about what level of quality we need from the AI passes/microtasks in\n> Deliberus and what that says about which models we could/should use.\"*\n\nAnd while the work was under way, about the router engine built the evening before:\n\n> *\"Hold off on deploying/switching on this feature for now, let's think it through\"*\n\n**Read for this**, in full unless marked: `microtasks-in-broad-daylight.md`,\n`proving-a-pass-reliable.md` and its literature report, `split-review-and-the-founders-intentions.md`,\n`bend-and-deliberus.md`, `what-confirmed-means.md`, `what-a-link-strength-means.md`,\n`earning-the-right-to-write.md`, `the-tidying-line.md`, `machine-work-as-a-standing-property.md`,\n`the-parts-review.md`, sessions 40 and 41, `whose-tokens-pay-for-mapping.md`; and in the parts that\nbear on machine passes: `jev-and-the-judgement-layer.md` (§§ 1–3, 3e–6e),\n`major-transitions-heredity-and-evolvability.md` (§§ 3, 7, 8), `relevance-and-the-two-factors.md`\n(§§ 5–7). A *pass* below is one kind of question the pipeline puts to a model: reading a text into\nclaims, linking two claims, cutting a claim into parts, and so on.\n\n---\n\n## 0. Answer first\n\n1. **The quality we need is not one number.** A pass has to be almost never wrong in the few ways\n   that do damage here, has to say so when it cannot tell, and has to leave everything it writes\n   marked and undoable. It does not have to be right about everything it is asked.\n2. **Three levels, by what a wrong answer does** (§ 2, proposed, the numbers are his to set). At\n   most 2 in 100 for an answer that turns a meaning around, fuses two things, or adds to what a\n   named person said. At most 5 to 10 in 100 for a marked, undoable item that moves no strength and\n   that no later machine work stands on. No number at all where careful people disagree: those\n   stay proposals.\n3. **The bound is on what the machine acts on, and it is reached by declining the rest.** So the\n   question about a model is how much of the work it can take at that bound per cent spent, never\n   whether it is good in general.\n4. **The model's tier is the third lever.** Of nineteen measured failures of machine passes in\n   this project, eleven trace to how the question was posed, six to the answer form or the code\n   that received the answer, and two to a model's own habits. One of those two was the strongest\n   model in use (§ 3, Claude's classification).\n5. **Four things measured today** (§ 4). Extraction quotes its source word for word in 98 of 100\n   claims where the source text was kept promptly, on the cheap model. One real sentence with an\n   *or* in it was cut the same wrong way by three of five models, Claude Sonnet among them, and\n   kept whole by the two cheapest. The two splits written on the live site in October were cut by\n   a fallback model nobody chose and nobody had measured. And on the one open reading task tried,\n   the strong model found more than the cheap ones.\n6. **Which models** (§ 5). For a question we can pose sharply, the cheapest model that answers the\n   same twice. For a question of degree, no model, until the question is posed differently. For\n   writing passes, one named model, checked by two cheap models of other families (a *family* is\n   one maker's models). The strong model has so far earned its price in one place: reading for\n   what a text leaves unsaid.\n7. **The lasting choice is the answer key, not the model.** Models are replaced every few weeks,\n   and a bound on a pass dies with the model it was measured on. A set of cases a person has\n   labelled is paid for once and admits or refuses every later model in minutes (§ 5e).\n8. **No person has labelled anything yet.** Both review tables are empty, and none of the 51\n   machine-cut parts in the graph has been touched (counted 2026-10-05). Every model choice today\n   is provisional, and an hour of his in a sitting buys more than any change of model.\n9. **On the hold.** The analysis supports it, and names three things that would lift it (§ 6).\n   *(Its deploying half was lifted the same day. The engine is in production, switched off, and\n   the three things now bear on switching it on.)*\n\n---\n\n## 1. What the late-September documents already settle\n\nStated once so that nothing below re-derives it.\n\n| Settled | Where | What it fixes about this question |\n|---|---|---|\n| Review need not keep pace with a proven pass; what compounds is depth. After *k* machine steps built on each other the share wrong is about 1 − (1 − ε)^k | canon, ruled 2026-09-29; `proving-a-pass-reliable.md` § 1 | A tolerance has to be read per step and multiplied along the chain |\n| An error rate belongs to the task as written before it belongs to the model; answers should be as deterministic and idempotent as possible | canon, ruled 2026-09-29 | A bad rate is a design problem first |\n| A microtask's truth is one of three kinds: checkable against the text, a matter of consensus, or contested | `proving-a-pass-reliable.md` § 3 | Only the first two can have a bound at all |\n| A pass is proven by a blind random sample a person labels: none wrong in 59 bounds the error under 5 %, none wrong in 149 under 2 % | same, § 2 | What a tolerance costs in his time |\n| A bound holds for one version of a pass on one model | same, § 4 | Changing the model starts the proof again |\n| A machine checking a machine counts only as far as the two fail differently | same, § 4 | The checker should be a model of another family |\n| A pass's own confidence is not a measurement | ledger R12, open; 14 false links were written at 0.6 to 0.8 | The signal has to come from outside the answer |\n| A machine may write what raises the number of places a reader can disagree; a split part caps its whole like a person's unless the check found invention | `earning-the-right-to-write.md`, ruled in parts | Which writes are in play |\n| An unexamined supporter moves nothing; a link a machine guessed between other people's claims is presumed to carry no weight until read | `relevance-and-the-two-factors.md`, ruled 2026-09-28 | Most machine-written links are inert in the strength layer today |\n\n**His bar, in his words.** *\"how confident can we be in the machine/system instead doing that work\non its own\"* (2026-09-21). *\"IFF it works reliably, which I suspect it will, otherwise we will\nreconsider\"* (same day). *\"in whatever circumstances we can't be 105% confident in the reliability\nof the targeted prompting of the LLM-step\"* (2026-08-30). *\"done with high confidence in\nsafety/accuracy\"* (2026-09-13). *\"most error rates are amenable to different\noptimization/innovation attacks to lower it\"* (2026-09-29).\n\n---\n\n## 2. What \"good enough\" has to mean here\n\n### 2a. Six things a wrong answer can do\n\nThe same error rate is harmless in one pass and ruinous in another, so the level has to be set by\nconsequence. Six consequences occur in this pipeline.\n\n| What the wrong answer does | Where it happens | Who sees it, and how it is undone | Proposed tolerance |\n|---|---|---|---|\n| **Speaks for a person wrongly**: turns a claim's polarity around, adds an assertion, raises its certainty | extraction, split parts, minted reports of what someone said, short titles | The author, if they ever look. Undone by a correction, and finally only by the author's own word | **2 in 100**, counted on the harmful kinds only |\n| **Fuses two things**: calls two claims the same | the sameness keeper, any later merge | Hard to see, and it chains: one false *same* in a hundred becomes about one false link in ten across a cluster of ten | **2 in 100 on *same***, counted per cluster |\n| **Turns a relation around**: a reason filed as an objection | the linker, the relevance judge's direction | Invisible while the link is inert, and it flips a strength the day the link gains weight | **2 in 100** |\n| **Hides**: a screen drops a true candidate, or a pass never looks | every gate and pre-filter, every pass that runs at one door | Nobody. An absence leaves no trace | no rate on the output; **a sample the screen never touched**, to count what it dropped |\n| **Adds a marked, undoable item** that moves no strength and that nothing is built on | links guessed between other people's claims, flags, proposals | Readers, as clutter. Undone by retiring it, the old link kept visible | **5 to 10 in 100** |\n| **Puts a number on a degree** | how far a link bears, the old link-strength rating | It looks like a measurement | **none**: no bound exists, so nothing is written as a reading |\n\n**Why 2 in 100 for the first three.** Chains in this graph are three or four machine steps long:\na text is read into claims, two claims are linked, the link's weight is read; or a claim is cut,\na part is linked, the link is read. At 2 in 100 per step the end of such a chain is wrong about 6\nto 8 times in 100. At 5 in 100 per step it is wrong 14 to 19 times. And 2 in 100 is about where\nlabelled data itself stops: on product pairs the published gold labels are wrong 0.5 to 2.5 times\nin 100 (`proving-pass-reliability/literature-report-2026-09-29.md`).\n\n**Why the last row gets no number.** Five models asked how far twelve links bear gave five\ndifferent readings, and one answered *strong* to all twelve (`whose-tokens-pay-for-mapping.md`\n§ 3e). People disagree on such questions too. A rate needs a right answer to be wrong about.\n\n### 2b. The bound is on what the machine acts on\n\nA pass that is right 80 times in 100 across everything it is shown can still be right 98 times in\n100 on the answers it is sure of, if it declines the rest. The published work that makes guarantees\nof this kind does exactly that: a cheap model answers first, a dearer one takes what the cheap one\ndeclined, and the cases all of them decline turn out to be the ones people disagree on\n(`proving-a-pass-reliable.md` § 5). So the tolerance above is a bound on the answer that triggers a\nwrite, and what the pass declines goes to a person or stays a proposal.\n\nThis changes what is asked of a model. A cheap model with a sound way of declining can meet a tight\nbound and take less of the work. A strong model takes more of it at the same bound. **Choosing a\nmodel is choosing coverage per cent, at a bound fixed beforehand.**\n\n### 2c. Which way the errors lean matters more than how many there are\n\nErrors that scatter both ways wash out across a graph. Errors that lean one way are the\nconvergence illusion this project's threat model is written against. Four leans are on record:\ntoward *same* (a 2023 model was right on 88 of 100 paraphrases and 44 of 100 non-paraphrases),\ntoward more certainty than the source (unmeasured here, and the existing instrument cannot see it),\ntoward the common view at the cost of the rare one, and toward whatever the question already\nsuggests (§ 3, the link whose filing one model followed twice). A tolerance should be read on each\nside separately.\n\n### 2d. Steadiness comes before accuracy\n\nA pass that gives a different answer when asked again cannot be bounded at all. Asking twice needs\nno labels and costs cents, so it is the first test of any model on any pass. The part test\n(*if this part were false, would the claim be weaker?*) answered 45 of 45 the same twice on Claude\nSonnet. The five-step scale of how far a link bears did not, on any model.\n\n**The linking step, asked twice (measured 2026-10-07).** 54 pairs from a real run were read a second\ntime the way the step reads them, on the same model with the same instructions: the same sameness\nanswer on 42 (78%), the same relation on 46 (85%), and 20 of 24 attack links came back as attacks.\nSo about one of its links in seven would not be written the same way twice, and any cheaper reader\nof pairs is measured against 85%\n([auto-connect-upgrade.md](auto-connect-upgrade.md) § 2c, *What the step costs*).\n\n---\n\n## 3. Where the measured failures came from\n\nNineteen failures of model-asked passes are on record in this corpus with a number or a named\ninstance. Sorted by where the fault lay. **The sorting is Claude's reading.**\n\n| Failure | Measured | Fault |\n|---|---|---|\n| A fifteen-day extraction outage in April | 8 of 10 users had no successful extraction | answer form: one field typed in a way the model's reply could not satisfy |\n| The scheme classifier never said *does not fit* | 0 of 3,666 links | question: a costless catch-all sat beside it |\n| *Supports* written between two wordings of one claim | 14 links, retired | question: the menu had no *same* |\n| *Part of* written between barely related claims | 0 genuine of 16 read; 84 links | question: *unrelated* was one option of six |\n| The sameness keeper's verdicts lost | 64 of 75 on one source | code: one of its two flags was never read |\n| Merge marks on unrelated pairs | 47 of 47 | code: a placeholder read before its gate |\n| Six judging passes gave the verdict before the reason | six passes | answer form: field order |\n| The splitter's first version | 6 harmful parts of 37 | question: no rule about a clause the claim denies |\n| The relevance judge's first trial | 0.5 on 8 of 12 | question: a bare number was asked for |\n| How far a link bears | differs on every model; one answered *strong* to all twelve | question: a degree was asked for |\n| *Overlaps* | read two ways | question: our own instructions define it twice |\n| Unstated premises are sparse | 10 from 107 arguments | question: the instruction says prefer fewer |\n| The contested-concept detector misses words | `correlated`, `possibility` | question: a word list written before the corpus existed |\n| An author's own labels flattened in a hand-made map | 24 of 116 claims | **a strong model's inattention**: it read the content and not the author's own scale |\n| *Claude 3 Opus* written where the source says *Opus 4.6* | 1 of 1,144 claims (§ 4a) | **a cheap model's stale knowledge** written over the text |\n| A link's filing followed against the claims' own wording | twice, by Claude Sonnet | question: the prompt tells the judge how the link is filed |\n| An *or* cut into two asserted parts | 3 of 5 models (§ 4b) | question: the instructions have no rule for alternatives |\n| A split written on the live site by an unmeasured model | 2 splits (§ 4c) | code: a silent fallback picks the model |\n| Short titles made from the first 200 characters of a claim | 330 of 1,989 claims are longer | code |\n\nEleven question, six form or code, two a model's own habit: a cheap model's stale knowledge and a\nstrong model's inattention. In two of the eleven a model's habit showed inside a badly posed\nquestion: one cheap model answered a scale with a single word, and the strong one followed the\nfiling it was told about.\n\n**What this does and does not say.** It says where the failures that were found came from. It does\nnot say models never differ: failures of tier are the ones a project like this is least equipped to\nsee, since nobody has labelled a sample. § 4d is the first place a difference of tier showed.\n\n---\n\n## 4. Four measurements made on 2026-10-05\n\n### 4a. Is a claim's quoted passage really in its source?\n\n*(2026-10-06: a second thing code can check about a claim with no model call: which of its words are its author's and which are found nowhere in the source. `scripts/claim_wording_sources.py`; [store-the-uses-read-the-meaning.md](store-the-uses-read-the-meaning.md) § 3c, ledger AA6.)*\n\nExtraction asks the model for an exact quote from the source with every claim and stores it.\nNothing had ever checked one. `scripts/measure_quote_fidelity.py` does, with no model call, for the\n16 stored extractions whose source text was kept (1,144 extracted claims). It also flags a claim\nthat carries a number found nowhere in its source.\n\n| | Claims | Quoted word for word | Near | Partly | Not found |\n|---|---|---|---|---|---|\n| All 16 sources | 1,144 | 1,076 (94.1 %) | 16 | 34 | 18 |\n| Five sources extracted in July, text kept 36 to 37 days later | 380 | 373 (98.2 %) | 4 | 3 | 0 |\n| Two sources extracted by hand by a strong model in August, text kept the same day | 134 | 133 (99.3 %) | 1 | 0 | 0 |\n| Nine sources extracted in March and April, text fetched again 110 to 135 days later | 630 | 570 (90.5 %) | 11 | 31 | 18 |\n\n*Word for word* includes a quote cut with an ellipsis and differences of typography only. The\nolder sources score lower partly because their pages changed before the text was kept, so their\nfigure is a lower bound.\n\n**The three July deviations, read by hand**: a quote whose words were changed inside the quote\nmarks (*individuals* for *persons*), a quote field that swallowed the start of the next field, and\none word misspelled while copying. No invented passage.\n\n**The four number flags, read by hand**: one real substitution (*Claude 3 Opus* for the source's\n*Opus 4.6*, the model's older knowledge written over the text); two faithful conversions (*one\nquarter* to 25 %), one of which also added the word *federal*; one that cannot be judged because\nthe kept text is incomplete.\n\n**What it shows.** Pointing at real words is something the cheap model on the paid route already\ndoes at the ceiling, as well as a strong model working by hand. **What it cannot show**: that the\nclaim says what its passage says. A claim can quote faithfully and still turn the meaning around.\nThat half needs a second reader, and no instrument here measures it yet.\n\n**A lesson from building it.** An earlier version of the check reported 39 faithful quotes as not\nfound, all for one reason: the kept text is markdown, and *\\*Adkins\\* v. \\*Children\\*'\\*s\nHospital* holds the same words as the plain quote. Reading seventeen of the misses beside their\nsources showed it in minutes. A check that had been believed would have reported a fault that was\nits own.\n\n### 4b. One sentence with an *or*, five models\n\nThe claim, from the live graph: *\"The elimination of low-productivity, low-wage jobs as part of a\ntransition to a high-productivity economy should be viewed as a positive or necessary outcome\nrather than a negative one.\"* It asserts that the outcome is positive **or** necessary. Neither\nalternative is asserted alone.\n\n**As splitter** (`scripts/probe_split_one_sentence.py`, one call each, the splitter's own\ninstructions unchanged):\n\n| Model | Cents | What it did with *positive or necessary* | Other |\n|---|---|---|---|\n| Gemini 3.5 Flash | 1.45 | two parts, each stated as fact | the *as part of a transition* clause written as a fact |\n| Gemini 3 Flash | 0.28 | two parts, each stated as fact | also wrote *the jobs are low-wage* as a part |\n| Claude Sonnet 5 | 1.45 | two parts, each stated as fact | the clause written as a fact |\n| Gemini 3.1 Flash Lite | 0.13 | **kept together** | three parts |\n| GPT-6 Luna | 0.08 | **kept together** | cut the noun phrase into fragments |\n\nFive models, four different cuts. The three dearer ones split the alternatives. The two cheapest\nkept them.\n\n**As checker of the four parts the live site wrote** (the write gate holds a split when any part\nis answered anything but *weaker*):\n\n| Model | Cents | Would hold | Why |\n|---|---|---|---|\n| Gemini 3.5 Flash, in production on 3 October | free tier | nothing: it wrote the split | all four *weaker* |\n| Gemini 3.5 Flash, asked again today | 0.58 | the clause | *unaffected*: a different answer from the same model |\n| Gemini 3 Flash | 0.20 | the clause | *unaffected* |\n| Gemini 3.1 Flash Lite | 0.11 | nothing | all four *weaker* |\n| GPT-6 Luna | 0.07 | **both alternatives** | *\"one of two alternatives\"*; it answered in Spanish, and filed them under *supposes* for want of a fitting label |\n| Claude Sonnet 5 | 0.73 | *should not be viewed as negative*, and the clause | read the claim's own denial, correctly carried, as a denied part: a false alarm |\n\n**What it shows.**\n\n1. **The *or* error is in the instructions.** They say what to do with a clause the claim denies,\n   grants or supposes, and nothing about alternatives. The checker's five labels have no place for\n   one. The dearest model made the same cut as the cheap one.\n2. **A split is not the same on two models.** For a writing pass, *each pass names its model* is\n   a necessity.\n3. **A checker of another family did catch what the writer's own family missed.** One did. Each checker\n   also had its own miss or false alarm, so one checker is not enough.\n4. **The check is not steady on this claim.** The model that passed it on 3 October held it on\n   5 October.\n5. **Price did not track correctness.** The call that saw the *or* cost a twentieth of the call\n   that did not.\n\nOne sentence, one run each. It shows that these things happen, not how often.\n\n**Corrected the same day: an alternative can be a part. What it may not be is a required one.**\nThe proposal drawn from this probe (§ 7, Y7) first said the alternatives *are not parts*. The\nstrength code has had a third way for parts to combine since August (`deliberus/support_semantics.py`): parts tagged *alternative* are\nrival routes, the strongest one decides, and none of them caps the whole. That is the logic of\n*positive or necessary*. The fault in the live split is narrower than first stated: the splitter\nwrites every part untagged, an untagged part counts as required, and so refuting *necessary* would\nhold the whole claim down though *positive* still stood. Two repairs follow, and § 7 (Y7) now\ncarries both: keep the alternatives in one part, or cut them and tag them as alternatives. The\nreview page had the matching answer all along, key 3, *one of several routes*. The first wording\ncame from proposing a rule about parts before opening the module that says how parts combine.\n\n### 4c. Who actually answers on the live site\n\nThe direct Gemini route asks for `gemini-3-flash-preview` and, when that model has no quota left,\nis served by `gemini-3.5-flash` and then `gemini-3.1-flash-lite` without a word\n(`deliberus/extraction/client.py`). The free tier allows about twenty calls a day per model, so\nwhich one answers depends on the hour.\n\nRead from the splitter's own record on the live graph, 2026-10-05:\n\n- Nine claims hold machine-cut parts, 51 parts. Seven splits (45 parts) were cut by Claude Sonnet\n  in September under the splitter's first version. **Two (6 parts) were cut on 3 and 4 October by\n  `gemini-3.5-flash`**, when a reader pressed the split button on the live site.\n- On the second of those, the split was cut by one model and checked by another, inside one\n  request.\n- The splitter's second version was measured on Claude Sonnet only\n  (`microtasks-in-broad-daylight.md` § 3e). So the two October splits were written into the public\n  graph by a model on which the pass had never been measured.\n- One of the two is the *or* claim of § 4b. The other reads as a clean split.\n- **Of the 25 places in the code that ask a model, three keep a record of which model answered**:\n  the splitter's two calls and the relevance judge's. For everything else the question *which\n  model wrote this* has no answer in the graph.\n- **His own word on those stand-in models, 2026-08-27**, when a comparison had run on them:\n  *\"Comparing with the free tier fallback for Gemini is not meaningful. Not high quality enough,\n  cannot do the full thing.\"* That was said of whole extractions. Both live splits were cut by the\n  first of the two stand-ins.\n- **The review sitting built for the splitter held none of these six parts** until 2026-10-05\n  (§ 6b). It served 37 of the 51, all from the first version.\n\n### 4d. Reading for what a text leaves unsaid, four models\n\nThe pass that finds unstated premises was run on two sources whose claims a strong model had\nextracted by hand in August, every model given the same claims and the same text\n(`scripts/probe_unstated_premises.py`).\n\n| Model | Cents per source | Premises found, source 1 and 2 | Kind |\n|---|---|---|---|\n| GPT-6 Luna | 0.11 to 0.15 | 0 and 1 | one peripheral remark |\n| Gemini 3.1 Flash Lite | 0.28 to 0.56 | 3 and 2 | restatements of doctrine; one touches the crux |\n| Gemini 3 Flash | 0.58 to 0.73 | 3 and 4 | restatements of doctrine; two touch the crux |\n| Claude Sonnet 5 | 3.2 to 3.6 | 5 and 3 | the only stated value the argument rests on (*law should be read so as to leave no State defenceless*), a factual premise about the attack itself, and the crux put most sharply |\n\nOf the six premises in the hand run, the models reached one, and three of the four reached that\nsame one. The hand run had read both texts of the pair and the pass reads one, so the other five\nwere out of its reach by design.\n\n**What it suggests.** This is the first place a difference of tier showed, and it is the opposite\npattern from §§ 4a and 4b: the cheapest model found almost nothing, and the strong one found more\nand found the value. **Read by Claude, on two sources.** More premises are not better premises:\none the author would refuse is an invention in their name.\n\n---\n\n## 5. What this says about models\n\n### 5a. By kind of question\n\n| Kind of question | Examples | What has been measured | What follows |\n|---|---|---|---|\n| **Closed and sharp** | are these two unrelated; does this link bear at all, and which way; would the claim be weaker if this part were false; is this reported speech | four models gave the stored answer on 20 of 20 and on 11 or 12 of 12, at a thirtieth of a cent on the cheapest | the cheapest model that answers the same twice |\n| **Closed, with a loosely defined word** | *overlaps*; a part against evidence for it | one model repeats itself 16 times in 20, others agree 6 to 11 times | fix the word, then measure. No model helps |\n| **A degree** | how far a link bears | no two models agree; one gave a single answer to all | ask something else (two likelihoods, or a ladder of yes and no). Until then, write nothing |\n| **Copying** | the quoted passage | 98 in 100 word for word on the cheap model | the cheap model, with the check in code |\n| **Cutting and rewording** | the splitter, the rewording of a claim to stand alone | five models, four cuts; the missing rule fails on cheap and dear alike | one named model, the rule fixed, two checkers of other families |\n| **Open reading** | what arguments a text holds; what it leaves unsaid; kinship across debates | the strong model found more and found the value; the cheapest found nothing | the one place the strong model has shown its price is earned. Few calls per source |\n| **A verdict on contested ground** | is this value claim true | a decision model took a side on 4 of 8 where a reasoned model declined all 8 | no model. This project does not ask it |\n\n**The pattern.** A model's tier bought recall on open reading. It bought nothing on any question\nwe managed to close. Microtasks are the act of closing questions, so the design this project\nalready prefers is the one that makes the cheap models sufficient, and it leaves a short list of\nopen reads where the strong one is worth paying for.\n\n### 5b. By pass\n\n| Pass | What its answer does | Consequence (§ 2a) | Measured on | On the live site today |\n|---|---|---|---|---|\n| Reading a text into claims | writes claims in an author's name | speaks for a person | quoting only (§ 4a); never by a person | paused |\n| Unstated premises | writes a claim marked as unstated, in the author's name | speaks for a person; hides when it misses | one probe (§ 4d) | paused; runs on one of two paths |\n| Links inside a source, with their patterns | writes links that carry half weight | turns a relation around | never | paused |\n| Linker across sources | writes links that carry no weight until read | adds a marked item; turns a relation around | its failures (§ 3); gate verified on four models | paused with extraction |\n| Sameness keeper | marks a pair as the same claim; records a split proposal nobody reads | fuses | 60 pairs across four models; no person | paused with extraction; no mark stands in the graph |\n| Splitter and its check | writes parts that cap their whole | speaks for a person | eleven claims, on Claude Sonnet, by Claude | **writing, on whichever free-tier model has quota** |\n| Relevance judge | would set a link's weight | puts a number on a degree; turns a relation around | twelve links, five models; no person | writes nothing; waits on his word |\n| Kind of part | proposes only | none while it proposes | 197 links, twice, on Claude | not run there |\n| Short titles | the words a map shows for a claim | speaks for a person | never | every claim has one |\n| Reported speech | mints what someone was said to have said | speaks for a person | 153 read, 46 minted, unlabelled | on new extractions only |\n| Reading a person's evidence into a link | gives a person's link its pattern | adds a marked item | never | on, on the free tier |\n| Answer to a question of the graph | prose a reader meets | never trusted: shown with its citations and what it left out | n/a | on, on the free tier |\n\n### 5c. What the models cost\n\nList price per million tokens in and out, read at the router on 2026-10-05, and what one call of\nthe splitter's question cost that day:\n\n| Model | In / out, dollars | One splitter call, cents |\n|---|---|---|\n| GPT-6 Luna | 0.10 / 0.50 | 0.08 |\n| Gemini 3.1 Flash Lite | 0.25 / 1.50 | 0.13 |\n| Gemini 3 Flash | 0.50 / 3.00 | 0.28 |\n| Gemini 3.8 Flash | 0.75 / 3.75 | untried |\n| Claude Haiku 4.5 | 1 / 5 | untried |\n| Gemini 3.5 Flash | 1.50 / 9.00 | 1.45 |\n| Claude Sonnet 5 and 5.5, GPT-6.1 Sol | 2 / 10 | 1.45 (Sonnet 5) |\n| Gemini 3.1 Pro | 2 / 12 | untried |\n\n- **Per call the spread is about twenty times**, more than the list prices suggest, because the\n  dearer models think longer before answering.\n- **The free tier's second model is as dear as Claude Sonnet on a paid route.** A fallback that\n  changes the reader would at least triple the bill as well.\n- **A whole extraction** of a 717-word source cost 13.5 cents on Gemini 3 Flash\n  (`whose-tokens-pay-for-mapping.md` § 3g). Moving its two open reads (the first read for\n  arguments, and unstated premises) to Claude Sonnet would add 3 to 7 cents, estimated from the\n  call prices above. A whole extraction on Claude Sonnet would cost roughly 50 cents to 1.35\n  dollars, estimated the same way and untried on the router.\n  **⚠ Corrected 2026-10-10: this line said only *untried*, and a whole extraction on Claude Sonnet 5\n  has run twice**, through the Claude Code engine, at list prices with no money paid on the plan:\n  1,459 words in 78 calls for $7.99 on 2026-09-09\n  ([claude-code-as-extraction-engine.md](claude-code-as-extraction-engine.md) § 7.8c), and two texts of\n  3,466 words together on 2026-10-06, where the calls that were not linking came to about 300 and $26\n  ([mapping-the-machine-consciousness-debate.md](mapping-the-machine-consciousness-debate.md) § 5i).\n  That is about $5.50 and $7.50 per thousand words, against $0.70 to $1.90 per thousand in the\n  estimate: three to eleven times more. The two are not like for like, and nobody has measured why\n  they differ. The estimate is built from the price of one short splitter call; the first engine run\n  averaged about 10 cents a call, with the cost in a few long calls, and the engine runs at its\n  default thinking effort. So the estimate is unverified for the router, the route it was made for,\n  and a Claude run through the engine has cost several times what it says.\n- **Two cheap checkers of another family** on each of 49 claims add about 5 to 9 cents.\n- **Owed work can wait**, and the router lists a batch price at half for most of these models.\n  Untried.\n\n### 5d. The family before the tier\n\nMost of what the stored graph holds was read into claims by one family's model (Gemini, on the\npaid route until August). What has judged it since (the sameness keeper, the splitter, the census\nof parts) ran on another's (Claude, on the keeper's own plan). That was the engine at hand each\ntime, and nobody designed it so. The splitter's second version was measured with one model as\nwriter, checker and reader, which measures how well a model agrees with itself.\n\nOne router key reaches every family, so this is now a setting. Proposed: **the writer of a pass and\nits checkers come from different model families, and a write needs every checker's assent.** Two cheap\ncheckers cost a fifth of a cent a claim. What they dispute goes to a stronger model or a person.\n\nThe decision model keeps the place the measurements of 2026-10-04 gave it: a third kind of reader\nthat orders and flags, right at its confident ends, and never a verdict\n(`jev-and-the-judgement-layer.md` § 5d).\n\n### 5e. The answer key is the asset\n\nA ruling that names a model would be stale within weeks: three of the models in § 5c were released\nin the last fortnight, and the pipeline's default carries *preview* in its name. A bound on a pass\ndies with its model. What survives is the set of cases with known answers.\n\nProposed: **each pass keeps an answer key, and a model is admitted to the pass by clearing it.**\nFour steps, cheapest first:\n\n| Step | What it asks | Who does it | Cost |\n|---|---|---|---|\n| 1. The form | does every reply fit the answer form | code | cents |\n| 2. Steadiness | the same question twice, the same answer | code | cents |\n| 3. Known mistakes and laws | does it repeat a mistake already made here; does it break a rule code can check (the quoted passage, the numbers, the reply's language) | code | cents |\n| 4. The labelled sample | does it agree with what a person labelled blind | a person, **once per pass** | his hour, then cents for every later model |\n\nSteps 1 to 3 need nobody and can run on every new model the week it appears. Step 4 is the only\none that can prove anything, and its labels are used again for every later model and every later\nwording of the question.\n\n**What each pass's answer key holds today** (2026-10-05):\n\n| Pass | Known mistakes on file | A sample for a person | Labelled by a person |\n|---|---|---|---|\n| Sameness keeper | 47 pairs falsely marked; 14 false *supports*; 20 pairs that attack each other and so are not the same | 120 pairs, built, drawn by resemblance and not from the keeper's own verdicts (§ 6b) | **0** |\n| Splitter | 6 harmful parts; the *or* claim | 51 parts, live since 2026-10-05 (§ 6b; it served 37 before) | **0** |\n| Kind of part | 18 links where two readers disagree | 60 links, built | **0** |\n| Relevance judge | one link whose filing is disputed | about forty links, proposed, not built | **0** |\n| Linker | 84 false *part of* links; the 47 | none | 0 |\n| Reading a text into claims | one substituted name; one swallowed field; two hand runs as a reference | none | 0 |\n| Unstated premises | six premises from the hand runs | none | 0 |\n\n---\n\n## 6. What this says about switching the engine on\n\nHis word, 2026-10-05: *\"Hold off on deploying/switching on this feature for now, let's think it\nthrough\"*. That morning nothing was deployed and nothing switched on; production ran the commit of\nthe evening of 2026-10-04 (`0c4e974`).\n\n**Lifted at midday, as to deploying**, once the review page of § 6b was live: *\"Actually, I think\nwe can merge and deploy the held engine, the paid model route. Merge everything into master and\ndeploy.\"* Since 11:42 that day production runs master (`bede232`) with the engine's code in it,\nswitched off: in the running container its own check answers no, and none of its three settings\nis present. He did not say to switch it on. What follows was written as thinking about the hold\nand now reads as thinking about that next step.\n\n**What the analysis adds to the thinking.**\n\n- **Switched on as built, the engine would answer the live site's click routes with one named\n  model, and keep a record of every call.** Today those routes are answered by whichever of three\n  models has quota, and mostly unrecorded. On that one point it is an improvement.\n- **It would not make any pass measured.** The model it names, Gemini 3 Flash at low thinking\n  effort, has been measured here for whether its replies fit the forms and for nothing else.\n- **The live site's splitter is the case in hand.** It writes into the public graph today on an\n  unmeasured model. Holding the engine does not change that. Whether the splitter should keep\n  writing there, or hold its cuts as proposals until a model has cleared its answer key, is the\n  nearer question.\n\n**Three things proposed before the engine is switched on** (first written as what would lift the\nhold):\n\n1. Every pass that writes stores which model answered, on what it writes.\n2. The splitter writes only when that model is on its list, and holds its cut as a proposal\n   otherwise. No pass is served by another model without saying so.\n3. The splitter's list has one model on it that a person's labels put there.\n\n**What waited on that deploy** and had nothing to do with the engine: the linker reading its gate\nfirst, the two whole-graph batch jobs made the keeper's alone, the title and dating passes reading\neach claim whole, and the review of machine-cut parts (§ 6b). All are live since 2026-10-05. For\nfour minutes that day production ran a branch cut without the engine (`deploy/without-engine`,\nmaster with the engine's one commit taken out), which he deployed himself while the hold stood;\nmaster then replaced it.\n\n**Ruled 2026-10-05, on the splitter.** Asked whether the live site's split button should hold its\ncuts for approval until a model has passed a test he labelled, the founder: *\"Let's just have me go\nthrough and label/review quite soon, we don't have to hold\"*. So the splitter keeps writing on the\nlive site, and his labelling is the safeguard. The first sentence of item 2 above is thereby\ndeclined for the live site. Items 1 and 3 and the rest of item 2 stand as proposed. The engine's\nhold was a separate matter and was lifted later that day, as to deploying (above).\n\n## 6b. The sitting his ruling rests on, and four gaps in it\n\nHis ruling makes one page carry the weight: the review sitting for machine-cut parts\n(`/part-review/splits-1`). Four pieces of work meet in that page, done in four sessions, each\nwithout the results of the next. Read together on 2026-10-05, they left four gaps in it.\n\n| When | Session | What it did | What it could not know |\n|---|---|---|---|\n| 22 Sept | the parts review | a second machine, the parts judge, read every part-of link then in the graph, 197 of them | every part cut afterwards has no reading |\n| 28 Sept | the sitting | reused that review's page for the splitter's parts, taking the links the judge had read | the judge would never run again |\n| 29–30 Sept | the splitter's faults | found six parts that state as fact what their claim denies or grants; wrote version 2 | the sitting had no answer for that fault |\n| 3–5 Oct | this document | version 2 cut two claims on the live site, on a stand-in model | none of those parts were in the sitting |\n\n**The four gaps, each measured or read from the code:**\n\n1. **It left out 14 of the 51 machine-cut parts**: the eight parts of a split made again on\n   29 September, after a re-store had erased it, and all six cut by version 2. The sitting took only links the parts judge had read. Its own description\n   said *every machine-cut part*, and its test passed, because the test's stand-in graph answers\n   the query whatever the query filters on.\n2. **It had no answer for the fault it exists to count.** A part its claim denies is closely\n   related to the claim, is not evidence about it and is not a piece of it. The six keys offered\n   *barely relate* or *cannot tell*.\n3. **It read all the splitter's parts as one population.** Forty-five were cut by version 1 on\n   Claude Sonnet and six by version 2 on `gemini-3.5-flash`. A measured rate holds for one\n   version on one model (§ 1), so a pooled share describes neither. The rule he accepted on\n   28 September, that the correction panel goes if nine parts in ten hold up, was set when every\n   part was version 1's.\n4. **It was built once and held for the life of the server process**, so a part cut after the\n   first visit would not appear until a restart.\n\n**Fixed and deployed the same day** (read over the wire afterwards: 51 parts, seven answers, 14\nwith no machine reading):\n\n- The sitting holds every machine-cut part. One no machine has read is served with no reading\n  and is marked so.\n- A seventh answer: *not a piece — the claim denies it, only grants it, or never says it*. It is\n  key 5; *barely relate* moved to key 6. Nobody had answered yet, so nothing was renumbered under\n  anyone.\n- The results are read per splitter version and per model, and an answer of *one of several\n  routes* is counted apart from a required piece.\n- A sitting that takes every link of one writer is built fresh on each request.\n- The splitter now writes the model that cut a part, and the model that checked the cut, on the\n  link itself. A claim cut a second time replaces the record on the claim, and the link is where\n  both names outlast that.\n- The same afternoon, because he was about to sit it on a phone: the next item starts at the top\n  of the page, and an answer that does not reach the server is sent again and, failing that, comes\n  up again before the page can say the review is done. Before, such an answer was never asked\n  again, and a refusal by the server was not noticed at all. The answers are the answer key, so a\n  hole in them matters more than a repeat (`the-parts-review.md` § 4c).\n\n**What his sitting will and will not tell, as the parts stand:**\n\n- **It gives the splitter its first answer key.** Fifty-one labelled parts, with real faults\n  among them: the six known ones are the cases any checker, on any model, must catch. That is the\n  asset of § 5e, and no later change of model or wording spends it.\n- **It says little yet about the splitter that runs.** Six parts from two claims. Read alone,\n  six clean parts would bound nothing. Proving version 2 needs about sixty (§ 1), which means\n  cutting more claims with it first.\n- **It answers two open questions by cases.** Whether something a claim says in passing is a part\n  (ledger T6) is answered each time he presses *survives without it* or *never says it* on such a\n  part. Whether alternatives are parts is answered by key 3.\n\n**Order, proposed.** Sit the parts first, and decide on replacing the seven first-version splits\n(ledger T7) with his answers in hand, since they say which of those cuts are worth replacing. The\norder costs the answer key nothing either way: as the sitting is built, a replaced part stays in\nthe graph, marked, and so stays in the sitting, and fresh cuts are added to it.\n\n**He took the other order, 2026-10-06** (*\"Fresh cuts blessed\"*, then *\"We can always review the\ncuts later after the reviews that's fine\"*). The seven were cut again that evening on\n`claude-sonnet-5`, with the four claims whose first-version proposals had been held, and the sitting\nchanged as the paragraph above said it would: the 45 replaced parts are still in it, the six cut on\nthe live site are still in it, and the fresh parts joined them. Two of the statements above are\ntherefore dated. *Fifty-one labelled parts* is now more than a hundred, and *six parts from two\nclaims* is no longer all there is of the splitter that runs: version 2 on the model it was measured\non has about the sixty parts a proof needs, cut from eleven claims. The count, the reading of the\nfresh parts and what the run showed about the engine are in\n[microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 3g.\n\n**The sameness sitting needs no change.** Its 120 pairs were drawn by resemblance and by known\nconflicts, not from the keeper's verdicts, so his labels there do not depend on any keeper\nversion or model. A reshaped draw from the keeper's own verdicts was designed on 22 September and\nnever prepared; as an answer key the first draw is the better one, for that reason. Its page words\n*overlaps* as *they share part of a claim*, which is the reading his ruling of 13 September meant.\nThe keeper's instructions say *partly advance*, which models read as *is evidence for*\n(`microtasks-in-broad-daylight.md` § 3f, ledger T5). So he can sit it as it is, and what waits is\nbringing the keeper's wording to the page's before anyone scores the keeper against his answers.\n\n**The same gap, one level up.** Of the 206 part-of links standing in the graph, the parts judge\nhas read 161. The other 45 were written after it ran and carry no reading: these 14, and 31 the\nlinker wrote. (Twenty-nine more were read and then retired on 27 September; they stay in the graph,\nmarked.) The general parts review cannot see those 31. It is the case\n`machine-work-as-a-standing-property.md` describes: a pass that runs on one day, over what exists\nthat day, and is never asked again. Nothing is owed before he sits: a part with no machine reading\nis put to him without one, which is the cleaner label.\n\n---\n\n## 7. Proposals (ledger Arc Y; unruled except the first clause of Y5)\n\n1. **Y1. Three levels of quality, by consequence** (§ 2a): 2 in 100 for an answer that turns a\n   meaning around, fuses, or adds to what a named person said; 5 to 10 in 100 for a marked,\n   undoable item that moves no strength and that nothing is built on; no number where careful\n   people disagree, which stays a proposal. A bound on the acted-on answer, each lean counted\n   separately. The numbers are his.\n2. **Y2. The answer key is the asset.** Each pass keeps one; a model is admitted to a pass by\n   clearing it; it is run again on every new model and every new wording. Sharpens ledger X10: a\n   pass names the test a model must pass, and the list of models follows from it.\n3. **Y3. Four steps** to admit a model to a pass (§ 5e), of which only the last needs a person.\n4. **Y4. Writer and checkers from different model families**, and a write needs every checker's\n   assent.\n5. **Y5. The splitter writes only when the model that answered is on its list; every pass stores\n   which model answered; none is served by another model silently.** The first clause extends to\n   another pass when that pass has an answer key. It is not a rule for extraction today, which has\n   none. **First clause declined 2026-10-05**: *\"Let's just have me go through and label/review\n   quite soon, we don't have to hold\"* (§ 6). The other two clauses stay open; the splitter's half\n   of the second is built (§ 6b).\n6. **Y6. The quoted-passage check and the number check run on every extraction**, on every model:\n   a claim whose passage is not in its source, or that carries a number the source lacks, is asked\n   again once and then marked. It is never stored silently.\n7. **Y7. Alternatives a claim leaves open are never written as required parts.** Narrowed\n   2026-10-05 from *are not parts* (§ 4b): the strength code already holds alternatives, as parts\n   of which the strongest decides. The splitter gets a rule for them and its check a label:\n   *positive or necessary* asserts neither alone. (*It fails whether demand rises or falls* is\n   different: that claim asserts both.) It then either keeps the alternatives in one part, or cuts\n   them and tags them as alternatives; whether a machine may set that tag by itself is part of the\n   ruling. Also a rule for a describing phrase: *low-wage jobs* does not assert that the jobs are\n   low-wage.\n8. **Y8. Open reading is where the strong model answers first**: the first read for arguments,\n   unstated premises, premises across sources. Everywhere else the cheap models answer first and\n   the strong one takes what they dispute (§ 5d). Pending a comparison against the hand runs of\n   July, which the router makes cheap.\n\n---\n\n## 8. What would show this wrong\n\n- **Y1** is too strict if a person's sample shows that readers are not misled by errors of the\n  first kind at 5 in 100, and too loose if one harmful part in fifty is enough to make an author\n  refuse the map. Only authors reading their own maps can say.\n- **Y2 and Y3** fail if answer keys go stale as fast as models do, which would happen if each\n  new register of text broke the labelled cases. The three-runs law predicts some of that.\n- **Y4** is wasted if checkers of different families agree on the same wrong answers. The\n  literature says they do about 60 times in 100 on hard items. § 4b is one case where they did\n  not.\n- **Y7** is unnecessary if the *or* case is rare. Counting claims with *or* between two predicates\n  would say. Not counted.\n- **Y8** is wrong if a person reading § 4d's premises blind prefers the cheap model's, or if the\n  strong model's extra premises are ones their authors refuse.\n- **The sorting in § 3** is the weakest part: it is one reader's classification of nineteen cases,\n  and it covers only failures somebody noticed.\n\n## 9. Considered and rejected\n\n- **Naming one model for the whole pipeline.** § 5a: the kinds of question differ, and the price\n  spread is twentyfold.\n- **The strongest model everywhere, for safety.** It made the same cut on the *or*, followed a\n  link's filing against the text, and flattened an author's labels. Its price would also end the\n  standing passes: reading every live link once costs about 27 dollars on it and about one on the\n  cheap ones.\n- **The cheapest model everywhere.** It found nothing on the open read.\n- **Trusting agreement between models as proof.** It orders a person's attention and proves\n  nothing.\n- **Waiting for labels before doing anything.** Steps 1 to 3 need none.\n- **A guard on extraction built now.** Y6 changes what extraction stores, so it is his to rule.\n  The measuring script changes nothing.\n\n---\n\n## Cross-references\n\nCode and data: `scripts/measure_quote_fidelity.py` (tests in `tests/test_quote_fidelity.py`) ·\n`scripts/probe_split_one_sentence.py` · `scripts/probe_unstated_premises.py` ·\n[`pass-quality-and-the-choice-of-model/`](pass-quality-and-the-choice-of-model/) (the three result\nfiles) · `deliberus/extraction/client.py` (the fallback chain) · `deliberus/proposed_split.py` (the\nwrite gate, and the one record of who answered).\n\n[microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) ·\n[proving-a-pass-reliable.md](proving-a-pass-reliable.md) ·\n[earning-the-right-to-write.md](earning-the-right-to-write.md) ·\n[machine-work-as-a-standing-property.md](machine-work-as-a-standing-property.md) ·\n[whose-tokens-pay-for-mapping.md](whose-tokens-pay-for-mapping.md) (§ 3e, the first measurement\nacross models; § 3g, the engine) ·\n[jev-and-the-judgement-layer.md](jev-and-the-judgement-layer.md) ·\n[split-review-and-the-founders-intentions.md](split-review-and-the-founders-intentions.md) ·\n[relevance-and-the-two-factors.md](relevance-and-the-two-factors.md) ·\n[the-parts-review.md](the-parts-review.md) ·\n[session26-open-rulings.md](session26-open-rulings.md) (Arc Y; T1, T3, T5, X10, R8, R12) ·\n[listening-for-the-calculus-and-sturdy-microtasks.md](listening-for-the-calculus-and-sturdy-microtasks.md)\n(episodes found 2026-10-05 on judging by machine: agreement corrected for chance, yes-or-no over scales,\nand what a small labelled set can show about two models) ·\n[the-evals-loop-and-our-machine-passes.md](the-evals-loop-and-our-machine-passes.md) (2026-10-05: bears on § 5e. A key used to improve a wording proves less when it is\nthen used to admit a model, so a share of it is sealed; and a check is reported by two rates, never\none agreement figure. Ledger Y9 to Y14)\n"}