{"path":"research/proving-a-pass-reliable.md","content":"# Proving a machine pass reliable enough to stop checking every item\n\n**Date**: 2026-09-29 · **Type**: research and proposal · **Status**: nothing here is ruled.\nSupersedes ledger rows S2 and S5, written the same morning (§ 1).\n\n> ⚠ **Correction, later on 2026-09-29.** Two figures below were read too broadly. The 14–28% saving (§§ 5, 9f) was measured on zero-shot labelling of contested social-science categories, not on specified tasks like ours, so it is a poor prior on pace. And 62–68% (§§ 0, 7) is roughly the share of parts with no problem at all; the share that does harm is 6 of 37, two of them clearly, all six caught by the parts judge and none by the coverage check. Both are corrected in [microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) §§ 1, 3 and 3c.\n\n**Occasion.** After listening to the major-transitions walkthrough, on a walk, the founder wrote,\nVERBATIM: *\"The devil's in the details. For example, machine splitting and sameness detection and\nmerging (read up on both of these) can probably be trusted since (yet if and only if) they are\nreliable enough on the microtasks in question. Review doesn't need to keep up if it never has to\ncorrect anything since the accuracy and reliability of LLM passes are high enough for the tasks at\nhand. I think this can be proven? Read up deeply.\"*\n\n**What this builds on, rather than re-deriving.** The trust ladder (proposing → writing-marked →\ntrusted, [earning-the-right-to-write.md](earning-the-right-to-write.md) § 4) and the calibration\nprinciple (*the human calibrates the pass; the human never performs it*,\n[machine-work-as-a-standing-property.md](machine-work-as-a-standing-property.md) § 3d) already hold\nthe shape. The ladder says of its numbers: *\"The numbers are not mine to set and should not be\nguessed. They come out of the first review sitting.\"* This document supplies the arithmetic those\nnumbers come out of, the conditions under which they mean anything, and where each pass stands.\nLedger row R12 (*\"high confidence\" means measured calibration*) is the ruling candidate it\noperationalises. The literature pass behind §§ 5–6 is kept verbatim in\n[proving-pass-reliability/](proving-pass-reliability/).\n\n---\n\n## 0. Answer first\n\n**Yes, it can be proven, in a precise and bounded sense, and the founder's corollary is right.**\n\n1. **What can be proven is a bound, never a zero.** *\"Never has to correct anything\"* cannot be\n   shown. *\"Wrong at most 2% of the time, with 95% confidence\"* can. It takes about 150 randomly\n   drawn items that a person labels without seeing the machine's answer, with none of them wrong.\n   A 5% bound takes about 60 (§ 2).\n2. **Review does not have to keep pace with volume.** If a fraction ε of a pass's writes are wrong\n   and nothing builds on them, the share of wrong machine content stays at ε however much is\n   written. Sampling keeps ε measured; touching every item adds nothing but cost. What grows is\n   **depth**: a chain of machine steps on machine output is wrong about ε per step. This retires\n   the two rules proposed this morning (§ 1).\n3. **A bound holds for one microtask, one kind of claim and one version of the pass.** A bound\n   earned on long implicit premises says nothing about reported speech, and a model or prompt change\n   resets it (§ 4, devils 8–9).\n4. **Some microtasks have no answer key.** There the most that can be shown is that the machine\n   agrees with a careful person as often as two careful people agree with each other (§ 3).\n5. **Expect a guarantee to cover part of the stream, not all of it.** The strongest published\n   method lets the machine decline its uncertain items and proves the bound on the rest; in its\n   experiments it answered 56–68% of items at 80–90% agreement with people, and the items it declined\n   were the ones people disagreed on most (§ 5). So the likely outcome is a splitter trusted on\n   plainly bundled claims and handing the hard cuts to people, not a splitter trusted everywhere.\n\n**His position after this answer** (§ 9, verbatim): error rates are costs that design can keep\nlowering, *\"just like Deliberus aims to lower the cost of bridging minds and perspectives\"*, balanced\nagainst an edge beyond which people review and contribute without being overwhelmed. The canon line\nof § 1 was admitted the same turn with *\"Yes.\"*\n\n**Where things stand.** No pass is proven. Every accuracy figure in the corpus is a model reading a\nmodel's output: no split, sameness verdict or merge has a human label. The only readings on the\nsplitter are not reassuring on the material they cover (a second model called 62–68% of written\nparts genuine; the pass's own coverage check held 4 of 11 proposals for invented content), and that\nmaterial is narrow. The instruments that would prove or demote it are built: three review sittings,\nall unsat (§ 7).\n\n---\n\n## 1. The correction: S2 and S5 rationed the wrong quantity\n\nThis morning's reading of the major transitions ([major-transitions-heredity-and-evolvability.md](major-transitions-heredity-and-evolvability.md)\n§§ 7–8) proposed two rules:\n\n- **S2**: *\"A machine may check what a machine wrote, but may not build on it until a person has\n  engaged with it.\"*\n- **S5**: *\"A daemon's proposal rate must stay below what people can review.\"*\n\nBoth ration machine work by human review capacity. Eigen's error threshold, which the same document\nuses in § 3, limits something else: **the error per copy multiplied by the length of the copying\nchain.** In Eigen's model the population's size does not appear. Applied here:\n\n- **Writes that nothing builds on form no chain.** If a fraction ε of splits are wrong, then after a\n  hundred splits or a hundred thousand, the wrong ones are a fraction ε. Undetected errors grow in\n  number and not in share. A graph of machine content at ε = 1% is 1% wrong at every size.\n- **What compounds is depth.** A part cut from a machine-cut part, or a link drawn to one, inherits\n  the upstream error. At 2% per step, two steps are wrong about 4% of the time, five steps about\n  10%, ten steps about 18% (computed: 1 − (1 − ε)^k).\n- **The chain starts earlier than S2 assumed.** Extraction is itself a machine step, and no\n  human-labelled error rate for it appears anywhere in the corpus, so a machine-cut part of an\n  extracted claim is already two machine steps from the source text.\n\nSo review needs to keep up with nothing; it needs to keep ε **measured**, and the depth of\nunexamined machine chains **capped by that measurement**. Both rules were also written without\nreading the trust ladder, whose third rung is sampled review. And S5's reading of the\nminted-to-examined ratio, that above some ratio the graph's information degrades, conflicts with the\nend-state blessed on 2026-08-31, under which untouched machine structure is the permanent honest\ndefault under provenance display. The revised rules (ledger T1 and T2):\n\n> **T1, replacing S2.** A machine pass may build on machine output only as deep as its measured\n> error rate allows. With no measured rate, it may not build on machine output at all, and a person\n> engages first.\n\nThe old S2 survives as the special case of an unmeasured pass. Its countable dial survives too:\n**unexamined machine-generation depth** per claim, counted from the source text or the last human\ntouch, now capped at the largest depth k where k·ε stays under a tolerance the founder sets.\n\n⚠ **2026-10-10: the second sentence of T1 was never ruled, and for one pass he has now ruled the\nother way.** His *\"Yes.\"* of 29 September was to the line that measurement caps how deep machine\nwork builds on machine work. *With no measured rate it may not build at all, and a person engages\nfirst* stayed a proposal. Asked on 10 October whether the pass that writes out what a link rests on\nmay write deeper steps itself or should only find and ask until its error is measured, he answered:\n*\"It may and should write deeper steps. But the microtasks in charge of doing so need to be\nquality-controlled, just like the other microtasks that are in play in Deliberus.\"* (ledger AG6).\nThe dial above is what that pass is then owed on every sentence it makes, and the measuring is\nwhat its quality control is for:\n[quality-control-for-deeper-steps.md](quality-control-for-deeper-steps.md).\n\n> **T2, replacing S5.** What must stay under a tolerance is a daemon's unexamined error, not its\n> proposal count. Review capacity caps the rate only while the daemon's error rate is unmeasured.\n\nFor the analogy daemon that proviso is likely to bind for a long time, since kinship between\ndomains has no answer key (§ 3), so its outputs stay proposals that people select. The rest of S5\nstands: the score-free nursery is a germinal centre and needs a stricter export gate than the\ncommons. An export gate is a checker, and a checker's miss rate is exactly the kind of number § 2\nproves. The *minted-to-examined* dial from [drowning-in-claims.md](drowning-in-claims.md) gets its\nreading back: multiplied by a pass's proven error bound, it is the expected number of unexamined\nerrors. Alone it says only how much stands on the bound.\n\n---\n\n## 2. What \"proven\" means in numbers\n\n**The procedure.** Draw items at random from what the pass actually wrote, not the longest first\nand not the ones someone complained about (*\"An audit needs a denominator independent of who\nhappened to complain\"*, `machine-work-as-a-standing-property.md` § 3d). A person labels each one\nwithout seeing the machine's answer. Count the wrong ones and take the one-sided 95% upper bound of\nthe binomial (Clopper–Pearson). Computed 2026-09-29:\n\n| Bound wanted (95% confidence) | 0 wrong | 1 wrong | 2 wrong |\n|---|---|---|---|\n| under 10% | 29 items | 46 | 61 |\n| under 5% | 59 | 93 | 124 |\n| under 2% | 149 | 236 | 313 |\n| under 1% | 299 | 473 | 628 |\n\nWith zero wrong in n items the bound is about 3/n (the *rule of three*, Hanley & Lippman-Hand\n1983, whose title asks *\"If nothing goes wrong, is everything all right?\"* and whose answer is: only\nup to that bound, and only for the population sampled). Eleven items all found correct would bound\nthe rate at 24%; the 37 written parts, if all were found correct, would give 7.8%. The binomial\narithmetic assumes the reviewer makes no mistakes, which is what the seeded wrongs of § 4 check.\n\n**Keeping it proven: a trickle and a sequential test.** A bound is a snapshot. The standing form is\na small random trickle of new writes, labelled blind, read with Wald's sequential probability ratio\ntest, which stops as soon as the evidence is decisive. With both error probabilities at 5%:\n\n| Distinguish | Clean items to confirm | Each wrong item adds | Wrong items in a row that fail it |\n|---|---|---|---|\n| ≤ 1% from ≥ 3% | 144 | 54 | 3 |\n| ≤ 2% from ≥ 5% | 95 | 30 | 4 |\n| ≤ 5% from ≥ 10% | 55 | 13 | 5 |\n\nThose are the fastest cases. On average a decision takes about twice as long: for 1% against 3%,\nabout 287 items when the pass is good (computed, Wald's approximation). That table is the ladder's\nautomatic demotion in numbers: a pass on the *trusted* rung stays there\nwhile its trickle keeps confirming, and drops a rung when the test fails.\n\n**What it costs, estimated rather than measured.** If a split takes about 20 seconds to judge and a\nsameness pair about 10, a 150-item proof sitting is under an hour for splits and under half an hour\nfor pairs, and a trickle of one write in fifty costs minutes a month. The first real sitting will\nreplace these guesses with timings.\n\n---\n\n## 3. The microtasks do not share one kind of truth\n\n*\"Is the splitter reliable?\"* is a bundled claim, and it decomposes the way the corpus decomposes any\nbundle: into microtasks that can be judged separately. They come in three kinds, and each kind can\nbe proven to a different standard.\n\n- **Checkable against the text.** An answer key exists in the material itself. Did a part say\n  anything the parent did not (invention)? Did the cut drop something (loss)? Is a pair joined by an\n  attack edge *the same claim*? (It is not, by construction.) These can be proven to tight bounds,\n  and a second machine can do the checking, provided the checker's own miss rate is measured. Part\n  of the invention check can even be a law in code rather than a judgment: a part may not contain a\n  number, date or proper name absent from its parent ([bend-and-deliberus.md](bend-and-deliberus.md)\n  § 5, where laws for code and samples for judgments are set side by side).\n- **Consensus.** No key, but careful people agree. Does each part read on its own to a stranger?\n  Are two claims the same claim in different words, at a coarse grain? Proven against a panel, with\n  a ceiling set by how often two people agree.\n- **Contested.** People disagree. Where exactly to cut; whether a part is necessary or corroborative\n  (*\"There is no key\"*, [the-parts-review.md](the-parts-review.md) § 1); fine sameness against\n  overlap; whether two domains are kin. Here a machine can only be shown to agree with a person as\n  often as two people agree with each other. Past that ceiling, *accuracy* is undefined. This is\n  also where the founder's worry of 2026-09-19 about *\"saboteurs/spammers adding bogus parts\"* bites\n  hardest, since no key exists to catch a bogus part, and why relevance is argued there rather than\n  voted (ledger Q33).\n\n| Pass | Microtask | Kind | What measures it today |\n|---|---|---|---|\n| Splitter | invention | checkable | the coverage check holds it (4 of 11 held); the check's own miss rate is unmeasured |\n| Splitter | loss | checkable | the coverage check's *dropped* verdict (6 of 37 written edges, all from one parent) |\n| Splitter | each part stands alone | consensus | splits sitting (37 links, 8 served blind) |\n| Splitter | the right cut | contested, or a key if his prior holds | splits sitting; a second person sitting the same items |\n| Splitter | part or evidence | contested | parts review (60 edges, including the 18 where two readers disagree) |\n| Keeper | same or opposes | checkable | sameness sitting: 20 seeded wrongs from attack-joined pairs |\n| Keeper | same or unrelated | consensus | sameness sitting |\n| Keeper | same, contains or overlaps | contested at fine grain | sameness sitting, then a friend sitting it too |\n| Merge | acting on *same* | inherits the keeper's precision on *same*, then transitivity (§ 4, devil 4) | nothing yet |\n\n**His prior is testable here.** On 2026-09-21 he said of splitting: *\"I think logic would reign here.\nI don't expect different cuts to be a matter of taste, this is probably more a science than an\nart?\"* If two careful people cut the same claims the same way most of the time, the cut has an\neffective key and moves from the contested row to the consensus one, and the splitter can then be\nproven against it. The friends' round he asked for on 2026-08-31 is that measurement.\n\n---\n\n## 4. The devils\n\nEach of these can make a proof look sound when it is not.\n\n1. **\"Correct\" needs a reference, and the reference is people.** The ceiling is how often two\n   careful people agree. Measure it before reading any machine score: 90% agreement with one person\n   means one thing if two people agree 95% of the time and another if they agree 80%. The ceilings\n   the literature reports for nearby tasks are lower than intuition suggests: the majority vote of\n   five crowdworkers reached 81% accuracy on a standard paraphrase benchmark, and annotators matching fact-checked claims\n   across languages agreed only moderately (Fleiss' κ 0.60–0.62). For decomposition, the only human\n   figure found is that experts revising a 2022 model's atomic facts split 18% of them and merged\n   34%; no study measures how often two people cut a claim the same way.\n2. **Precision is not recall.** Sampling what the pass wrote measures how often its writes are\n   wrong. It says nothing about what it failed to do: the bundle left unsplit, the duplicate left\n   unmerged. A miss leaves the graph as it was, while a wrong write adds damage, so precision is the\n   right first target. Recall still needs its own sample, drawn from what the pass left alone:\n   labels taken only from inside the candidate set cannot bound what the candidates missed.\n3. **The base rate decides what an accuracy figure means.** Suppose 1,000 candidate pairs, of which\n   5% are truly the same claim, and a detector that finds 90% of those and wrongly calls 2% of the\n   rest *same*. Its accuracy is 98%, and 30% of the merges it proposes are wrong. What must be\n   proven is **precision on the answer that triggers action**, on the real stream of candidates. The\n   sameness sitting samples three similarity bands to straddle the decision boundary, which is right\n   for calibration; a proof reweights its answers to the bands' shares in production. The direction\n   of the error matters too: in one small 2023 test a chat model recognised 88% of true paraphrases\n   and only 44% of non-paraphrases, so its mistakes ran toward false *same*, which is the costly\n   direction for merging. On well-defined product records current models do far better (in one 2026 preprint, wrong on\n   0.7–4.5% of an audited set, close to the benchmark labels' own 0.5–2.5% error), but a product\n   record is not a contested claim.\n4. **Pairwise accuracy does not compose into cluster accuracy.** Merging is transitive: A = B and\n   B = C put A and C together. At a 1% false-*same* rate, a cluster chained through ten links holds a\n   false link about 10% of the time. The corpus's cautionary case is a closure over `owl:sameAs` that\n   falsely unified 177,000 names. So no silent transitive chaining (already a corpus rule), and a\n   merge proof bounds the cluster, either by judging every pair inside it or by proving a much\n   stricter pairwise bound. The same lesson appears in the conformal literature for reasoning:\n   filtering claims one at a time fails when claims depend on each other, and the guarantee returns\n   only when it is calibrated over connected sets of claims (*coherent factuality*, ICLR 2025).\n5. **Errors compound with depth** (§ 1), and the first link is extraction, which is unmeasured too.\n   Every pass downstream inherits its error, so the proof programme has a natural starting point.\n6. **Only blind labels prove anything about the machine.** A person shown the machine's answer\n   agrees with it far more often: in the detect-but-defer study, 80.7% detected an AI error and\n   followed the AI anyway. The sittings serve one item in five blind, which measures that anchoring.\n   The same design cannot prove the machine's error rate from its non-blind part, so a proof sitting\n   should be blind throughout, and the mixed sittings kept for what they measure: the\n   person-plus-machine system that runs in practice.\n7. **Without seeded wrongs, \"none found\" means nothing.** A reviewer who misses planted errors\n   misses real ones. The sameness sitting gets them free from attack-joined pairs; the parts review\n   has none by design; for splits they are cheap to plant (drop a qualifier from a correct part, add\n   a clause the parent never said). Seeds stay inside sittings: live seeded probes were dropped on\n   2026-08-31. And as § 3d of the calibration doc says, a seed tests attention to known mistakes, not\n   whether reviewer and model share a blind spot.\n8. **A proof is a snapshot of one version.** A new model, a new prompt or a new backend resets it:\n   the Claude and Gemini backends are different passes. (First seen with numbers on 2026-10-04: on\n   one prompt, Claude repeats its own step for a link 10 times in 12 and three other models match\n   it 4 to 6 times, [whose-tokens-pay-for-mapping.md](whose-tokens-pay-for-mapping.md) § 3e.) So a bound is stored with the pass's version\n   (the watermark of ledger R8) and re-checked with a small sample on any change. A concrete\n   instance arrived the same evening: every stored split proposal named the model asked for rather\n   than the one that answered, so a proof would have been filed against the wrong model (fixed,\n   [microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 3d).\n9. **The sample must match the stream.** The splitter's first eleven were taken longest first, seven\n   of them implicit premises; no ordinary claim extracted from a source has ever been split. A bound\n   covers the population its sample came from. Either stratify by kind of claim (extracted fact,\n   extracted value judgement, implicit premise, reported speech, written by a person), at about sixty\n   items per stratum for a 5% bound on each, or prove the mixture and accept that a rare kind could\n   hide a high rate. *\"A new register begins outside the existing grant\"* (§ 3d) is this devil stated\n   as policy.\n10. **A machine checking a machine is independent only if it fails differently.** The splitter\n    writes only what its coverage check passes, so the check's miss rate is part of the written\n    error rate. When two language models both err on a multiple-choice question, they choose the\n    same wrong answer about 60% of the time against a chance rate of one in three, more so the more\n    accurate they are, and a model judge inflates the accuracy of models from its own provider. So\n    the 62–68% reading of the splitter's parts may be off in either direction, and a second model\n    family helps without guaranteeing independence: give the checker its own seeded test, and\n    measure how often the two families fail together. *(Seen on one real split, 2026-10-05: of five\n    models asked to check it, the three of the writer's own family all passed the fault and one model\n    of another family caught it, while each of the five had a miss or a false alarm of its own:\n    [pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) § 4b.)*\n11. **A model's confidence is not a calibrated probability.** The fourteen false support links\n    retired on 2026-09-13 were written at strengths of 0.6–0.8 (ledger R12). A confidence score\n    becomes usable only when a threshold on it is set against human labels (§ 5).\n12. **Per-item accuracy cannot see drift in the whole.** Every split can be right while the graph\n    still moves: reuse-driven linking can favour vaguer claims (ledger S1, blind selection), or the\n    cuts can steadily preserve one framing over another. That needs measures of the population, not\n    audits of items: the share of vague claims over time, the head-and-tail balance\n    ([saturation-and-the-long-tail.md](saturation-and-the-long-tail.md)), disagreement preservation.\n\n---\n\n## 5. Abstention buys the guarantee\n\n**The strongest tool in the literature is to let the machine decline, and to prove the bound only on\nwhat it does not decline.** *Trust or Escalate* (Jung, Brahman and Choi, ICLR 2025) is the worked\ncase for exactly this kind of task, a model judging items where people are the reference:\n\n- A model judge gives an answer and a confidence, and declines below a threshold. On a\n  human-labelled calibration set of about 500 items, the threshold is lowered step by step, each step\n  tested with an exact binomial bound, and the lowest threshold that still meets the target is\n  kept. The result holds with a stated probability: among the items the judge answers, it agrees\n  with the human majority at least as often as the target.\n- **Coverage is the price.** At a 90% agreement target on one dataset the judge answered 56% of\n  items; at 85% on another, 63%; a cheaper cascade reached 68% at 80%.\n- **The items it declined were the ones people disagreed on** (human agreement 0.815 on them against\n  0.902 on the rest). The machine's uncertainty tracks the task's own contestedness, which is the\n  division of labour Deliberus wants: the plainly bundled cut goes through, the contested one goes\n  to a person.\n- **A threshold set from a point estimate is not a guarantee.** Thresholds chosen from the raw\n  agreement rate held the target in only 47.5–79% of repeated trials; the bound-based procedure held\n  it in over 90%.\n\n**The general recipe** is *Learn Then Test* (Angelopoulos, Bates, Candès, Jordan and Lei, 2025):\nchoose candidate thresholds, test each against the target with a finite-sample p-value, and keep\nonly thresholds that survive a multiple-testing correction. It covers several risks at once, needs\nno assumption about the model, and **may honestly return no threshold at all**, which is the\nproof saying that no automation level meets the target.\n\n**A whole-dataset version exists.** *Probably approximately correct labels* (Candès, Ilyas and\nZrnic, 2026) has experts label every item above an uncertainty cutoff, set so that the error over the\nentire dataset stays below a target. For GPT-4o labelling text at a 5% target, it saved 14–28% of\nthe expert labels, while the model's labels alone were 19–38% wrong. That is the sober\nprior: on hard text tasks a guarantee buys partial relief, not freedom from review.\n\n**Deliberus already has the shape in two places and lacks the calibration in both.** The keeper has\nits confession key (*can't tell*), and the splitter's coverage check holds proposals with invented\ncontent. Neither threshold has been set against a person's labels. Setting them is the sitting's\njob, and it turns *\"the human calibrates the pass\"* from a principle into a procedure: the person\nlabels a calibration sample and a trickle; the pass acts where it has earned it; the rest routes to\nthe triage feed. One refinement from the literature applies directly: the confidence measure in\n*Trust or Escalate* was agreement across several differently-prompted runs, which roughly halved the\ncalibration error of the model's own stated probability. Deliberus could ask a split or a sameness\nverdict three ways and read their agreement as the confidence, at three times the model calls.\n\n---\n\n## 6. The model-read numbers are not proof, and not wasted\n\nThe census of all 197 decomposition edges (run twice, agreeing at 45/78/74 and 46/70/81 parts,\nmiscast and no-fit) and the second model's reading of the 37 written parts are a model judging a\nmodel. Alone they prove nothing. Combined with a small human-labelled sample they can: the human\nsample measures the model reader's bias, and correcting for it gives a valid interval narrower than\nthe human sample alone would give (*prediction-powered inference*, Angelopoulos and colleagues,\n*Science* 2023). In a later evaluation study the gain was worth about 50% more labelled data.\n\n**The limit is known and sets expectations**: when the judging model is no more accurate than the\nmodel it judges, no such correction can more than halve the human labels needed (Dorner, Nastl and\nHardt, ICLR 2025). So the census can at best halve the size of the sittings. It cannot replace them.\n\n---\n\n## 7. Where each pass stands, and the instruments already built\n\n| Pass | Rung | Evidence today | What proves or demotes it |\n|---|---|---|---|\n| Extraction | writes (the base of every chain) | no human-labelled error rate | a blind sample of extracted claims against their source spans |\n| Splitter | writing-marked, by ruling on 2026-09-21 | model readings only: 62–68% genuine parts, 4 of 11 proposals held for invention, 0 human touches | splits sitting (37 links, 8 blind) waits on a deploy from branch `deploy/splits-sitting`; then a blind, stratified proof sitting |\n| Keeper (sameness) | runs in auto-connect; *same* marks a merge candidate and writes no edge | model runs only | sameness sitting 1: 120 pairs, 20 seeded wrongs, 24 blind |\n| Merge | switched off | none | the keeper's precision on *same*, then a cluster-level check |\n| Parts judgement | proposing | the census, twice | parts review 1: 60 edges |\n\n**The one decision that unblocks the rest is his time**: the sittings are built and nothing is\nproven until someone sits them. A proof also needs one thing the built sittings do not yet do,\nsince each serves most items with the machine's answer visible: a fully blind sitting sized to the\nbound he wants.\n\n---\n\n## 8. Literature\n\nRead depth is marked in the verbatim report,\n[proving-pass-reliability/literature-report-2026-09-29.md](proving-pass-reliability/literature-report-2026-09-29.md);\nfigures quoted above come from sources read in full or in their main sections, except where the text\nsays *one preprint* or *one small test*.\n\n- **Bounds from zero failures**: Hanley & Lippman-Hand, *JAMA* 1983 (the rule of three); Clopper &\n  Pearson, *Biometrika* 1934 (exact limits); Wald 1945 (the sequential test); Howard, Ramdas,\n  McAuliffe & Sekhon, *Annals of Statistics* 2021 (confidence sequences valid at any stopping time).\n- **Guarantees with abstention**: Jung, Brahman & Choi, *Trust or Escalate*, ICLR 2025; Angelopoulos\n  et al., *Learn Then Test*, *Annals of Applied Statistics* 2025; Angelopoulos et al., *Conformal\n  Risk Control*, ICLR 2024; Candès, Ilyas & Zrnic, *Probably Approximately Correct Labels*, 2026.\n- **Guarantees on generated claims**: Mohri & Hashimoto, *Language Models with Conformal Factuality\n  Guarantees*, ICML 2024 (drop sub-claims below a calibrated threshold; guarantee lost under\n  distribution shift); Rubin-Toles et al., *Coherent Factuality*, ICLR 2025 (calibrate over\n  connected claims, not one at a time).\n- **Few labels plus many predictions**: Angelopoulos et al., *Prediction-powered inference*,\n  *Science* 2023; Boyeau et al., *AutoEval Done Right*, ICML 2025; Dorner, Nastl & Hardt, *LLM as\n  Judge won't beat twice the data*, ICLR 2025.\n- **Correlated errors and self-preference**: Kim, Garg, Peng & Garg, *Correlated Errors in Large\n  Language Models*, ICML 2025; Panickssery, Bowman & Feng, NeurIPS 2024.\n- **Building on generated output**: Shumailov et al., *Nature* 2024 (replacing data with model\n  output collapses the tails first); Gerstgrasser et al., 2024 (accumulating model output beside the\n  original data keeps the error bounded however many generations pass, proven for linear models). The analogy to Deliberus is\n  loose, since those results concern training data, but it points the same way as § 1: what keeps\n  error bounded is keeping the originals and limiting how far generated material feeds on itself.\n- **Measured accuracy on nearby tasks**: entity matching (Peeters, Steiner & Bizer, EDBT 2025;\n  Steiner & Bizer 2026, a preprint); paraphrase (Nangia & Bowman, ACL 2019); claim matching\n  (Pisarevskaya & Zubiaga, COLING 2025; Vykopal et al., EMNLP 2025 Findings); decomposition (Min et\n  al., *FActScore*, EMNLP 2023; Wanner et al., *SEM 2024 and EMNLP 2025).\n- **Recall needs its own sample**: Anthony & Nobari 2026 (preprint, abstract only); Marchant &\n  Rubinstein, PVLDB 2017.\n\n---\n\n## 9. His position after the correction: an error rate is a cost, and the edge is a frontier\n\n**VERBATIM** (2026-09-29, the reply that admitted the canon line of § 1 with *\"Yes.\"*):\n\n> *\"And most error rates are amenable to different optimization/innovation attacks to lower it. Just\n> like Deliberus aims to lower the cost of bridging minds and perspectives. But of course it's a fine\n> balance between challenging the status quo and SoTA beliefs about how far and deep this can go, vs\n> accepting where the edge of possibility lies and empowering humans to review and contribute beyond\n> it (or to check the machines' mistakes without being overwhelmed). The more we succeed in this\n> designing of sturdy microtasks and other creative innovative challenging of what's possible to\n> automate safely and how to maximize human+machine augmentation and synergistic reliability/accuracy,\n> the less humans need to feel overwhelmed and the more they can ponder what's really truly\n> meaningfully decision-relevant on a higher and higher level and reach helpful insights and\n> agreements and fruitful nonzero sum collaboration faster. I think these intuitions hold weight.\"*\n\nHis instruction: *\"Document all this carefully.\"* What follows is Claude's reading: it takes the\nstatement apart into five claims, ties each to where the corpus already holds it, and checks each\nagainst what was measured and read today. None of the reading is ruled; his words are the position.\n\n### 9a. An error rate is a cost with an owner, not a property of the model\n\n**His parallel places this inside the founding conviction.** `vision.md` holds that the obstacle\nbetween minds is a cost, not a mystery: findable, and lowerable by infrastructure. He applies the same\nshape to the machine's side. An error rate is not a fixed fact about a model but a cost that design\ncan attack. The harness conviction says the same from the other direction: the quality of machine\nreasoning is a function of harness and grounding ([the-harness-conviction.md](the-harness-conviction.md)).\n\n**Every lever this document found is an attack of this kind, and none of them changes the model:**\n\n- split the judgment into microtasks so that part of it becomes checkable, as the invention law does\n  for splits ([bend-and-deliberus.md](bend-and-deliberus.md) § 5);\n- let the machine decline, and prove the bound only on what it answers (§ 5);\n- ask several ways and read the agreement as the confidence, which roughly halved calibration error\n  in *Trust or Escalate*;\n- check with a second model family, after measuring how often the two fail together (devil 10);\n- and model progress itself: on well-defined record matching, one 2026 preprint finds current models\n  erring at about the rate of the benchmark's own labels.\n\n**What keeps the intuition honest is that each attack is scored the same way**, by whether it moves a\nmeasured bound (§ 2). Not every attack pays. Putting reasons before verdicts in the parts judgement\n(2026-09-22) was a sound repair, and the census re-run afterwards did not move its distribution. An\nattack that has not moved a bound has not lowered anything yet.\n\n### 9b. The balance: push the edge, and respect it where it truly is\n\nThe two ways to get it wrong face each other.\n\n- **Accepting a limit that is not real**: an untried *\"machines can't do that\"* hardens into a\n  documented limitation that later readers inherit as fact. The recent record pushes the other way:\n  the splitter went from proposing to writing on his word (2026-09-21), and the keeper was built and\n  wired into auto-connect within a day (2026-09-22).\n- **Pretending there is no edge**: the two error classes the harness conviction reserves for people\n  (what the corpus silently lacks, and the frame the graph grew inside) have no key a machine could be\n  checked against. Neither do contested microtasks, such as exactly where to cut or whether a part is\n  necessary.\n\n**The three kinds of truth of § 3 are a map of that edge, and the edge moves.** A contested task\nbecomes a consensus task when careful people turn out to agree, which is exactly what his prior about\nsplitting predicts and what a second person sitting the same items would test. A consensus task\nbecomes partly checkable when someone writes a law for part of it. That movement is what\n*\"optimization/innovation attacks\"* means at the level of tasks rather than models. What stays past\nthe edge, for now or for good, is where people *\"review and contribute beyond it\"*, and the design\nduty there is his parenthesis: *\"without being overwhelmed\"*.\n\n### 9c. Sturdy microtasks, on both sides of the edge\n\n**His phrase continues a thread he started fifteen years ago.** On 2011-12-29: *\"The general strategy\nis to leverage automation as far as possible, then move on to crowdsourcing; designing microtasks for\nusers to do the work that still is beyond computers.\"* On 2012-01-01: *\"Yes, designing microtasks will\nbe key, just need to design macrotasks first to be able to split them up.\"* (both in\n[split-review-and-the-founders-intentions.md](split-review-and-the-founders-intentions.md) § 1a). The\nmachine's share has grown since. The word has not changed.\n\n**A sturdy microtask**, as this reading takes his phrase, is one small enough to have a clear answer,\nshaped so that its mistakes can be noticed, and routed to whichever side does it more reliably. It is\nthe interface on both sides of the edge:\n\n- **for the machine**, a checkable sub-question it can answer or decline (the invention law; the\n  keeper's *can't tell*);\n- **for people**, a judgment that takes seconds and can be made blind and smoothly: the sittings, and\n  the triage feed that sends attention only where the instruments are unsure.\n\nThe same decomposition that makes a claim addressable makes a task reliable. That is the project's\ncore move turned on its own workflow.\n\n**A worked instance, 2026-09-30** ([microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md)\n§ 3e). The coverage check passed parts that stated as fact what their claim denied or conceded: its\nquestion was right and was read by the words. Rewritten as one numbered question per part (*would the\nclaim be weaker if this were false?*), it caught all six harmful parts and answered 45 of 45 parts the\nsame way on repeat. Two lessons for sturdiness: ask per item rather than over the whole, and gate on the\none question that decides, since a first gate that also required a second label to agree held faithful\nparts.\n\n### 9d. Synergy is measurable, and it has a condition\n\nThe literature read today holds several cases where a person and a machine together beat either\nalone. Each has the same shape: the machine does the bulk, people do a small and well-chosen part, and\nthe combination carries a guarantee neither has alone.\n\n- **Cascades** (*Trust or Escalate*): the machine answers what it is sure of and escalates the rest,\n  and the escalated items were the ones people disagreed on most.\n- **Prediction-powered inference**: many machine labels plus a few human ones give a valid estimate\n  narrower than the human labels alone.\n- **Probably approximately correct labels**: people label only above an uncertainty cutoff, with a\n  guarantee on the error of the whole dataset.\n\n**The condition is that the human part is genuine.** Teams gain only when people interrogate rather\nthan accept ([what-human-judgment-is-for.md](what-human-judgment-is-for.md)), which is why a proof\nsample is labelled blind (devil 6), and why a sturdy microtask for people is one where their own\nmistakes can show (seeded wrongs, devil 7).\n\n### 9e. The payoff: attention climbs\n\n**The last sentence of his statement names where this leads**: fewer people overwhelmed, and more\nattention on *\"what's really truly meaningfully decision-relevant on a higher and higher level\"*,\nreaching agreements and nonzero-sum collaboration faster. The corpus holds each link of that chain:\n\n- **the same aim, sixteen days earlier**, in the housekeeping ruling: *\"to instead free up their\n  mental cycles for the uniquely human-needing tasks\"* (canon, 2026-09-13);\n- **where the freed attention should go**: the triage feed and the two reserved error classes, and the\n  canon line that an act's value is not its rung, since the most valuable human acts need a position\n  rather than fluency in the ontology;\n- **why it pays**: the cooperation synthesis, in which the shared bottleneck to nonzero-sum\n  collaboration is the cost of mutual understanding\n  ([non-zero-sum-economics-and-civilizational-cooperation.md](non-zero-sum-economics-and-civilizational-cooperation.md)).\n\n### 9f. How the intuitions hold up\n\nHeld fallibly, as the convictions list asks, and checked against what was measured and read today.\n\n- **Consistent with every result read today.** Each lever in 9a lowers an error rate that the\n  literature measured, and each synergy in 9d is a published result rather than a hope.\n- **Where the evidence pushes back, it pushes on pace, not direction, and less than first written.**\n  This bullet first cited a guarantee sparing people 14–28% of the labelling at a 5% target; those\n  were zero-shot labels on categories where careful people disagree (news bias, headline stance,\n  misinformation), a poor prior for specified microtasks like ours\n  ([microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 1). The abstention figure\n  (56–68% of items answered at 80–90% agreement) comes from pairwise preference judgments, also a\n  contested kind. Treat both as floors. Our own measured error came largely from our instructions,\n  which is the attack that moves the edge fastest.\n- **Two limits no attack on the model removes.** Past the key an error rate is undefined, so the\n  attack has to be on the task (make a key) or on the human interface (make review cheap). And\n  correlated errors mean some attacks buy less than they appear to, since a second model agrees with\n  the first on wrong answers more often than chance.\n- **One condition the payoff needs.** Freed attention helps only if it is sent somewhere. That is\n  ledger R13: automation without routing produces a tidier graph that nobody looks at.\n\n**Verdict, unruled: the intuitions carry weight.** The evidence is strongest for their direction\n(error rates are lowerable, and synergy is real) and thinnest for their pace (how fast the edge\nmoves), which only running the programme can show.\n\n**Admitted as a canon line, 2026-09-29.** Offered a draft, the founder: *\"Add it as written (except\nremove the instructions examples parenthesis, instead just let it say generally that it's about the\nprompting rules for the machine passes since we want as deterministic/idempotent responses as\npossible)\"*. The line, now in CLAUDE.md's principles: *An error rate belongs to the task as written\nbefore it belongs to the model*; much of the error measured in our machine work has come from the\nprompting rules we give the passes, so a bad rate is first a design problem, since we want each\npass's answers as deterministic and idempotent as possible; each microtask made sturdier moves out\nthe point where people must check. The evidence that earned it is\n[microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md).\n\n---\n\n## Cross-references\n\n[earning-the-right-to-write.md](earning-the-right-to-write.md) (the ladder) ·\n[machine-work-as-a-standing-property.md](machine-work-as-a-standing-property.md) § 3d (the\ncalibration principle) · [split-review-and-the-founders-intentions.md](split-review-and-the-founders-intentions.md)\n(the splitter's record, § 3) · [the-parts-review.md](the-parts-review.md) ·\n[soft-canonical-clustering-and-reversible-merge-semantics.md](soft-canonical-clustering-and-reversible-merge-semantics.md)\n(the merge rulings and the evidence gate) · [auto-connect-upgrade.md](auto-connect-upgrade.md) § 2c\n(the keeper) · [major-transitions-heredity-and-evolvability.md](major-transitions-heredity-and-evolvability.md)\n§§ 3, 7–8 (the rules this corrects) · [drowning-in-claims.md](drowning-in-claims.md) ·\n[bend-and-deliberus.md](bend-and-deliberus.md) (laws for code, and the order-dependence measurement) ·\n[pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) (what tolerance\neach kind of error needs, which model that implies, and the labelled sample as the thing that\noutlasts a model) ·\nledger rows R8, R12, R13, S2, S5, T1–T4 in [session26-open-rulings.md](session26-open-rulings.md) ·\n[listening-for-the-calculus-and-sturdy-microtasks.md](listening-for-the-calculus-and-sturdy-microtasks.md)\n§ 3 (episodes and talks on these methods, found 2026-10-05: the author of the guarantees in § 5, and\npractitioners on checking a machine judge against one person's labels).\n· [the-evals-loop-and-our-machine-passes.md](the-evals-loop-and-our-machine-passes.md) (2026-10-05: that practice read in full. It adds what this document lacks: a\nstopping rule for *finding* kinds of fault, which comes before any proof, since § 4's reset means a\nproof made while the instructions still change is paid for twice; and a share of the labels kept\nsealed until a wording is frozen. The order is a standing principle since 2026-10-06, on the founder's\n*\"Sure\"*: a person reads what a pass produced before anyone proves how often it is wrong)\n"}