{"path":"research/the-evals-loop-and-our-machine-passes.md","content":"# The evals loop, read against our machine passes\n\n**Status (2026-10-05)**: Claude's reading of one podcast episode and its authors' written guides,\nmade at the founder's request. **Two of its proposals are ruled: the note box (Y10), built the same\nevening, and the order, read before proving (Y9), a standing principle since 2026-10-06. The rest are\nunruled.** They are in the ruling ledger as Y9 to Y14 ([session26-open-rulings.md](session26-open-rulings.md), Arc Y).\n\n**The request**, verbatim: *\"Look into this one deeply and relate it to all parts and processes and\nmachine passes and the \"machine work as graph property\" doc etc etc. Ground extremely deeply before\nyou think deeply about this. Then make an audio walkthrough about it.\"* The walkthrough is\n[docs/listening/2026-10-05-the-evals-loop-and-our-machine-passes.md](../listening/2026-10-05-the-evals-loop-and-our-machine-passes.md).\n\n**What was read, and how far**\n\n| Source | Read |\n|---|---|\n| *Why AI evals are the hottest new skill for product builders*, Hamel Husain and Shreya Shankar on Lenny's Podcast, 25 September 2025, 1 h 46 min ([YouTube](https://www.youtube.com/watch?v=BsWxPI9UM4c)) | The transcript in full, twice (19,402 words, from a public archive of the show's transcripts). Not listened to. Times below are the transcript's. |\n| *AI Evals: Everything You Need to Know*, Husain and Shankar, published 18 September 2026 ([hamel.dev](https://hamel.dev/blog/posts/evals-faq/)) | About twenty of its sixty answers in full: sample sizes, scoring, validation, who labels, what a machine may do, sampling, review screens, products that are hard to check, material in a language the builder does not read. The rest by heading only. |\n| *Using LLM-as-a-Judge For Evaluation*, Husain, October 2024, revised September 2026 ([hamel.dev](https://hamel.dev/blog/posts/llm-judge/)) | The steps on the expert, on pass or fail with a written reason, and on validation, in full. The steps on building a dataset by heading only. |\n| *Who Validates the Validators?*, Shankar and four co-authors, April 2024 ([arXiv:2404.12272](https://arxiv.org/abs/2404.12272)) | Abstract only. |\n| Our side | [machine-work-as-a-standing-property.md](machine-work-as-a-standing-property.md), [proving-a-pass-reliable.md](proving-a-pass-reliable.md), [microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) and [pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) in full; [earning-the-right-to-write.md](earning-the-right-to-write.md) §§ 3–4; the code of both review pages and of their scoring; the two review tables on the live site, both empty at 18:43 on 2026-10-05 and again at 22:05, after the note box went live. |\n\nThe texts as they were read are kept privately (`.private/sources/evals-episode-and-guides-read-2026-10-05/`), since the transcript and the guides are other people's writing and a web page can change.\n\n**Filed to be read later, nothing read yet.** The founder sent these as further material for this note:\n\n| Source | Sent | Read |\n|---|---|---|\n| *Evals 101*, Doug Guthrie of Braintrust, on the AI Engineer channel ([YouTube](https://youtu.be/bk0TmxoZlUY)) | 2026-10-08: *\"File this to be processed later as part of the evals materials\"* | Not read. Only its title and channel are known, from YouTube's own record. |\n\n---\n\n## In plain language\n\nThe two guests teach a procedure for finding out where a machine's output goes wrong, and for\nkeeping it from going wrong again. They call it *evals*. Its core is unglamorous: **a person who\nknows the subject reads real output and writes down what is wrong with it.** Everything after that\nis built from those notes.\n\nWe have built most of the later steps of their procedure, two of them more carefully than they\ndescribe. **We skipped the first step.** No person has read our machine output and written notes on\nit in any form that was kept. The reading has been done by Claude, and a machine reading a\nmachine's work is the one arrangement they say does not work.\n\nThree things they do that we do not:\n\n1. **They read before they measure.** Free notes on real cases come first, and stop when no new\n   kind of fault turns up. We went straight to measuring.\n2. **They report two numbers for every machine checker**: how many of the real faults it caught,\n   and how many of the sound cases it let through. We report one agreement figure.\n3. **They keep part of the person's labels sealed** until the wording of a check is frozen. We have\n   no such share.\n\nThree things we do that they do not:\n\n1. **We test the person.** Our sittings plant pairs that are certainly wrong, to see whether the\n   reviewer is reading.\n2. **We measure what seeing the machine's answer does to the person's answer**, by serving some\n   items without it.\n3. **We sort questions by what kind of truth they have.** They let one trusted person settle every\n   question. We hold that some questions cannot be settled that way.\n\n---\n\n## 1. The loop they teach\n\nSix steps. The times are where the episode covers each; the numbers in brackets come from the\nwritten guides, which are more exact than the conversation.\n\n**Step 1. A person reads real cases and writes a note on each.** One person, the one who knows the\nsubject best (0:25:41, where they call this person a *benevolent dictator*, meaning one trusted\njudge in place of a committee). The note is free text, and it records **the first thing that is\nwrong**, then the reader moves on (0:21:56). No list of criteria exists yet. Their example is an\nassistant for people who rent out apartments. A customer asks for a flat with a study. The machine\nsays none is free and wishes them a nice day. Nothing in that answer is false. The note reads\n*should have handed off to a human* (0:17:59). A machine asked to do this reading, they say,\nmostly reports that a conversation looks fine, because it lacks the context (0:23:55). The example\ngiven there is another conversation, in which the assistant offered a virtual tour that the company\ndoes not have. [At least the first thirty cases are read by hand before any machine suggests a\nfault, since its guesses would steer the reader.]\n\n**The reader stops when nothing new turns up** (0:30:07). They borrow the name *theoretical\nsaturation* from social science: you have read enough when further cases stop showing new kinds of\nfault. About a hundred cases as a rule of thumb, sometimes forty, sometimes fifteen.\n\n**Step 2. Group the notes and count.** A machine sorts the notes into kinds of fault, and here they\nwelcome it (0:31:42). The person renames the kinds until each one says what to do about it: *human\nhandoff issue* is useful, *capability limitation* is not (0:34:49). They add a kind called *none of\nthe above*, so the machine can say the list is incomplete (0:43:34). Then a plain count per kind\n(0:44:40). The count is not the whole ranking, since a rare fault can be the worst one.\n\n**Step 3. Fix what is obvious.** Many kinds vanish with one sentence added to the instructions, or\none bug fixed (0:46:07). No checker is built for those.\n\n**Step 4. For the stubborn kinds, build a checker each.** Code where a rule can decide. A machine\njudge where judgment is needed, and then a narrow one: **one kind of fault, answered yes or no**\n(0:50:49). Never a score from one to five, which they call a way of not deciding (0:52:16). They\nend up with four to seven such judges for a whole product (1:05:19).\n\n**Step 5. Check the judge against the person.** A judge is trusted only as far as it agrees with\nthe person's own labels (0:56:28). One agreement figure misleads: if a fault shows up in one case\nout of ten, a judge that always says *fine* agrees nine times in ten and catches nothing (0:58:41).\nSo they look at a table of four cells and at the two ways of being wrong (0:59:18). [One hundred to\ntwo hundred labelled cases for each kind of fault. The labels are split three ways: a tenth to a\nfifth may be shown to the judge as examples, about two fifths are used to improve its wording, and\nabout two fifths stay hidden until the wording is frozen and are then used once. Thirty to fifty\ncases of each answer in each of the last two shares.]\n\n**Step 6. Keep running it.** The judges run on every change and on a sample of live output\n(0:51:28). The person keeps reading: about half an hour a week after three or four days of work up\nfront (1:30:45). [Fresh reading after every change of instructions or of model.]\n\nTwo things hold the loop together. **The criteria move while the person reads** (1:03:35): people\nneed criteria to grade output, and grading output is how they find their criteria. The paper names\nthis *criteria drift*. And **the aim is a better product, not a complete set of checks** (1:26:37,\n1:30:16).\n\n**Read again on 2026-10-08.** The loop's first two steps are a rule drawn from compared cases.\nNobody names the kinds of fault before reading, and the kinds are what grouping the notes yields.\nThe founder made the connection while listening to a later walkthrough on what a rule rests on\n([what-a-rule-rests-on.md](what-a-rule-rests-on.md) § 13.6). That note also gives a reason the\nreading comes first: comparing cases is what makes a regularity visible, and having met the cases\none at a time is not enough (its § 4).\n\n---\n\n## 2. Where we stand on each step\n\n| Their step | What we have | State |\n|---|---|---|\n| 1. A person reads and notes | Claude has read stored output and written the findings. The founder corrects what he meets in conversation, case by case, and those corrections are not kept as a set. | **Not done by a person in any kept form** |\n| 1b. Stop when nothing new turns up | No stopping rule for reading. Our only stopping rule is for proof (59 clean cases bound the error under 5%, 149 under 2%: [proving § 2](proving-a-pass-reliable.md)). | Missing |\n| 2. Group and count | Done once, by Claude, over our own failures: of nineteen, eleven came from how the question was posed, six from the answer form or the code that received the answer, two from a model's habits ([pass-quality § 3](pass-quality-and-the-choice-of-model.md)). | Done by the machine; no person's notes to group |\n| 3. Fix what is obvious | The same nineteen: seventeen were fixed in the question or the code. | Done, and it matches their proportions |\n| 4. One checker per stubborn fault | Code checks exist (a quoted passage is really in its source; reasons come before verdicts; a question carries both its endpoints). One yes-or-no machine check exists and is steady: *would the claim be weaker if this part were false?* answered 45 of 45 parts the same way twice. Two many-way questions are not steady. | Partly |\n| 5. Check the judge against the person | Two sittings are built and live. Nobody has sat them: both tables held no rows at 18:43 on 2026-10-05. Their scoring reports single agreement figures. No sealed share. | Built, unused, and it would under-report |\n| 6. Keep running it | The standing-property proposal says a pass should be owed by a condition in the graph and not fired once at one door. Two such debts are defined (claims without an embedding, claims without a headline). Each is paid when somebody runs its job; nothing runs them on a schedule. | Proposed; two debts defined; no schedule |\n\n---\n\n## 3. What the episode adds\n\n### 3a. The first step was done by the wrong reader\n\nEvery measurement in the four pass documents was read by a model. The documents say so pass by\npass: *never by a person*, *no person*, *by Claude* ([pass-quality § 5b](pass-quality-and-the-choice-of-model.md)).\nThat reading found real defects, and they were all of one sort: faults visible from the text and\nthe structure alone. Three quarters of our part-of links are not parts. A check passed parts that\nstated what their claim denied. A verdict was computed and then never read.\n\nThe faults a model cannot find are the ones in their first example: output that is correct and\nstill not what the owner wants. We have met two of those, and **both times a person found them**. A\nfriend read the map of his own position and said his careful wording had gained weight it did not\nhave. The founder read a provisional label and said the page put too much emphasis on a guess.\nNeither fault would show in any count.\n\nThe paper's abstract gives the general form: machine-made checkers inherit the problems of the\nmachines they check. Our own canon has the same sentence, *never ask a mind to audit its own\nframe*, and our one attempt at a self-graded score was removed in August because nothing defined\nit.\n\n**The conversation is a kind of reading, and it has a blind side.** The guests grant that a team\nwhich uses its own product all day can get far without formal checks (1:13:53), and warn that most\nteams who say they do this do not do it closely enough (1:16:02). The founder reads the pages he\nvisits. He does not read links, strengths or bulk passes, and that is where our four largest\nsilent faults sat: fifteen days of failed extractions, 1,567 objections counted as support, 47\npairs falsely marked as the same claim, 84 false part-of links. **A layer nobody reads needs a\nsample that somebody reads.**\n\n### 3b. Two stopping rules, and we have one\n\nFinding the kinds of fault and proving a rate are different jobs with different stopping rules.\n\n- **Finding** stops when new cases stop showing new kinds of fault.\n- **Proving** stops at a sample size fixed in advance.\n\nWe have the second and had not named the first. The order matters because a proof holds only for\nthe instructions it was measured on ([proving § 4](proving-a-pass-reliable.md)). Each fault found\nchanges the instructions, and each change voids the proof. **So hours spent proving before the\nfinding has settled are spent twice.**\n\nTheir sizes also say how small our batches are. They ask for a hundred to two hundred labelled\ncases for one kind of fault, with at least fifty that pass and fifty that fail, so that both rates\ncan be measured. They also give a floor: under sixty cases the margins are usually too wide to\nconclude anything (the judge guide, on how many examples are needed). Our sameness sitting holds\n120 pairs. The batch of machine-cut parts holds 51 links and the general parts review 60 (the first\ntwo read from the live site on 2026-10-05 shortly after 23:10, the third from its definition in the\ncode).\nEach covers several kinds of fault at once, and the machine-cut batch is read per version of the\nsplitter as well. Those are sittings for finding, whatever they were built to be.\n\n**Ruled 2026-10-06: the order.** Asked whether to add it as a standing principle, in the words *\"A\nperson reads a machine pass before anyone measures it. Rates are measured only on the kinds of fault\nthat reading found.\"*, the founder: *\"Sure\"*. It is resident in `CLAUDE.md` with one word narrowed\nbefore it was written there: *measures* became *proves*. Every such line gets three checks (does the\nheadline alone forbid something we do and want, would a stranger follow it, does one of its words\nalready do another job here), and the first one failed. We count by code and by a second machine and\nwant to, and those counts belong to the finding. So the resident line forbids less than the sentence\nhe blessed. The checks should have been run before the sentence was put to him; they were run after.\nThe two stopping rules above are not in that sentence and stay open (ledger Y9b).\n\n### 3c. A note with every verdict\n\nTheir expert does not only press pass or fail. He writes why, and the written reasons do three\njobs: they become the examples the machine judge is shown, they show which rule is unclear, and\nthey make the expert notice his own inconsistencies.\n\nOur two review pages take a key press and nothing else. The store has a column for a note and the\ninterface that receives answers accepts one; the pages offer no place to type it (checked in the\npage code on 2026-10-05). A sitting held today would keep the verdicts and lose the reasons.\n\n**The pages are bare on purpose, and that purpose is his.** He asked on 2026-08-31 for *\"a test\nwhere I get to be more lazy but still show discernment\"*, going *\"smoothly\"* through the list\n([soft-canonical-clustering-and-reversible-merge-semantics.md](soft-canonical-clustering-and-reversible-merge-semantics.md),\nthe evidence gate). A note that had to be written would break that. So the proposal is a box that\nis optional and out of the way: left empty, nothing about the sitting changes. Whether even that is\nwanted is his call, which is why it was not built with this document.\n\n**Ruled and built the same evening.** His words: *\"Add it!\"*, and a few minutes later *\"Do it for\nall sittings/reviews I guess\"*. There turned out to be three places where a person judges machine\nwork, and all three now take an optional reason:\n\n| Where | How | Who can read the note |\n|---|---|---|\n| The sameness sitting | a closed box above the answers (a tap, or the N key) | only the reviewer |\n| The parts review | the same box | only the reviewer |\n| Keep, reword or drop on a claim's machine-written part | one line under each part | everyone: it is shown with the part |\n\nThe difference in the last column is deliberate. A note in a sitting is a reviewer thinking aloud,\nand the results pages are public, so they carry none. A reason given with a drop on a claim page\nexplains a public act on the public graph, where the dropped link stays visible by ruling. An answer\ngiven without a note is the same request it always was, a note comes back when the reviewer returns\nto the item, and answering again with the box emptied removes it.\n\nNot covered, and said so: the chooser for parts the splitter *held* for a person (keeping some and\nleaving others) records no reason for the ones left out.\n\n**A stored note needs a reader.** No page lists the notes written in a sitting, by design, so\nwithout a tool they would be kept and never used, which is the fault of\n[machine-work § 2b](machine-work-as-a-standing-property.md) made once more. The keeper's reader is\n`scripts/read_review_notes.py`: it prints every note grouped by the answer it came with, beside the\ntwo claims it is about, with reviewers numbered and never named. Grouping by answer is only a first\narrangement. Sorting the notes into kinds of fault, their second stage, stays a reader's work. The\nscript reads the live stores and writes nothing. Run on 2026-10-05 at 23:14 it printed\n*No notes yet*.\n\nFor the passes that write (reading a text into claims, splitting, headlines) no sitting of their\nfirst kind exists at all: one item, its source beside it, and a box for the first thing wrong.\n\n### 3d. Both kinds of miss\n\nOur parts review reports how often the person agreed with the machine's word, and how often they\nagreed on *a piece or not a piece*. Each is one figure. Our own numbers show why one figure is not\nenough. A model reading found that six of thirty-seven machine-written parts failed the test of a\npart ([microtasks § 3](microtasks-in-broad-daylight.md)). A checker that called all thirty-seven\n*a part* would agree with that reading 84 times in 100 and catch none of the six.\n\nThe two questions to ask of any checker are these. **Of the faults the person found, how many did\nit catch? Of the sound items, how many did it let through?** The principle is already in our\ndocuments ([pass-quality § 2c](pass-quality-and-the-choice-of-model.md): which way the errors lean\nmatters more than how many there are). The built reader does not apply it.\n\nThe sameness sitting has a different gap. Its scoring never compares the machine's verdict on a\npair with the person's. It measures the person: whether they caught the planted wrongs, and whether\nseeing a proposal moved them. Both are worth having. Neither says how good the machine is.\n\nThe same holds for every detector we run. The detector of weighing language, the detector of\nopposing stances, and the alarm for softened disagreement are all checkers in their sense, and none\nhas its two rates against a person's labels. One was found matching nothing at all, 0 of 1,678\nclaims, months after it shipped.\n\nA corollary from their written guide: **the labelled set should hold cases whose right answer is a\nrefusal.** A machine's *cannot tell* or *fits none of these* can be scored only if some items are\nknown to deserve it. Our scheme classifier's refusal fired 0 times in 3,666, and nothing would have\nshown that except a count taken by hand.\n\n### 3e. A share kept sealed\n\nNone of our four pass documents mentions a share of labels kept out of sight. The written guide\ngives the reason one is needed, and it is subtle: every time a disagreement is read and a wording\nchanged, information from those cases enters the wording, **even if no case is ever shown to the\nmachine**. After enough rounds the check fits the cases it was tuned on and nothing else.\n\nThis bears directly on the proposal that each pass keeps an answer key and that a model is admitted\nby clearing it ([pass-quality § 5e](pass-quality-and-the-choice-of-model.md)). The key will be used\nfor two things: improving a wording, and admitting a model. If the same cases serve both, the\nadmission proves less than it appears to. The reader of the disagreements here is Claude, so the\nseal has to be mechanical: the sealed share is chosen when the labels are stored, the results page\nleaves it out, and every opening of it is counted.\n\n### 3f. Yes or no, and one fault at a time\n\nTheir two rules for a machine judge are that it answers yes or no, and that it looks for one kind\nof fault. What we measured agrees on both.\n\n- Our one steady machine check is a yes-or-no question about one thing.\n- Our five-step question of degree found no two models in agreement, and one model gave the same\n  answer to all twelve links ([pass-quality §§ 3, 5a](pass-quality-and-the-choice-of-model.md)).\n- Our seven-way sameness question is steady at its two firm ends and loose in the middle, where one\n  key was defined three ways in three places\n  ([microtasks § 2](microtasks-in-broad-daylight.md)).\n\nTheir remedy for a question of degree is to replace the scale with several yes-or-no questions and\ncount the yeses. That is the unbuilt proposal already on file for the sameness keeper (four yes-or-no\nquestions in place of seven keys: [microtasks § 4](microtasks-in-broad-daylight.md), item 2), and\nit now has outside support.\n\nTheir written guide adds a diagnostic worth keeping: **when a person cannot decide whether a case\npasses, the definition is what needs work.** That is our own finding about the loose key, reached\nfrom the other side.\n\n### 3g. Generic scores\n\nThey are blunt about ready-made scores such as likeness between two texts: used as a measure of\nquality they give false confidence, and their one good use is to find cases worth reading (1:21:26,\nand the written guide).\n\nOur alarm for softened disagreement is a score of exactly that type. It measures how alike two\nopposing claims are once embedded as numbers. Its own description is careful: it calls itself an\nalarm and not a proof, and names two things it cannot see. Its figures are quoted in our index as a\ndataset all the same (0.82 to 0.94 across five kinds of text). By the guests' standard that figure\nis a way of choosing which pairs to read, until someone has labelled a few dozen pairs as *softened*\nor *not softened* and the alarm has its two rates.\n\n### 3h. A weekly sitting\n\nThe standing-property document says a person should calibrate a pass with one sample per pass,\nand should not perform its judgments one by one ([§ 3d](machine-work-as-a-standing-property.md)).\nIt leaves the rhythm open. The guests supply one: about half an hour a week, ten to twenty fresh\ncases, a larger read after any change of instructions or model, and **some randomly drawn cases in\nevery batch**, because cases picked by a signal show only the faults that signal already knows.\n\nThat last rule matters for our planned feed of things a person should look at\n([the-ratification-record-and-the-triage-feed.md](the-ratification-record-and-the-triage-feed.md)).\nA feed built only from the machine's own doubts shows what the machine already doubts.\n\n### 3i. Cheaper models, and searching for a wording\n\nOnce checks exist, they swap dear models for cheap ones and watch whether the two rates hold\n(1:34:49). That is our answer-key proposal seen from practice.\n\nThe written guide goes one step further, and names the decision-only model we have benchmarked\n([jev-and-the-judgement-layer.md](jev-and-the-judgement-layer.md)). Its value in their account is\nspeed and price: with labelled cases in hand, a program can try hundreds of wordings of a check and\nkeep the one that agrees best with the person. **A labelled set turns the writing of instructions\nfrom a craft into a search.** That is the founder's own position, that most error rates can be\nworked down, given an engine. The engine needs the sealed share of § 3e, or the search fits itself\nto its own cases. The guide names one such search program, GEPA, as an example.\n\nIt also says when not to reach for it: read a few of the disagreements by hand first, because a\nsearch over wordings cannot find a fault that is not in the wording, such as context the check was\nnever given or a wrong label. Our own record puts a number on that. Of nineteen measured failures\nof machine passes, eleven lay in how the question was posed, six in the form of the answer or in\nthe code that received it, and two in a model's own habits\n([pass-quality § 3](pass-quality-and-the-choice-of-model.md), Claude's sorting). **A search over\nwordings could reach the eleven at most.** The six sat in the shape of the answer or in the code\nthat received it, where no wording reaches.\n\n### 3j. What the first sitting will do\n\nCriteria drift predicts that the seven sameness keys will not survive first contact unchanged.\nThey have moved once already, before anyone sat: an *opposes* key was added. Two consequences. The\nfirst sitting should be planned as reading, with rewording expected. And answers given under one\nwording should never be pooled with answers given under another, which the store allows today only\nby starting a sitting under a new name.\n\n---\n\n## 4. What we have that they do not\n\n**A test of the person.** Roughly one pair in six in the sameness sitting is certainly wrong (the\ntwo claims attack each other, so they cannot be the same claim). A reviewer who calls those *the\nsame* is not reading, and the sitting says so. Their method has no check on the expert.\n\n**A measurement of anchoring.** They warn that a machine's early suggestions can steer the reader.\nWe serve one item in five without the machine's proposal and report the difference.\n\n**Three kinds of truth.** [Proving § 3](proving-a-pass-reliable.md) sorts every question a pass\nanswers: checkable against the text, agreed among careful readers, or contested. Their method\ntreats all questions as the second kind.\n\n**The refusal as a gate.** They put *none of the above* on the list beside the answers. We measured\nthat a refusal on the list loses to its neighbours, and ask it first, as its own question.\n\n**Work laid out to be checked.** Their written guide treats *this product is hard to evaluate* as\na fault in the product's design, and its remedy is to show the steps between the input and the\nfinished output, so that a person can check each one. Its example is a medical report: show the\ndoctor the extracted facts, each linked to its source, and let her correct one before the report\nis written. A claim graph is that layout made the product. A claim read from a source carries the\npassage it was read from, and every link is a separate thing to accept or dispute. Their advice on the review screen we mostly meet: number keys, a note without\nleaving the keyboard (the N key), and a count of how far along the reviewer is. One part we do not\nmeet: they want what the reviewer needs on the same screen, and the passage a claim was read from\nis one tap away, in a new tab. Whether it belongs on the screen is left until he has sat a batch,\nsince his own notes will say whether he needed it.\n\n**What compounds.** Their loop treats each check alone. We hold that an error rate stays the same\nshare at any volume, and that only machine work built on machine work grows it, so depth is what\nhas to be capped.\n\n---\n\n## 5. Where it does not transfer\n\n**One judge cannot settle a contested question.** Their written guide does discuss several\nreviewers: measure how far they agree beyond chance, find the rule that would have settled the\ncase, and where they still differ let the expert decide and record why. That suits a product, where\nsomebody owns the definition of good. It does not suit a question on which careful people differ\nfor reasons. There the spread between readers is the measurement, and a machine can be asked only\nto fall inside it.\n\n**A rendering of someone's position is theirs to judge.** Their expert decides whether an email is\ngood. Nobody in the building can decide whether a claim says what its author meant. A reviewer can\ncheck that the quoted passage is really in the source. The author is the only judge of the rest.\n\n**Their checks are never gamed.** Nothing in their setting adapts to the check. Ours will be\npublished and will meet people with a stake. A judge validated on today's cases says nothing about\na case written to pass it.\n\n**A check that always passes.** They retire one. For us a check that never fires is unproven until\nit has been shown to fire. The two rules agree once their own caveat is added: before building a\ncheck, produce the fault once.\n\n**The reader has to read the language.** Their guide answers the builder whose product speaks a\nlanguage he does not: a machine translation loses meaning and tone, so bring in a reviewer who\nknows both the language and the setting. Measured on the live graph on 2026-10-05: one source of\n33 is Swedish, an essay read in April. Its 120 claims are stored in English, and 116 of their 119\nquoted passages are Swedish. **So the pass that read it also translated it**, and by instruction:\nthe two extraction prompts say to translate all output to English when the source is not\n(`extraction/scout.py`, `extraction/focused.py`), while the splitter is told to answer in the claim's\nown language. Translation is the step their guide warns about, and nothing here has checked a\ntranslated claim against its passage. No\nclaim text in the whole graph holds an å, ä or ö (0 of 5,107), so a count of Swedish letters cannot\nfind these claims; the passages can. Neither of today's two batches touches that source (0 of 120\npairs, 0 of 51 links). Two consequences. A claim from a Swedish source needs a sitter who reads\nSwedish, which the founder does and a friend abroad may not. And today's batches say nothing about\nhow the passes read Swedish.\n\n**Their budget assumes a team.** Three or four days up front and half an hour a week is a team's\nbudget. Ours is one person's hours, which is the case for a short recurring sitting over a long\nsingle one.\n\n---\n\n## 6. One reading further\n\n*Claude's reading, and the least tested part of this document.*\n\nThe host remarks that a judge's instructions are a requirements document that runs (1:00:56): they\nstate, case by case, when a behaviour is owed. Carry that across. Each of our microtasks answers a\nquestion whose terms are the ontology's own: *the same claim*, *a part of*, *supports*. The\ninstructions for the task, together with the cases a person has labelled, are the working\ndefinition of that term. An edge type means, in practice, what its labelled cases say it means.\n\nIf that is right, the sittings do a second job nobody assigned them. A term whose cases careful\nreaders label alike is stable enough to be built into the structure. A term whose cases they label\ndifferently is contested, and by the test in\n[what-belongs-in-the-ontology.md](what-belongs-in-the-ontology.md) belongs among the claims and not\nin the structure. So the spread between two sitters would be evidence about the ontology itself.\n\nIt also sets a limit on one of our working beliefs. For ordinary software we hold that the\nspecification is the work. For a task of judgment the criteria are found by reading output, so the\nspecification is a result of the loop and cannot be its starting point.\n\n---\n\n## 7. Proposals, in order (unruled unless marked)\n\n| Ledger | Proposal | What it costs |\n|---|---|---|\n| Y9 | **Read before proving.** A person reads a pass's output and notes what is wrong before anyone proves a rate, and a rate is proven only for the kinds of fault found. Ruled 2026-10-06 (*\"Sure\"*): § 3b. | Nothing built; a change of plan for the first sittings |\n| Y9b | **The two stopping rules.** Finding stops when no new kind of fault turns up. Sampling for proof starts when the instructions have stopped changing. | Nothing built |\n| Y10 | **A note with every verdict.** Ruled and built 2026-10-05 (*\"Add it!\"*, *\"Do it for all sittings/reviews I guess\"*): § 3c. | Built: three places |\n| Y10b | **A first-thing-wrong sitting for the passes that write**: one item, its source beside it, and a box. | A new page |\n| Y11 | **A sealed share.** Two fifths of every labelled set is set aside when stored, left out of the results, and opened once to admit a wording or a model, each opening counted. | Small code. With 120 pairs it leaves 72 to learn from. |\n| Y12 | **Two rates, never one agreement figure**, for every machine checker and every detector, and labelled sets that include cases whose right answer is a refusal. The softened-disagreement figure is quoted as a way of finding pairs to read until it has its two rates. | A change to two results pages; a few dozen labels per detector |\n| Y13 | **His own labels first.** The first answer key for each pass is the keeper's labels alone, for questions that are checkable or agreed. A second sitter measures the spread afterwards. Contested questions are never settled by one person. | Loosens the protocol of 2026-08-31, which had friends sit the same test before any bar was set |\n| Y14 | **A short weekly sitting**: about twenty fresh items, some always drawn at random. | His time, about half an hour a week |\n\nNot proposed, and why: a committee of labellers (their advice against it holds for the first two\nkinds of truth); a refusal placed on the list of answers (measured here to fail); scores from one to\nfive anywhere (measured here to be unsteady).\n\n---\n\n## 8. What would show this reading wrong\n\n- If the founder's first sitting produces notes that only repeat what the model readings already\n  found, § 3a is overstated: the model was a good enough first reader for these passes.\n- If the keys and their definitions come through the first sitting unchanged, § 3j is wrong here,\n  and proof could have started at once.\n- If two sitters agree on the questions we call contested, § 5's first point is wrong for those\n  questions, and one judge would have served.\n\n---\n\n## Cross-references\n\n- [machine-work-as-a-standing-property.md](machine-work-as-a-standing-property.md) § 3d: the person\n  calibrates a pass. This document adds the rhythm and the two rates.\n- [proving-a-pass-reliable.md](proving-a-pass-reliable.md) §§ 2–4: sample sizes, kinds of truth, and\n  what resets a proof. This document adds the stopping rule for reading.\n- [pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) § 5e: the\n  answer key. This document adds the sealed share.\n- [microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 4: yes-or-no questions for\n  the keeper.\n- [earning-the-right-to-write.md](earning-the-right-to-write.md): the ladder a pass climbs. Their\n  live sampling with a margin of error is what its automatic demotion would run on.\n- [listening-for-the-calculus-and-sturdy-microtasks.md](listening-for-the-calculus-and-sturdy-microtasks.md):\n  the list this episode came from.\n- `scripts/render_listening.py` and [how-to-write-for-listening.md](../listening/how-to-write-for-listening.md):\n  the walkthrough of this document was the first rendered with marked pauses.\n"}