{"path":"listening/2026-10-05-the-evals-loop-and-our-machine-passes.md","content":"# The evals loop, and how we check our machines: a listening walkthrough (2026-10-05)\n\n- **What it is**: the eighth listening piece, on the founder's request: *\"Look into this one deeply\n  and relate it to all parts and processes and machine passes and the \"machine work as graph\n  property\" doc etc etc. Ground extremely deeply before you think deeply about this. Then make an\n  audio walkthrough about it.\"* The episode is Hamel Husain and Shreya Shankar on Lenny's Podcast\n  (25 September 2025). The research it walks through is\n  [the-evals-loop-and-our-machine-passes.md](../research/the-evals-loop-and-our-machine-passes.md),\n  written and filed before this was rendered.\n- **The first with room to think.** His request the same day: *\"I often need a second or three to\n  ponder a complex sentence. [...] At the very least I need a few seconds of silence sprinkled in\n  throughout the walkthrough at moments after you've delivered some sentence/s where there's many\n  concepts or inferences at play.\"* **A line reading `[[pause 3]]` below is not spoken: it is three\n  seconds of silence.** How that is done, and how to write for it:\n  [how-to-write-for-listening.md](how-to-write-for-listening.md) § Room to think.\n- **Covers**, in seven parts: the two guests and the one example they build on; their method's\n  first half, reading; its second half, checking; where we stand, with the finding that no person\n  has yet read our machine output and written notes on it; four things they do that we do not;\n  what we do that they do not, and where their advice stops fitting; and the six questions it ends\n  on (ledger Y9 to Y14, all open).\n- **Rendered** 2026-10-05, about 19:00: full `eleven_v4`, Lucy, eight requests cut at the part\n  boundaries, 18,018 characters for 2,180 credits. **24.2 minutes**: 19.8 of speech, slowed to 0.92\n  of the voice's own speed, and 4.4 of added silence (40 marked pauses, 106 seconds; 270 sentence\n  ends, 94; 56 paragraph breaks, 45; 7 part breaks, 18). 3,324 words: 168 a minute while speaking,\n  **138 a minute overall**. The seventh piece ran at 183.\n- **Checked**: a speech model heard, on each side of all 40 marked pauses, the words the text has\n  there. A first assembly had put one pause a word late and cut five others outside a real gap. The\n  cause was found before anything was sent (the voice service's report of when each sentence\n  begins can run a third of a second late), and the file was put together again from the same\n  rendered voice.\n- **Status**: a snapshot of 2026-10-05. Both review tables were empty when it was made (read at\n  18:43). The audio is kept out of git.\n- **What changed after it was rendered** (the text below is as spoken, and unchanged): the first\n  half of the second closing question, room for a note with every answer, was ruled the same\n  evening. His words: *\"Add it!\"*, then *\"Do it for all sittings/reviews I guess\"*. A reviewer can\n  now write a reason with each answer, in three places, live since 22:06. Its second half, a\n  sitting for the passes that write, is open as Y10b. The first closing question, reading before\n  proving, was ruled on 2026-10-06 (*\"Sure\"*) and is a standing principle; its two stopping rules\n  are open as Y9b. The other four questions are open\n  ([the research doc](../research/the-evals-loop-and-our-machine-passes.md) § 3c and § 7).\n\n---\n\n[warmly] This is a walkthrough of one podcast episode, and of what it says about the way we check our machines.\n\nBefore the content, one thing about the sound. You told me the walkthroughs go too fast. In your words: \"I often need a second or three to ponder a complex sentence.\" So this one has silences in it, on purpose. When the voice stops for a few seconds, nothing is broken. That is your time to think.\n\n[[pause 3]]\n\nThe voice is also a little slower. And I say fewer things per minute. When a sentence carries a lot, I say it again in plainer words, or I give an example.\n\nHere is the map. There are seven parts. Part one is the two guests, and the one example they build everything on. Part two is the first half of their method, which is reading. Part three is the second half, which is checking. Part four is where we stand, and it holds the uncomfortable finding. Part five is four things they do that we do not. Part six is what we do that they do not, and where their advice stops fitting us. Part seven is short: six questions waiting for you.\n\nYour request was this. \"Look into this one deeply and relate it to all parts and processes and machine passes and the machine work as graph property doc, et cetera. Ground extremely deeply before you think deeply about this. Then make an audio walkthrough about it.\"\n\nHow deep did I go? I read the full transcript twice, since I cannot listen. I read the two guests' own written guides, which give exact numbers the conversation only hints at. And I read the summary of the research paper they cite. On our side, I read our four documents on machine passes again, and the code behind your two review pages.\n\n---\n\nPart one. Two teachers and an apartment.\n\nThe guests are Hamel Husain and Shreya Shankar. They teach a course on checking the output of language models. Several thousand engineers and product managers have taken it. Shankar is also the researcher behind a paper we will come to.\n\nTheir word for the subject is evals, short for evaluations. It sounds technical. Their own definition is plain. An eval is a systematic way to measure how well a machine does its job, so that you can improve it.\n\n[[pause 2]]\n\nEverything they say hangs on one example. So let me give it to you slowly.\n\nA company makes an assistant for people who rent out apartments. Customers send it text messages. It answers their questions, it books viewings, and it passes the hard cases to a human.\n\nHere is one real conversation. A customer writes: do you have a one-bedroom with a study? The assistant looks it up. It answers: we have one-bedrooms, but none with a study. The customer asks: can you tell me when one comes free? The assistant says it has no information about that. The customer says thank you. The assistant says you're welcome.\n\nNow a question for you. Was anything in that conversation wrong?\n\n[[pause 4]]\n\nEvery sentence the assistant wrote was true. And the conversation still failed. Someone who wanted an apartment was sent away. The whole purpose of the product is to turn people like that into tenants. The right move was to hand the conversation to a human.\n\n[[pause 2]]\n\nThat is the heart of the episode. So here it is once more, in other words. The answer was correct, and it was not what the owner wanted. No fact was wrong. What was wrong was a judgment about what the product is for.\n\n[[pause 3]]\n\nAnd here is why that matters for us. Shankar says that when they ask a language model to do this kind of reading, it mostly reports that the conversation looks fine. It lacks the context. Her own example is a different conversation, in which the assistant offered a virtual tour. The company has no virtual tours. Only someone who knows the company can see that fault.\n\nSo, part one in a line. The faults that matter most can be invisible to a machine. They are about purpose, and purpose lives in a person's head.\n\n---\n\nPart two. Their method, first half: reading.\n\nTheir method has six steps. The first three are about reading. I will take them one at a time.\n\nStep one. One person reads real output, and writes a short note on each case.\n\nThree details matter here.\n\nFirst, it is one person. They call this person the benevolent dictator. That means one trusted judge who knows the subject, in place of a committee. Their reason is cost. If reading needs a meeting, the reading does not happen.\n\nSecond, the note is free text. There is no form, and no list of criteria yet. The reader writes the first thing that is wrong, and moves on. For the apartment conversation, the note was seven words. Should have handed off to a human.\n\nThird, the person reads by hand before any machine suggests anything. Their written guide says at least the first thirty cases. The reason is that a machine's suggestions steer the reader.\n\n[[pause 2]]\n\nWhen does the reader stop? The idea is simple. You stop when new cases stop teaching you new kinds of fault.\n\n[[pause 2]]\n\nHere is how that looks. Suppose you read ten cases and find four kinds of fault. You read ten more, and find two new kinds. Ten more, and one. Then twenty more, and nothing new. That is the moment to stop. The name for it, borrowed from social science, is saturation. In practice it tends to arrive around a hundred cases, sometimes after forty.\n\n[[pause 2]]\n\nStep two. Group the notes, and count.\n\nNow the machine is welcome. It sorts a hundred notes into a handful of kinds. The person then renames the kinds, until each name says what to do. Their example: a kind named capability limitation tells you nothing. A kind named human handoff issue tells you what to fix.\n\nThen you count. In their example, one pile held seventeen cases of muddled conversation flow. Counting is the whole technique. Husain calls it the most powerful tool in data science, because it is so simple that people undervalue it.\n\n[[pause 2]]\n\nStep three. Fix what is obvious.\n\nMany kinds of fault vanish with one sentence added to the instructions. Nobody told the machine how to format its answer, so you tell it. No checker gets built for those.\n\nSo, part two in a line. Read real cases. Write down what is wrong. Stop when nothing new turns up. Count the kinds. And fix the easy ones at once.\n\n---\n\nPart three. Their method, second half: checking.\n\nSome kinds of fault survive the easy fixes. For each of those, they build a checker. That is step four.\n\nIf a rule can decide, the checker is ordinary code. Is the answer short enough? Code can say.\n\nIf judgment is needed, the checker is a second language model. They call it a judge. And they give the judge two strict limits.\n\nIt looks for one kind of fault only. And it answers yes or no.\n\n[[pause 2]]\n\nWhy not a score from one to five? Their answer is sharp. A score lets you avoid deciding. And nobody can say what three point two means, next to three point seven.\n\n[[pause 2]]\n\nStep five is the one I most want you to hold on to. Before you trust the judge, you check it against the person.\n\nThe person has already labelled cases by hand. So you run the judge on the same cases, and compare. The tempting thing is to report one number. The judge agrees with the person ninety percent of the time. They say that number is dangerous. Here is why.\n\nSuppose the fault shows up in one case out of ten. Now imagine a judge that is completely useless. It says fine to everything. How often does it agree with the person?\n\n[[pause 4]]\n\nNine times out of ten. It is right about all nine sound cases. It misses the one faulty case. Ninety percent agreement, and it has never caught a single fault.\n\n[[pause 3]]\n\nSo they ask two questions in place of one. Of the real faults, how many did the judge catch? And of the sound cases, how many did it let through? A good judge needs both numbers high. The useless judge scores zero on the first.\n\n[[pause 3]]\n\nTheir written guide adds one more discipline, and it is subtle. Part of the person's labels are kept sealed. Nobody looks at them while the judge's wording is being improved. When the wording is finished, the sealed cases are used once, as the final test.\n\nWhy seal them? Because every time you read a disagreement and adjust the wording, a little of that case leaks into the wording. Do that for long enough, and the judge fits the cases you tuned it on, and nothing else.\n\n[[pause 2]]\n\nIt is the same reason a teacher does not hand out the exam questions as homework.\n\n[[pause 2]]\n\nStep six. Keep running it. The checkers run whenever anything changes. And the person keeps reading. After a few days of work at the start, it takes about half an hour a week.\n\nOne more idea holds all six steps together. It comes from Shankar's research, and its name is criteria drift. People need criteria to grade output. But grading output is how people discover their criteria. So the criteria move while you read.\n\n[[pause 3]]\n\nIn plainer words: you cannot write the rulebook first. You find out what you think by looking at cases.\n\nSo, part three in a line. One narrow yes-or-no checker for each stubborn fault. Test it against a person's labels, with two numbers, on cases kept sealed. And expect your own standards to shift as you go.\n\n---\n\nPart four. Where we stand.\n\n[thoughtful] Now I hold our own work against those six steps. I will start with the uncomfortable one.\n\nStep one is a person reading, and writing notes. We have not done it.\n\n[[pause 3]]\n\nLet me be exact about that, because it is my doing. Everything we measured about our machine passes this autumn was read by a model. Mostly by me. Our own documents say so, pass by pass: never read by a person. And both of your review pages are still empty. I checked this evening, at a quarter to seven.\n\nA machine reading a machine's work is the arrangement the guests say does not work. So what did my reading find? And what could it not find?\n\nIt found real things. Three quarters of our links that say one claim is a part of another are no such thing. A check was letting through parts that said the opposite of their claim. A verdict was being computed and then never read. Those are faults you can see from the text and the structure alone.\n\n[[pause 2]]\n\nWhat it could not find is the apartment kind of fault. Output that is correct, and still not what you want.\n\nWe have met that kind twice. Both times, a person found it. A friend of yours read the map of his own position. He said his careful wording had gained a weight he never gave it. And you looked at a page, and said it put too much emphasis on what was only a guess. No count I ran would have shown either one.\n\n[[pause 3]]\n\nThere is a fair objection here. You do read our output. You read the pages you visit, you tell me what is wrong, and I fix it. The guests would call that using your own product. They grant that it can carry a team a long way.\n\nBut it has a blind side. You read pages. You do not read links, or strengths, or the bulk work that runs behind the pages. And that is exactly where our four worst silent faults sat. Fifteen days of failed extractions. More than fifteen hundred objections, counted as support. Forty-seven pairs wrongly marked as the same claim. Eighty-four false part-of links.\n\n[[pause 2]]\n\nSo here is the rule I take from it. A layer that nobody reads needs a sample that somebody reads.\n\n[[pause 3]]\n\nThe rest of the scorecard is kinder. Take step three, fixing the obvious. Of nineteen failures we measured, seventeen came from the question or the code, and only two from the model. That matches what they describe. Take step four, the narrow yes-or-no checker. We have one, and it is our steadiest. It asks: if this part were false, would the claim be weaker? Asked twice, it gave the same answer both times on forty-five parts out of forty-five. Our many-way questions are not that steady.\n\nSo, part four in a line. We built the later steps and skipped the first. And the reader we used in its place cannot see the faults that matter most.\n\n---\n\nPart five. Four things they do that we do not.\n\nFirst. They have two stopping rules, and we have one.\n\nFinding the kinds of fault is one job. Proving how rare a fault is, is another job. Our documents this autumn went straight to the second. We worked out that fifty-nine clean cases in a row show an error rate under five percent. I wrote them that way, and I never named the reading that has to come first.\n\nHere is why the order matters. A proof only holds for the instructions it was measured on. And each fault you find changes the instructions.\n\n[[pause 3]]\n\nThink about what that means for your hours. Say you sit down to prove a pass before the reading has settled. The first fault you find changes the wording. And the proof starts again from zero.\n\n[[pause 3]]\n\nSo reading comes first. Proving comes after the wording has stopped moving.\n\nSecond. A note with every verdict.\n\nTheir expert presses pass or fail, and then writes why. Those reasons turn out to be the most useful thing in the whole exercise. They show which rule is unclear. And they become the examples the checker learns from.\n\nYour two review pages take a key press and nothing more. The odd thing is that the storage behind them already has a place for a note. The pages simply never show a box to type in. If you sat today, we would keep your verdicts and lose your reasons.\n\n[[pause 3]]\n\nThird. Two numbers, where we report one.\n\nOur parts page reports how often you agree with the machine. One figure. And our own data shows the trap. In one reading, six of thirty-seven machine-written parts failed the test of a part. Now picture a checker that calls every one of them a part. It would agree eighty-four times in a hundred. And it would catch none of the six.\n\n[[pause 3]]\n\nFourth. The sealed share. None of our documents has one.\n\nThis matters for an idea I proposed this morning. Each pass keeps an answer key, and a new model is admitted by passing it. We would use that key for two things: to improve our wording, and to admit models. If the same cases do both jobs, the admission test is partly an open-book exam.\n\n[[pause 2]]\n\nAnd notice who reads the disagreements here. I do. So the seal cannot rest on my good behaviour. It has to be built into the page.\n\n[[pause 2]]\n\nSo, part five in a line. Read before proving. Keep the reasons. Report two numbers. And seal part of the key.\n\n---\n\nPart six. What we do that they do not, and where their advice stops fitting.\n\nIt would be wrong to leave you thinking they are ahead of us everywhere. In three places we are the more careful ones.\n\nWe test the person. One pair in six in your sameness sitting is planted, and certainly wrong. If a reviewer calls those the same, the reviewer is not reading, and the page says so. Their method never checks the expert.\n\nWe measure the pull of the machine's answer. They warn that seeing a suggestion steers the reader. We show one item in five without the suggestion, and compare.\n\nAnd we sort questions by their kind of truth. This is the important one.\n\n[[pause 2]]\n\nThey let one trusted person settle every question. That works for them. Somebody owns the product, so somebody owns the definition of good.\n\nSome of our questions are like that. Is this quoted passage really in the source? Anyone careful can check.\n\nOthers are not. Are these two value claims the same claim? Is this part required, or does it only add support? Careful people differ on those, and they differ for reasons. If one person's answer became the truth there, we would be building your reading into the machine, and calling it a measurement.\n\n[[pause 3]]\n\nAnd one kind of question belongs to nobody in the room. Does this claim say what its author meant? You can check the quote. Only the author can judge the rest.\n\nSo their single judge is right for two of our three kinds of question, and wrong for the third.\n\n[[pause 3]]\n\nTwo smaller differences. Nobody tries to fool their checkers. Ours will be public, and will meet people with a stake. And they put the answer none of the above on the list, beside the other answers. We have measured what happens to an answer like that. Ours was chosen zero times in more than three and a half thousand tries, while a vaguer answer sat beside it. It began to fire only when we took the vague answer away. Since then we ask the refusal first, as a question of its own.\n\n[[pause 2]]\n\n[thoughtful] One last thought, and it is the least tested thing I will say today.\n\nThe host remarks that a checker's instructions are a requirements document that runs. Carry that over to us. Each of our small machine tasks answers a question in the ontology's own words. The same claim. A part of. Supports.\n\nSo the instructions, together with the cases you have labelled, are what that word means in practice.\n\n[[pause 4]]\n\nIf that is right, your sittings do a second job. Where two careful people label the cases alike, the word is solid enough to build on. Where they label them differently, the word itself is contested. And that tells us something about the ontology, as well as about the machine.\n\nSo, part six in a line. Their single judge fits our checkable questions and our agreed ones. It does not fit the contested ones.\n\n---\n\nPart seven. Six questions to carry.\n\nThese are in the ledger as Y nine to Y fourteen. All six are open. None is urgent tonight.\n\nY nine. Should the first sitting on each pass be for reading and finding faults, with proof coming only when the wording has stopped changing?\n\n[[pause 3]]\n\nY ten. Should every verdict you give have room for a note in your own words? And should the passes that write get a sitting of their own: one item, its source beside it, and the first thing wrong?\n\n[[pause 3]]\n\nY eleven. Should two fifths of everything you label be sealed, kept out of my sight, and opened once for each test?\n\n[[pause 3]]\n\nY twelve. Should every machine checker be reported with two numbers, faults caught and sound cases passed, and never with one agreement figure?\n\n[[pause 3]]\n\nY thirteen. Should your own labels be enough for the first answer key, on the questions that are checkable or agreed? This one loosens something you set in August, when friends were to sit the same test before any bar was set. So it is yours to weigh.\n\n[[pause 3]]\n\nY fourteen. Would you want a short sitting every week? About twenty fresh items, half an hour. It is your time, so I only ask.\n\n[[pause 3]]\n\nIf you keep one sentence from all of this, keep this one. The first step of checking a machine is a person reading its work and writing down what is wrong. And that is the step we have not yet taken.\n\n[[pause 2]]\n\n[warmly] That is all. Tell me how the pace was.\n"}