{"path":"listening/2026-09-29-microtasks-in-broad-daylight.md","content":"# Our machine tasks in broad daylight: a listening walkthrough (2026-09-29)\n\n- **What it is**: the third listening piece of 2026-09-29, on the founder's request: *\"What does\n  \"hard text tasks\" mean exactly? Are they as well-defined as what we're talking about with the\n  keeper keys of sameness and with the splitting / the separability/decomposition floor (what\n  prompting/rules is needed for the splitting and how reliable/error-prone is it really? How hard\n  can it be if you look at the concrete task in broad daylight and see what variance there can be\n  in interpreting the instructions?) and the relevance judge etc? I want you to dive deep into this\n  and read up on what the corpus and codebase holds and then generate a new walkthrough about\n  this.\"* Delivered after the work it describes, on his *\"Yes, relabel them, then do the\n  walkthrough.\"*\n- **Covers**: what the literature's \"hard text tasks\" were and why their figure does not transfer;\n  the keeper's one key with three definitions; the splitter read part by part (six of 37 written\n  parts fail *if this part were false, would the claim be weaker?*); the relevance judge; what\n  makes a microtask sturdy, and the canon line admitted that evening; the split a re-store erased\n  and its restore; the four decisions waiting.\n- **Rendered**: the full `eleven_v4`, voice *Lucy*, stability 0.5, similarity 0.8, speed 1.0, with\n  `scripts/render_listening.py`. Eight requests, cut at the part boundaries (the `---` lines below);\n  13,531 characters, 13 minutes 8 seconds, 817 quota units (3,108 to 3,925 of 10,000), the same\n  rate of about 0.06 units per character as the day's other renders.\n- **Checked**: transcribed locally with the tiny Whisper model. All 42 paragraphs present, no tag\n  read aloud (seven spots differ only as the small model spells them, such as digits for numbers).\n- **Snapshot**: the decisions in part seven were open on this date. Do not retro-edit; mark them in\n  this header as they are ruled.\n- **Superseded or coarse, marked 2026-09-30**: part four's closing refinement (ask how likely the\n  claim is if the supporter is true and if false, and take the difference) is not needed under the\n  ruling of 2026-09-30, where relevance means the gain and the judge's current question asks it; it\n  returns only on the starting-value path (`relevance-and-the-two-factors.md` § 7, ledger Q35). Part\n  three's *\"a whole can't read stronger than its weakest part\"* is the full-weight form: a machine-cut\n  part is presumed half-needed (`support_semantics.REQUIREMENT_WEIGHT_PRESUMED_PART`), so a refuted\n  part caps its claim at neutral, not below, unless someone argues the claim needs that part.\n- **Overtaken, marked 2026-09-30**: part three's *\"the coverage check passed all six\"* was true of\n  the check as then written; rewritten at his question (*\"Should we rather not just fix the \"coverage\n  check\"?\"*) it asks of every part whether the claim would be weaker were it false and catches all six\n  (research doc § 3e). Of part seven's decisions: the fourth is half done (the splitter's polarity rule,\n  built with the check); the keeper's one definition per key is ledger T5, still open; the second and\n  third fold into ledger T7, replacing the seven old splits. A new question, whether background a claim\n  takes for granted is a part, is ledger T6.\n- **Audio**: a local copy sits in `.private/local-only/listening-audio/` (not in git). The text below\n  re-renders it.\n- **Method**: [how-to-write-for-listening.md](how-to-write-for-listening.md) · **Research**:\n  [microtasks-in-broad-daylight.md](../research/microtasks-in-broad-daylight.md),\n  [proving-a-pass-reliable.md](../research/proving-a-pass-reliable.md) § 9.\n\n---\n\n[warmly] Hi Fredrik. This one answers your question from this afternoon: what did \"hard text tasks\" actually mean, and are our own machine tasks as loosely defined as those, or better? It runs about twelve minutes, in seven parts. First, the hard text tasks themselves. Second, the keeper's keys in broad daylight. Third, the splitter, read part by part, which is the longest. Fourth, the relevance judge. Fifth, what makes a microtask sturdy, and the line you added to the principles this evening. Sixth, the split that went missing and came back. And last, the decisions waiting for you.\n\n---\n\nPart one. What \"hard text tasks\" meant.\n\n[thoughtful] This morning I quoted a sobering number: on hard text tasks, a method with a statistical guarantee saved only fourteen to twenty-eight per cent of the expert work. I used it as a prior for how fast our own review could shrink. [short pause] That was a loose transfer, and I want to own it, because the framing was mine.\n\nThe number comes from a paper this year by the statistician Emmanuel Candès and two colleagues, called Probably Approximately Correct Labels. Its text tasks were three labelling jobs from computational social science. Is this news article left, centre or right? Does this headline agree that global warming is a serious concern? Is this text misinformation? A language model, GPT-4o, labelled each item from a short instruction with no examples, and then said how sure it was.\n\nOn those tasks the guarantee saved about fourteen per cent of the expert labels for political bias, twenty-eight for climate stance, and eighteen for misinformation. Left on its own, the model was wrong on about thirty-eight, twenty-five and nineteen per cent of items.\n\n[serious] Here's why that doesn't transfer. Those labels are disputed among careful people. Whether a headline \"agrees\" depends on who reads it. The group that collected the labels measured agreement between people and the model on the climate task with a statistic called kappa, where zero is chance and one is perfect. It came out at zero point five seven, and they note it was the highest of their settings. When the right answer is itself a matter of who you ask, the machine's confidence can't separate its right answers from its wrong ones, and that separation is exactly what the saving depends on.\n\nOur tasks are a different kind. Each has a written definition, a named test and worked examples. So fourteen to twenty-eight per cent is a floor for contested labelling, and it says little about specified work.\n\n[warmly] The nearest published cousin of our sameness check is a task called natural language inference: does one sentence follow from another, contradict it, or neither? In the part of the Stanford collection of such pairs that was checked, five people labelled each one. Fifty-eight per cent of pairs were unanimous, and ninety-eight per cent had a majority of three or more. When another group asked a hundred people about the pairs where only three of five had agreed, the majority answer changed for roughly a fifth to a quarter of them. And models were nearly perfect on the pairs people agreed on, and close to chance on the ones they didn't. [short pause] So even a well-defined relation has a contested tail, and the machine's errors live there. That's the division of labour worth wanting: let the machine settle the clear cases, and send the tail to people.\n\n---\n\nPart two. The keeper's keys, in broad daylight.\n\nThe keeper is the pass that reads each pair of claims before any link is written, and answers with one of seven keys: the same claim, one contains the other in either direction, they oppose each other, they overlap, they're unrelated, or it can't tell.\n\nIn its own instructions it's well defined. Each key has a one-line definition, sameness is a consent test, and it writes its reason before its answer. [thoughtful] The variance I found isn't in the model. It's in us. One key means three different things in three places.\n\nIn the machine's instructions, \"overlaps\" is about evidence: establishing one claim would partly advance the other. On the page where people will score the keeper, \"overlaps\" reads \"they share part of a claim\", which is about wording. And the code that routes the answer sends \"overlaps\" and \"contains\" on as proposals that a claim is bundled, while the instructions define \"contains\" as one claim following from another. A hedged claim is contained by its confident version, but it isn't a bundle of it.\n\n[serious] So a person scoring the keeper would be grading a different question from the one the machine answered, and their disagreement would measure the distance between our definitions as much as the machine's errors.\n\n---\n\nPart three. The splitter, part by part. This is the longest part, because it's where the real errors are.\n\nThe splitter cuts one long claim into its separate parts. Its instructions carry two good tests with worked examples. The philosopher Bertrand Russell's: could a reader accept one part and deny another? And the philosopher Donald Davidson's: strip a modifier, and if the rest still stands, the modifier was its own part. Then a second, separate call rebuilds the claim from the parts before it sees the original, and reports anything dropped or added. That's the coverage check.\n\n[thoughtful] When you asked for the errors exactly, I read every written part against its claim: thirty-seven parts from six claims. Six of them fail one question: if this part were false, would the claim be weaker?\n\nTwo fail it clearly. [short pause] The first comes from a claim that the death records in places with legal assisted dying are accurate, rather than obscured by a rule on death certificates or by the destruction of the underlying records. One part says, flatly, that the underlying records are destroyed. But the claim denied that destruction obscures anything. If the records are in fact kept, the claim that the records are reliable is helped, not hurt.\n\nThe second comes from a claim that a minimum-wage rise should be judged by whether it costs any low-skilled worker a job, \"even when the great majority of low-wage workers who keep their jobs end up earning more as a result\". One part states as fact that most of them do earn more. The claim only conceded that, for the sake of the argument.\n\nFour more have the same shape, milder: a background fact the claim concedes, written as though the claim rested on it. A legal rule about what death certificates must list. And three parts saying the death penalty, as practised, has documented errors, biases and inconsistencies, with examples the claim never gave, such as race, class and geography.\n\n[serious] Here's why this matters. On the twenty-first you ruled that a written part caps its claim: a whole can't read stronger than its weakest part. So refuting a flipped part pulls its claim down. The strength of that claim runs backwards.\n\nAnd it answers the question you asked that day. In your words: \"We should expect good splits, and I have some trouble imagining wrong splits, what would that even look like?\" [gently] This is what it looks like. Every word came from the sentence. Nothing was dropped and nothing was added, so the coverage check passed all six. Only the polarity changed: something denied or conceded came out asserted. The second model, the one that checks whether each piece is really a part, flagged all six.\n\nOne correction of my own. This morning I wrote that this second model's figure, sixty-two per cent genuine parts, was mostly a disagreement about what a part is. Read part by part, its objections are mostly real. Sixty-two per cent is roughly the share of parts with no problem at all. The share that does harm is six in thirty-seven, and two of those are clear.\n\nThe rest are small: two parts that add a \"rather than\" the claim never stated, one that adds a verdict of its own on a research method, a university's name dropped from a study, and a couple of near-duplicates.\n\n---\n\nPart four. The relevance judge.\n\nThis one asks three questions in order. Does the supporting claim bear on the other at all? In which direction? And how much, on five named steps from marginal up to settling it? [thoughtful] The first two are crisp. The third is a graded judgement, the one place where two careful readers might differ by a step.\n\nThe trials show the instructions doing the work. With a bare number from zero to one, the first trial answered one half on eight of twelve links, which is a shrug. With the five named steps, the second trial answered all twelve with confidence, and filed none of them against the direction the link claimed. A refinement is already on file: ask how likely the claim is if the supporter is true, how likely if it's false, and take the difference.\n\n---\n\nPart five. What makes a microtask sturdy.\n\n[warmly] Your phrase from this afternoon, \"sturdy microtasks\", fits exactly. The way I've written it down: a sturdy microtask is small enough to have a clear answer, shaped so its mistakes can be noticed, and routed to whichever of machine or person does it more reliably.\n\nThe fixes I've proposed, none ruled yet, are all of that kind. Give each key one definition, the same in the instructions, on the review page and in the routing. Tell the splitter to rewrite in the claim's own words only, and to keep a denied or conceded clause denied or conceded. Ask of every part, before it's written: if this were false, would the claim be weaker? Back the model's check with a plain rule the machine can't argue with: a part may not name a number, a date or a name the claim didn't. Pair the coverage check with the parts check, because each catches what the other misses. And ask each question twice, to see whether the machine gives the same answer.\n\nThis evening you added a line to the principles. In your words: \"Add it as written (except remove the instructions examples parenthesis, instead just let it say generally that it's about the prompting rules for the machine passes since we want as deterministic/idempotent responses as possible)\". [short pause] So it now reads: an error rate belongs to the task as written before it belongs to the model. Much of the error measured in our machine work has come from the prompting rules we give the passes, so a bad rate is first a design problem, and each microtask made sturdier moves out the point where people must check.\n\n---\n\nPart six. The split that went missing, and came back.\n\nWhile counting, I found one split missing. On the twenty-first the splitter wrote seven. By today there were six. The missing one belonged to the goal claim of your friend's AI-governance map. When that map was stored again on the twenty-second, with twenty-four of its claims corrected, the storing step recreated its claims from scratch, and the split went with them: the proposal and its seven parts. Nothing reported it, and the documents went on saying seven for a week.\n\n[thoughtful] You asked me to restore it, and it took three tries. The first came back with eight parts, one of which added a thought of its own: that what counts as \"sufficient\" coordination is itself a further judgement. The rule holds a split like that for a person, so it was held. The second came back with every field filled with the single word \"test\", and it was stored as though the claim were one indivisible thing. The third was clean: eight parts, nothing dropped or added, every part passing the question from part three. That one is written, and it's live.\n\n[intrigued] And the two usable answers read the claim's phrase \"broad human control\" two different ways: the first as control wide in scope, the third as control spread across many people. Both are defensible. That's instruction variance in miniature, the exact thing this evening's principle is about.\n\nThree guards came out of it, each tested against the failure it answers. Storing a map again now keeps the split of any claim whose words haven't changed, along with any review marks on its parts. Every split proposal now records the model that actually answered. All eleven from the twenty-first said Gemini, while the log shows Claude Sonnet five answered every one, and on your word I relabelled the ten still stored this evening. And an answer that's just one short word in every field is now refused and retried, for every judge that runs on the Claude backend, not only the splitter.\n\n---\n\nPart seven. What's waiting for you.\n\n[gently] Four questions to carry.\n\nFirst. This morning's rule about building machine work on machine work had two clauses you never saw. Until a machine step's error rate has been measured, which today means every step, should a person look at its output before more machine work builds on it? And should the count of steps start at extraction, so that splitting an extracted claim is already the second step?\n\nSecond. The six parts that fail the question: retract them now, or leave them for a review sitting that will measure the splitter properly?\n\nThird. The four held proposals, on the law of occupation, Gaza's sovereignty, assisted-dying safeguards and wages set by the market. Keep or drop each?\n\nFourth. The fixes: build the polarity rule for the splitter, and one definition per key for the keeper, before anyone scores either?\n\n[warmly] In one sentence: our machine tasks are far better specified than the ones in that paper, most of the variance left in them is ours to remove, and this evening's restore showed both the errors and the guards in miniature.\n"}