{"path":"research/listening-for-the-calculus-and-sturdy-microtasks.md","content":"# Listening for the calculus and for sturdy microtasks\n\n- **What it is**: a tight list of podcast episodes and recorded talks for the two questions the founder\n  is still working through, asked for on 2026-10-05 while travelling: *\"What are some podcasts that touch\n  on the forefront of the questions about epistemic calculus that I'm still grappling with? (And perhaps\n  specifically how the microtasks can be made more sturdy/robust in order to lower the error rates etc,\n  we've thought about this in the recent past)\"*; then *\"Keep a rather tight net, I want to focus on\n  this/these questions about the microtasks that I'm asking about and priors/posteriors etc perhaps\"*;\n  then *\"Broaden the search to YouTube media as well.\"*\n- **Method**: searched online on 2026-10-05; nothing here is from memory. Every item's page was opened.\n  The last column of each table says what was read: a transcript, an abstract, a handout, or only the\n  listing. Lengths and dates are as read from the page, the audio file, or the video's own record.\n  **None of these was listened to.** What an item says is taken from what its page shows, and the entry\n  says so where that is thin.\n- **The earlier list**: [the-irit-harvest.md](the-irit-harvest.md) §§ 5–6 holds eighteen verified items\n  from 2026-08-31, matched to the decisions pending then. § 5 below reads it against today's two\n  questions: two of the eighteen bear on them, in part.\n- **Status**: a reading aid. Nothing here is ruled, and § 6 is Claude's reading of what the list shows.\n\n---\n\n## 0. The five to hear first\n\n| Order | Item | Length | Why first |\n|---|---|---|---|\n| 1 | **Mindscape 315, Branden Fitelson** on the logic and use of probability | not shown | He argues that the strength of an argument needs two numbers, and one of them is relevance. The nearest outside statement of the link rule ruled on 2026-09-28 |\n| 2 | **Richard Pettigrew, John Locke Lecture 2** (Oxford, May 2025) | 1 h 04 min | Where a starting value comes from, and what happens to it when the options are cut differently. Free audio |\n| 3 | **Christian Dahlman (Lund)**, *Prior Probability and the Presumption of Innocence* | 47 min | The case for a starting value fixed by rule, the same for everybody. The opposite answer to the one proposed here |\n| 4 | **Linear Digressions**, *LLM as a Judge* (28 September 2026) | not shown | One week old. What it means for a machine to agree with a person, and what counts as correct where people disagree |\n| 5 | **Lenny's Podcast, Hamel Husain and Shreya Shankar**, from 16:51 | 1 h 46 min in all | The review sitting as industry practice: one trusted person, yes-or-no answers, the judge checked against that person's labels |\n\n---\n\n## 1. The questions this list is cut to\n\n**The calculus**, each with its home in the corpus:\n\n- **C1. What a claim is worth before anyone examines it.** Proposed: the starting value is a claim too,\n  taken from the claim's kind. Open: whether some starts may be fixed constants\n  ([starting-values-and-the-problem-of-the-prior.md](starting-values-and-the-problem-of-the-prior.md)\n  § 9, question 1).\n- **C2. What the middle means.** His objection of 2026-09-29: the middle must not suggest some truth\n  where no good reason has been found\n  ([relevance-and-the-two-factors.md](relevance-and-the-two-factors.md) § 6–7).\n- **C3. What a failed reason does.** Ruled for now: it drops out, and failure counts only through a\n  contribution that says why. Open: whether a source's own *because* counts as such a contribution\n  (ledger Q34–Q36).\n- **C4. How several reasons add up**, and how one finding written twice is counted once (accrual, still\n  open in the same § 7).\n- **C5. Conclusions that hold under every reasonable start, against ones that flip.** Worldviews as\n  sets of starts (starting-values § 6.6, § 9 question 5).\n- **C6. A claim of several kinds**: pool the kinds, never pick one (§ 10.3).\n- **C7. Estimates by groups of people**: may they set a start, or only raise an alarm (§ 9 question 6).\n- **C8. The report is not the event**: who asserted a claim is evidence, never its start (§ 6.2).\n\n**Sturdy microtasks**:\n\n- **M1. The task as written**: a question small enough to have a clear answer, asked per item, in a form\n  a machine answers the same twice\n  ([microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 4).\n- **M2. Proof from a sample a person labelled blind**, and that sample as the thing that outlasts a model\n  ([proving-a-pass-reliable.md](proving-a-pass-reliable.md) § 2;\n  [pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) § 5e).\n- **M3. Declining**: a bound on what the machine answers, reached by handing the rest to a person\n  (proving § 5).\n- **M4. Steadiness before accuracy**: the same answer when asked again (pass-quality § 2d).\n- **M5. A checker from another maker**, and how often two models fail together (proving § 4).\n- **M6. Where careful people disagree** there is no answer to be wrong about (proving § 3).\n- **M7. Depth**: error compounds along a chain of machine steps, and taking a claim apart is itself a way\n  to hide an error (canon; the threat model's entry on obfuscated arguments).\n\n---\n\n## 2. The calculus\n\n| Item | Where, when, how long | Bears on | What to listen for | How far it was checked |\n|---|---|---|---|---|\n| **Branden Fitelson**, *The Logic and Use of Probability*. Sean Carroll's Mindscape 315 | [preposterousuniverse.com](https://preposterousuniverse.com/podcast/2025/05/19/315-branden-fitelson-on-the-logic-and-use-of-probability/) · 19 May 2025 | C1, C3, C4 | Around 0:54–0:59 he sets out a **two-dimensional account of argument strength**: the conclusion must be probable given the premise, and the premise must be relevant, with relevance measured by the likelihood ratio (how much likelier the evidence is if the claim holds than if it fails). A premise that is irrelevant is no reason at all, however probable the conclusion. Around 0:29–0:39: that ratio can be measured without knowing anybody's starting value, which is why papers report it and leave the start to the reader. Around 0:43: no single probability function fits every argument, but each argument has a suitable one. Around 0:41: Popper was right that a refutation is the strongest evidence against, and wrong that it is the only kind of evidence | Transcript read from 0:29 to 0:59 |\n| **Richard Pettigrew**, John Locke Lectures 2025, *From a Point of View*. Lecture 2, listed as *On Uncertainty: priors, credal hinges, and epistemic risk*; its handout is titled *The initial point of view*, 7 May 2025 | [Oxford's lecture page](https://www.philosophy.ox.ac.uk/john-locke-lectures) · [audio](http://media.philosophy.ox.ac.uk/locke/TT25_lec2.mp3), 1 h 04 min · [handout](https://www.philosophy.ox.ac.uk/sitefiles/lecture-2-handout-rp-tt25.pdf) | C1, C5, and the splitter | **A prior is a credence we posit in order to judge later ones**, never what somebody believed at the start of life. Which priors are allowed depends on a person's attitude to risk in belief (William James: seek truth, or shun error): the cautious end yields the even spread, the bold end permits lopsided starts. Then his own objections: why not roll a die among the permitted ones, and why may one not flip between them. Then **what to do when the set of possibilities changes**, by a new option or by a finer cut: old proportions cannot simply be kept, so choose a start over the new set and update again on all the evidence so far. Last: people in one conversation hold starts within a narrow range, and we speak as if that range were what the evidence itself says, which makes it a range and never a point | Handout read in full (six pages); audio file found and measured, not heard. Lecture 3, *Changing our point of view: evidence, inquiry, and doxastic crises* ([audio](http://media.philosophy.ox.ac.uk/locke/TT25_lec3.mp3), 1 h 03 min), continues it; its handout was not opened |\n| **Alan Hájek**, *Puzzles and paradoxes in probability and expected value*. 80,000 Hours 139 | [80000hours.org](https://80000hours.org/podcast/episodes/alan-hajek-probability-expected-value/) · 28 October 2022, reissued 25 February 2025 · 3 h 38 min | C1, C2 | From 0:37:17 to 1:21:24, about 45 minutes: theories of probability, everyday Bayesianism, frequentism and its reference classes, ranges of probabilities. His worry in one line: a prior chosen with no evidence behind it still leaves its mark on every later answer, which is anchoring by another name. He wrote the 2007 paper the starting-values document leans on | Chapter list and the transcript's sections on Bayesianism and frequentism read |\n| **Christian Dahlman** (Lund University), *Prior Probability and the Presumption of Innocence*. Isaac Newton Institute, Cambridge | [YouTube](https://www.youtube.com/watch?v=G5-F0nwDXM8) · given 29 September 2016, uploaded to the institute's channel 15 December 2025 · 47 min | C1, C8 | How a court should set the probability of guilt before any evidence. If the start is set on evidence alone, **being picked for prosecution is itself evidence of guilt**, which the presumption of innocence forbids. One over the number of people who could have done it varies from trial to trial, which treats defendants unequally. His proposal: a standard number, for example one in a hundred, in every trial: the start as **an institutional fact** | Abstract read ([talks.cam](https://talks.cam.ac.uk/talk/index/67706/)); video found on the talk's own page and measured, not watched. The corpus cites his 2021 paper with Kolflaath; this is the earlier and plainer statement. Also listed: Excited Utterance 77, 23 September 2019, on whether a base rate alone can prove a case |\n| **Ulrike Hahn** (Birkbeck), two talks. *Bayesian Argumentation in the Real World*, Isaac Newton Institute; and *Argument Quality in Real World Argumentation*, in the series *Different Perspectives on Argumentation* | [YouTube, 2016 talk](https://www.youtube.com/watch?v=KkMADmufvTY), 48 min · [YouTube, 2021 talk](https://www.youtube.com/watch?v=d1yX7hJnvVo), 1 h 27 min, published 27 November 2021 on the channel *Katedra logiky a metodológie vied* | C3, C4 | The programme that treats an argument's strength as a probability and the classical fallacies as arguments that are weak or strong by content. Its **argument from ignorance** is the model for the ruled third contribution (a thorough search that found nothing). Her keynote abstracts of 2024 and 2026 claim the approach handles how several arguments combine, and can express one argument defeating another | Abstracts read; neither talk watched. The 2026 keynote, *Bayesian Argumentation from Single Arguments to Large-Scale Debate*, was found as an abstract only |\n| **Andrew Gelman, Aki Vehtari and Richard McElreath**, *Bayesian Workflow*. Learning Bayesian Statistics 164 | [learnbayesstats.com](https://learnbayesstats.com/episode/bayesian-workflow-reverse-bayes-hierarchical-pooling-gelman-vehtari-mcelreath) · listed 31 August 2026, recorded 16 June 2026 · 1 h 35 min | C5, C6 | 0:39:03, **reverse Bayes**: fix the conclusion and ask which range of starts would justify it, then ask whether those starts are believable. That is the robust-or-prior-dependent reading run backwards. 0:52:24, when pooling stops being harmless: with few groups, how much to pool is itself poorly known. 1:02:05, always compare the pooled estimates with the unpooled ones, because where they differ the pooling is doing the work. 1:23:11, why report three scenarios and not one number | Page, chapter list and summary read |\n| **Richard McElreath**, Statistical Rethinking 2023, lecture 12, *Multilevel Models* | [YouTube](https://www.youtube.com/watch?v=iwVqiiXYeC4) · 8 February 2023 · 1 h 22 min | C6 | The plainest hour on pooling: 13:50 partial pooling, 32:02 learning from the data how much to pool. Background for recommendation 6 of the starting-values document | Outline read; a visual lecture, better watched than heard |\n| **Ezra Karger**, *What superforecasters and experts think about existential risks*. 80,000 Hours 200 | [80000hours.org](https://80000hours.org/podcast/episodes/ezra-karger-forecasting-existential-risks/) · [YouTube](https://www.youtube.com/watch?v=1WQzQ5hhz7k) · 4 September 2024 · 2 h 49 min | C5, the convergence wager | A measured case of a conclusion that depends on the start. Two groups of careful people stood a factor of 250 apart, spent weeks on each other's arguments, and stayed there. The best near-term question found would close about one point of a gap of about twenty-five. He names four explanations, the last being differences of worldview that reach beyond the topic | Transcript read in the sections on the four hypotheses and the crux questions |\n| **Ben Chugg and Vaden Masrani**, the critical-rationalist case against putting numbers on claims. Increments 69, *Contra Scott Alexander on Probability*; and the 2026 exchange with a Bayesian statistician: Learning Bayesian Statistics 160 and Increments 105 | [incrementspodcast.com/69](https://www.incrementspodcast.com/69), 20 June 2024, 1 h 45 min · [LBS 160 on YouTube](https://www.youtube.com/watch?v=St9EnAnBlEA), 29 June 2026, 1 h 40 min · [incrementspodcast.com/105](https://www.incrementspodcast.com/105), 17 July 2026, 1 h 19 min | C1, C2 | The hostile witness. Episode 69 argues that one figure must not mean both *equally likely* and *no idea*, which is his own objection to the middle. Their line throughout: a statistician's prior is fine because data stands behind it, and a number with no data behind it is not. Where Deliberus stands between the two camps is read in § 6, point 6 | Show notes of all three read; none heard. Increments 74 (1 October 2024) has David Deutsch on belief and probability |\n| **Scott Alexander on the Rootclaim debate**, read aloud. Astral Codex Ten Podcast | [libsyn](https://sscpodcast.libsyn.com/practically-a-book-review-rootclaim-100000-lab-leak-debate) · 12 April 2024 · length not shown | C4, the risk of false precision | What explicit numbers do when careful people set them for every piece of evidence: fifteen hours of debate, and answers many orders of magnitude apart. Rootclaim's own account afterwards says an extreme number is the numerical form of a straw man, and that an estimate must include the chance that the estimate is wrong | Episode listing opened; [Rootclaim's account](https://blog.rootclaim.com/rootclaims-covid-19-origins-debate-results/) read. The lesson is already drawn in starting-values § 3.5; this is the long form |\n| **Yoshua Bengio**, *Disentangling Agency & Predictive Power Without Solving ELK*. FAR.AI Alignment Workshop | [YouTube](https://www.youtube.com/watch?v=ZndiBDmss-w) · talk dated 30 November 2025 on [the event page](https://www.far.ai/event-recordings/yoshua-bengio-disentangling-agency-predictive-power-without-solving-elk), video published 17 February 2026 · 31 min | C8, C1 | His training data is rewritten so that *X is true* and *someone wrote X* are different sentences, chosen by the source. That is the report kept apart from the event, as a rule for a machine. On the start he says only that somebody supplies a fairly flat one, so C1 is left open there too | Transcript read. The longer version is his Karp lecture at the Simons Institute ([YouTube](https://www.youtube.com/watch?v=g0ljOgBo5uY), 17 April 2025, 1 h 14 min) |\n| **Anna Dreber** (Stockholm School of Economics), *Replication Markets*; and **Fiona Fidler**, the repliCATS project | [Behavioural Science Uncovered](https://bsuncovered.podbean.com/e/replication-markets-with-anna-dreber/), 12 May 2020, 28 min · [YouTube, RIOT Science Club](https://www.youtube.com/watch?v=KhDKzEsPouI), 27 July 2020, 1 h 01 min | C7, and the measured rates of starting-values § 2.6 | Where the measured starting rates for one kind of claim, a published finding, come from. Fidler's method: each person judges alone, the group discusses, each judges again in private | Listings read; neither heard. Both predate the 2026 results the corpus cites |\n\n---\n\n## 3. Sturdy microtasks\n\n| Item | Where, when, how long | Bears on | What to listen for | How far it was checked |\n|---|---|---|---|---|\n| **Linear Digressions**, *LLM as a Judge* | [lineardigressions.substack.com](https://lineardigressions.substack.com/p/llm-as-a-judge) · 28 September 2026 · length not shown | M2, M4, M6 | Raw agreement between a machine and a person overstates: 75 in 100 can be barely better than chance once chance is taken out. Where people disagree, *correct* becomes *inside the range of reasonable human answers*. A judge that answers consistently is not thereby accurate. Practical advice: small models with explicit instructions on tasks that can be taken apart, shuffle the order of options, repeat runs | Page read. It names a 2026 paper, *Reliability without Validity* (Norman, Rivera and Hughes), which was not opened |\n| **Hamel Husain and Shreya Shankar**, *Why AI evals are the hottest new skill for product builders*. Lenny's Podcast | [YouTube](https://www.youtube.com/watch?v=BsWxPI9UM4c) · 25 September 2025 · 1 h 46 min | M1, M2 | From 16:51. About 0:25: **one trusted person labels**, never a committee. About 0:30: read about a hundred real cases and stop when no new kind of error turns up. About 0:53: **yes or no, never a scale of one to five**. About 0:57–1:01: check the machine judge against that person's labels with a table of both kinds of error, never one agreement figure. About 1:04: the criteria change as the person reads more outputs | Episode page read; a transcript was checked on these five points. Written for product managers, so the opening is slow. **Since read in full, with the authors' two written guides: [the-evals-loop-and-our-machine-passes.md](the-evals-loop-and-our-machine-passes.md)** |\n| **Anastasios Angelopoulos**: *A Tutorial on Conformal Prediction*; *Prediction-Powered Inference*; and a conversation on AI reliability | [YouTube tutorial](https://www.youtube.com/watch?v=nql000Lu_iE), 2 July 2021, 38 min · [Simons Institute talk](https://simons.berkeley.edu/talks/anastasios-angelopoulos-2025-05-01), 1 May 2025, a 45-minute slot · [Variance, episode 2](https://hashcollision.substack.com/p/variance-episode-2-ai-reliability), 20 August 2025, 57 min | M2, M3 | The mathematics behind proving § 2 and § 5, from its author. The tutorial: a guarantee that holds for any model, bought with a labelled sample, where the hard cases get the wide answer. The Simons talk: a few human labels plus many machine labels give a valid estimate narrower than the human labels alone. The podcast is the conversational version | Tutorial transcript read in part; abstract and listing read for the other two. The tutorial is visual |\n| **Evan Miller** (Anthropic), *Evaluating AI Models: A Seminar*. Klaviyo Data Science Podcast 56 | [klaviyo.tech](https://klaviyo.tech/klaviyo-data-science-podcast-ep-56-evaluating-ai-models-a-seminar-feat-evan-miller-746f306136bb) · 14 February 2025 · length not shown | M2 | On his paper *Adding Error Bars to Evals*: treat the test questions as a sample from a larger set, report the uncertainty, compare two models on the same questions, and work out how many questions a comparison needs. It bears on what a small labelled set can and cannot show about two models | Opening of the page read; the paper not opened |\n| **Jungwon Byun and Andreas Stuhlmüller** (Elicit), *Supervise the Process of AI Research*. Latent Space | [latent.space](https://www.latent.space/p/elicit) · 11 April 2024 · length not shown | M1, M7 | About 0:08: supervise the process, never only the outcome, because small steps are easier to check and show where something broke. They tested taking tasks apart with people in 2018–19. About 0:32: earlier attempts at a product built on trees of sub-questions failed, the first as overwhelming and the second as too general, before a narrower form worked. About 0:38: users spot-check through the highlighted passage of the source | Episode page and transcript read in part. His 2011 strategy, told by people who made a company of it |\n| **Geoffrey Irving**, *Can We Get Asymptotic Safety Guarantees Based On Scalable Oversight?* Simons Institute | [simons.berkeley.edu](https://simons.berkeley.edu/talks/geoffrey-irving-uk-ai-safety-institute-2025-04-15) · 15 April 2025 · video linked from the page | M7, and the calculus | Debate that takes a question apart can be beaten by an arguer who cuts an easy question into pieces too hard to check. The 2025 answer makes the other side put probabilities on the pieces, and it works only if the argument still holds when those probabilities are slightly off (the second half is from the abstract of the paper that followed, arXiv 2506.13609) | Talk abstract read. Earlier audio: AXRP 16 with Irving ([axrp.net](https://axrp.net/episode/2022/07/01/episode-16-preparing-for-debate-ai-geoffrey-irving.html), 1 July 2022) and AXRP 6 with Beth Barnes (8 April 2021); listings only |\n| **Omar Khattab**, *On Engineering AI Systems that Endure The Bitter Lesson*. AI Engineer | [YouTube](https://www.youtube.com/watch?v=qdmxApz3EJI) · 6 August 2025 · 19 min | M1 | His framework states what a step takes and returns apart from the wording of its instructions, and tunes the wording against a measure | Title and length only; not watched. Lowest confidence on this list |\n\n---\n\n## 4. Where the two questions meet\n\n- **Pettigrew's changed possibilities are the splitter's situation.** When a claim is cut into parts, or\n  a rival is added beside it, the set of options has changed, and a start that was an even spread over\n  the old set is not one over the new set.\n- **Irving's condition is a claim about arithmetic.** His protocol is safe only for arguments that\n  survive small errors in the probabilities of their parts. The same property is what a strength\n  calculus needs if its inputs are machine readings with a measured error rate.\n- **Elicit's failed trees are a warning about the page, never about the method.** Taking work apart\n  made their checking easier, and a product that showed people the whole tree overwhelmed them.\n\n---\n\n## 5. The earlier harvest, read against these two questions\n\n[the-irit-harvest.md](the-irit-harvest.md) §§ 5–6 lists eighteen items, and names two more as already\nheard (Ruth Chang; Sara Walker).\n\n- **Bear on today's questions, in part**: Excited Utterance 161 with Ronald Allen (whether a case is\n  weighed element by element or story against story: C4), and Mindscape 233 with Hugo Mercier (why\n  judging an argument is the reliable half of reasoning: who should check).\n- **Serve other decisions**: Christopher Manning (two episodes), Nicolaus Tideman, Francesca Toni (two\n  talks), Dan Kahan (two episodes), Chris Bail (two), Audrey Tang, Colin Megill, Tim van Gelder (two\n  talks), Hélène Landemore (two), James Fishkin.\n- **Its own note stands**: for Amgoud, Denœux, Jøsang and Potyka it found conference talks only. This\n  search looked again with video included, and § 7 says what it found.\n\n---\n\n## 6. What the list shows, as a reading (Claude's; unruled)\n\n1. **Fitelson's two numbers are not the link rule's two numbers.** His are the conclusion's probability\n   given the premise, and the premise's relevance. The rule ruled on 2026-09-28 multiplies the\n   supporting claim's own standing by the link's relevance. Both refuse to let one number do two jobs,\n   and both give relevance its own place. Worth hearing for whether his first number is something the\n   graph already has (the claim's computed strength) or something it lacks.\n2. **Dahlman answers question 1 of starting-values § 9 the other way, for a reason the proposal shares.**\n   He wants a constant fixed by rule so that every defendant starts alike. A start taken from a claim's\n   kind also treats every claim of that kind alike, whoever brought it. Where they part is whether the\n   number may be argued over. His point about selection is § 6.2's: being put forward is evidence.\n3. **Pettigrew's finer cut is the carving problem the weakest-part rule already meets on one side.**\n   Under that rule (the padding ruling of 2026-08-31) a claim cut into three parts or twelve scores the\n   same, because its parts cap it and are not summed. Starts by kind would need the same care on the\n   other side: a start must not change because a claim was cut.\n4. **Husain and Shankar predict what the first sitting will do.** Reading real cases changes the\n   criteria, so the instructions will change after it, and a changed instruction starts the proof again\n   (proving § 4). Their yes-or-no rule agrees with what was measured here on 2026-10-05: no model\n   answered the five-step question of degree the same way twice. *Read in full that evening, on his\n   request: [the-evals-loop-and-our-machine-passes.md](the-evals-loop-and-our-machine-passes.md). The loop starts one step earlier than this reading saw: a person reads\n   real output and writes what is wrong, before anything is labelled or proven.*\n5. **Linear Digressions gives the contested kind its measure.** Where careful people disagree, the\n   machine can be asked only to fall inside their range, and the range has to be measured by a second\n   person sitting the same items.\n6. **The sharpest disagreement on C1 is between people who both like Bayes.** The statisticians and the\n   critical rationalists agree a number needs data behind it. They part on whether a start may stand in\n   for data not yet seen. The proposal here sides with neither fully: no number for an unexamined claim,\n   and a start only where a measured rate for its kind exists.\n\n---\n\n## 7. Looked for and not found as a recording; papers seen in passing\n\n**No recording found on 2026-10-05**: Leila Amgoud and Nico Potyka on gradual argumentation (Potyka's\n2020 tutorial has slides only); Henry Prakken on how arguments accrue; Audun Jøsang on subjective logic\n(tutorial slides from 2016 and 2022); Barbara Plank's keynotes on disagreement between annotators (ACL\n2024 and others are listed, no public video found); Hanna Wallach on evaluation as measurement (a 2025\nposition paper, no talk found); the oral presentation of *Trust or Escalate* (its conference page shows\nno video); semantic entropy; Michael Bernstein's 2010 crowd-work pattern of find, fix, verify; Tongshuang\nWu's replication of crowd pipelines with language models; conformal guarantees on generated claims.\n\n**Papers seen in the search results and not opened**, kept here as leads only:\n\n- *Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels* (arXiv\n  2605.29800). Bears on M5.\n- *The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation* (arXiv 2606.13685). M4.\n- *Localize-Then-Decide Guarantees for LLM Judgments* (arXiv 2608.25824). M3.\n- *When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for\n  Structured Generation* (arXiv 2606.29054). M3.\n- *Avoiding Obfuscation with Prover-Estimator Debate* (Brown-Cohen, Irving and Piliouras, arXiv\n  2506.13609; abstract read), *How to Avoid Debate: Scalable AI Safety via Doubly-Efficient Interactive\n  Proofs* (arXiv 2607.03561) and *How to Have a Sensitive Debate: An Instance-Optimal Protocol for AI\n  Debate* (arXiv 2610.02557). M7.\n- Potyka and colleagues, *Balancing Open-Mindedness and Conservativeness in Quantitative Bipolar\n  Argumentation* (KR 2024): how far a strength may move from its starting value. C1.\n- *LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs*\n  (Wu and colleagues, arXiv 2307.10168). M1.\n\n---\n\n## Cross-references\n\n[the-irit-harvest.md](the-irit-harvest.md) §§ 5–6 (the earlier list) ·\n[starting-values-and-the-problem-of-the-prior.md](starting-values-and-the-problem-of-the-prior.md)\n(§ 2.7 law, § 3.5 Rootclaim, § 9 the open questions, § 10.3 pooling) ·\n[relevance-and-the-two-factors.md](relevance-and-the-two-factors.md) §§ 6–7 ·\n[proving-a-pass-reliable.md](proving-a-pass-reliable.md) §§ 2–5, § 9 ·\n[microtasks-in-broad-daylight.md](microtasks-in-broad-daylight.md) § 4 ·\n[pass-quality-and-the-choice-of-model.md](pass-quality-and-the-choice-of-model.md) § 2, § 5e ·\n[bengio-safety-from-honesty-and-deliberus.md](bengio-safety-from-honesty-and-deliberus.md) ·\n[session26-open-rulings.md](session26-open-rulings.md) (Q34–Q36, T1–T7, Arc Y).\n"}