{"path":"research/saying-what-kind-a-claim-is.md","content":"# Saying what kind a claim is: the pass that sets up a claim's prior\n\n*2026-10-24, late. An all-angles pass on his question. Ledger Arc BD (BD3 ruled, the rest open). Tests:\n[saying-what-kind-a-claim-is/cases.py](saying-what-kind-a-claim-is/cases.py); research report, verbatim:\n[saying-what-kind-a-claim-is/research-report-2026-10-24.md](saying-what-kind-a-claim-is/research-report-2026-10-24.md).\nBuilds on [priors-as-standing-work.md](priors-as-standing-work.md) §§ 9a, 10a.*\n\n## 0. The answer first\n\n**How rates from nested reference classes combine is solved and is arithmetic; which reference classes a claim belongs to is a\nclassification task a model can do well only in one shape, and nobody has published how often people agree on\nit.** Five findings set the design:\n\n1. **A narrower reference class predicts better when the claim truly belongs to it.** On the replication study of 2015,\n   each journal-and-subfield cell predicted from the other cell of its subfield missed by 0.06 on average,\n   against 0.17 from all the other cells pooled (§ 4).\n2. **A wrong narrower reference class costs as much as a strong reason.** An immuno-oncology drug filed under *solid\n   tumours* moves its prior by 0.74 log-odds, and in a constructed case with 100 counted cases per kind the\n   error reaches 1.5 log-odds, more than one fully relevant reason at 0.75 (1.1).\n3. **So the pass names the narrower reference class only when it is right often enough**, roughly 60 to 90 percent\n   depending on how harshly a large miss is scored, and otherwise leaves the claim at the broader reference class. Ruled\n   (BD3).\n4. **A model classifies well only from a shortlist, judging each candidate and its parent one at a time**:\n   94.3 percent on a 2,778-leaf taxonomy, against 58.6 percent picking in one pass and 50 percent walking the\n   tree (COLING-Industry 2025). Accuracy falls toward the leaves, and a \"none of these\" option offered beside\n   the answers lowers accuracy (30 to 50 percent when it was the right answer).\n5. **No model should state a rate.** Nothing found shows that rates a model states are accurate; one study found\n   most models scoring worse than the plain base rate. The pass names kinds; code reads counted cases.\n\n## 1. His words\n\n> *\"But for priors - how do we, in practice, assert what kind a claim is? Is that done by code or a model and how\n> can we rely on it enough? Is this somehow a solved problem that I've just not yet understood? How exactly is it\n> done best? Read up in the corpus etc.\"*\n\n> *\"Agreed about all of this, but we need to care for the reliability and sharpness and method/prompting of the\n> machine pass that gives its take on reference-class membership for prior setting.\"*\n\nRuled the same night: a claim's prior comes from its reference class by pooled nesting (BC1); a consideration counts once\n(BC8), so a kind is never described by a clash the graph holds as a link. On the break-even rule of § 3:\n*\"Sounds reasonable, ruled. We can always come back to this later.\"*\n\nThe open half, in one sentence: **how the pass that says which reference classes a claim belongs to should be built, and how\nfar it can be trusted.**\n\n## 2. What the corpus already held\n\n| Angle | Home | State |\n|---|---|---|\n| The reference class problem (Venn, Reichenbach, Kyburg, Hájek) | `starting-values-and-the-problem-of-the-prior.md` §§ 2.2, 10.2 | covered |\n| Pooling nested reference classes; picking a single class is beaten by shrinkage | same, § 10.3 (Paul 2026, Gelman) | covered |\n| Outside view, Flyvbjerg's databases, superforecasters | same, §§ 3.1, 10.2 | covered |\n| Measured rates for a dozen kinds | same, § 10.1 | covered |\n| Learned models relocate the choice; judge by outcomes | same, § 10.4 | covered |\n| A model proposes a kind, never sets a prior | same, § 7 rec 8; BC3 | proposed |\n| A kind never defined by a linked clash | `priors-as-standing-work.md` § 10a; BC8 | ruled |\n| How often people agree on a class | none | **uncovered** |\n| How well models assign items to taxonomy nodes, and in what shape | none | **uncovered** |\n| Whether model-stated base rates are accurate | partly (Schoenegger) | **uncovered** |\n| Real tables counted at two nested levels | none | **uncovered** |\n\n## 3. Theory\n\nPooling: a narrow reference class with *s* successes in *n* cases, under a broad kind with rate *p*, gives the prior\n(s + k·p) / (n + k), where *k* says how much the broad kind counts. Two consequences follow before any data.\n\n- **The cost of a wrong narrow reference class grows with how much data the narrow kinds hold**, since pooled rates move\n  toward their raw rates as *n* grows: in the constructed case (broad 10 percent, narrow kinds truly 5 and 25\n  percent, k = 20) the error is 0.04 at n = 5 and 0.20 at n = 1,000.\n- **Staying at the broad kind also costs**, the distance from the right narrow rate to the broad one. So\n  naming the narrow reference class pays only when the pass is right more often than a break-even accuracy:\n  60 to 72 percent if errors in log-odds are scored linearly, 84 to 92 percent if squared. This is BD3.\n\n**This composes with AY6 (ruled: a one-case claim is read through the narrowest rule with a suitable count)**: AY6 asks that the narrow reference class hold enough counted cases; BD3 adds that the claim's membership in it be sure enough.\n\n*k* should not be chosen. It is estimated from how much the narrow kinds of one broad kind differ among\nthemselves (empirical Bayes); the tests below use k = 20 as a stated choice, not a fit.\n\n## 4. Tests on real tables\n\n**The replication study of 2015** (Open Science Collaboration, Table 1): four cells, journal by subfield.\nEach cell predicted from the other three:\n\n| Cell | Observed | All other cells | The other cell of its subfield | Pooled, k = 20 |\n|---|---|---|---|---|\n| JPSP social | 0.23 | 0.42 | 0.29 | 0.35 |\n| JEP:LMC cognitive | 0.48 | 0.31 | 0.53 | 0.41 |\n| Psych. Science social | 0.29 | 0.38 | 0.23 | 0.29 |\n| Psych. Science cognitive | 0.53 | 0.33 | 0.48 | 0.42 |\n| Mean absolute error | | 0.165 | **0.059** | 0.080 |\n\nThe narrower reference class (subfield) wins clearly when membership is right. Pooling at an arbitrary k = 20 sits\nbetween; with *k* estimated from the cells' spread it would sit nearer the narrow reference class, since the subfields\ndiffer a lot. Four cells is a small test.\n\n**Drug development, phase II to III** (BIO, Informa and QLS 2021): oncology 24.6 percent (n = 1,732); solid\ntumours 23.4 (1,261), haematologic 27.8 (449), immuno-oncology 40.2 (244). An immuno-oncology drug filed as\n*solid tumours* moves from 0.39 to 0.23, **−0.74 log-odds**; left at the parent, *oncology*, it moves to 0.25,\n−0.67. Here staying at the parent is barely better than the wrong sibling, because the right kind sits far\nfrom both: the break-even of § 3 depends on where the kinds sit. Immuno-oncology also overlaps both solid and\nhaematologic tumours, so it is a crossed kind, not a child of either.\n\n**Hong Kong roadworks** (Flyvbjerg, Hon and Fok 2016): 9 of 10 of 863 road projects worldwide overran, 7 of 10\nof 23 in Hong Kong (significant, p = 0.006). Pooled at k = 20: 0.79. A narrow reference class with real data pulls the\nprior a long way from the broad one, which is what pooling is for.\n\n**One loose name, two populations**: social-psychology findings replicated 14 of 55 in the 2015 study (papers\nof 2008) and 13 of 21 in Camerer et al. 2018 (Nature and Science, 2010 to 2015): 1.56 log-odds apart. A kind\nmust name its population, its end point and its period, or two counts that disagree will be read as one kind.\n\n## 5. What the research adds\n\n- **Agreement between people on which class to use: not measured anywhere found.** The nearest measurement,\n  Theising (2023), backtested about 300 ways of defining a peer group for 21,808 firms: the standard industry\n  taxonomy ranked last, and peers matched on a measured feature ranked first, about six times better on fit.\n  A taxonomy node is not automatically the best kind.\n- **Models classify well only from a shortlist, judging each candidate with its parent** (94.3 percent, against\n  58.6 one-pass and 50.0 tree-walking; human labellers 65 to 90 percent under time pressure, same study).\n  Accuracy falls toward the leaves (TaxoGlimpse, VLDB 2024) and with the number of classes offered.\n- **\"None fits\" beside the answers lowers accuracy**, which matches the canon line that a machine's no-answer\n  belongs in a gate, never on the menu.\n- **No evidence that model-stated base rates are accurate**, and some that models do worse than the base rate.\n\n## 6. Recommendations (ledger Arc BD), in order of dependency\n\n1. **BD1, a reference class is a counted claim.** A reference class names its population, its end point and its period, with counted\n   successes and a total from a cited source, its parent reference classes and the reference classes it overlaps. It is a claim anyone\n   can attack. No model states a rate; code reads the counts.\n2. **BD2, the membership pass asks from a shortlist.** Code retrieves a shortlist of reference classes; a first question\n   asks whether any counted reference class fits at all (a gate); then the model judges each candidate and its parent one\n   at a time, yes or no, reasoning before verdict; it is asked twice in different words, and a disagreement\n   counts as not sure.\n3. **BD3, the narrower reference class only when right often enough** (ruled): the pass names a narrower reference class only when\n   its measured accuracy on that level exceeds the break-even of § 3; otherwise the claim stays at the\n   broader reference class.\n4. **BD4, nesting and overlap are recorded, and pooling follows them.** A child pools toward its parents; kinds\n   that only overlap combine only where counts for the overlap exist, and otherwise give a range, marking the\n   claim's prior as depending on which reference class holds.\n5. **BD5, pooling strength is estimated and kinds are scored.** *k* comes from how much sibling reference classes differ;\n   as claims resolve, each reference class and the membership pass are scored against outcomes, and a kind defined by\n   measured features replaces a taxonomy node when its record is better.\n\nHis care for the pass's method and prompting is BD2 with the person-reads-first principle: he reads its first\ncases one at a time, and its rate is proven only for the faults that reading found.\n\n## 7. What only he decides\n\n- BD1: whether kinds are claims in the graph with their counts, or a catalogue kept beside it.\n- BD5: whether a kind defined by matched features (not a named category) may set priors, when its record beats\n  a named one.\n\n## 8. A build order, proposed 2026-10-24 night (unruled)\n\nHis question: *\"Ok how would you propose to go about building, given this order? Think deeply about it.\"* The order\nis BC6's: theory, then every pass's instructions revised, then the seeding pass. The proposal splits the work by\nwhether a model is asked.\n\n**What a prior from a kind looks like, stored** (the BIO table, § 4):\n\n- the claim: *drug X passes from phase II to phase III*\n- the kind, a statistical generalization: *about a quarter of oncology drugs entering phase II reach phase III* (1,732\n  counted, BIO, Informa and QLS 2021)\n- membership, a claim: *drug X is an oncology drug*\n- the prior: 0.25 (pooled, BD4), written with the existing `set_prior` (`deliberus/graph/badge_inputs.py`), its\n  source the kind claim's id. No support link is drawn from the kind, so the kind's bearing counts once, through the\n  start (BC8).\n\nThree things follow from reading the rulings together:\n\n1. **A kind is a statistical generalization, so it needs no new object.** AT1 says such a rule carries one number with its\n   proportion in its words; AY3 reads that number from the cases counted under it, and AY2 gives cases a channel of\n   their own. A kind is that rule, and its counted cases are the table. BC1's ruled text already says the prior is\n   written *\"as a reference-class claim with its source\"*, which answers most of BD1's open half (graph, not a\n   catalogue).\n2. **The half that asks no model can be built before the passes are revised**, since the order binds machine passes:\n   the case channel (AY2, AY3, ruled, unbuilt when this was proposed), kinds as generalizations with parents and overlaps (BD4), pooling with\n   its strength estimated from siblings (BD5), and the prior written through `set_prior`. It is tested on the three\n   real tables of § 4 entered by hand, kinds and memberships written by a person, so the arithmetic meets known\n   outcomes (the replication study's held-out cells) before any model touches it.\n3. **The membership pass is written in step 2 of his order, the seeding pass after it.** The membership pass (BD2:\n   shortlist, gate, candidate and parent one at a time) is revised beside the other passes, against hand cases, read\n   by him one case at a time; BD3's break-even then decides how deep it may name kinds. The seeding pass (BC7's\n   implications, BC2's answers) comes last, one level deep, three or four calls per rebuilt pass (R8), off in\n   production.\n\n**Built the same night** on his word (the button *\"Build the half that asks no model now: the case channel, kinds as\nstatistical rule claims, pooling, the prior through set_start, tested on the three tables by hand.\"*, whose wording the\nsession wrote): `deliberus/reference_classes.py`, tested in `tests/test_reference_classes.py` on the three tables\n(`tests/fixtures/reference_classes/three_tables.json`). A reference class is a claim of kind `reference_class`; bundles\nof counted cases (`case_bundle`) are filed under it; a narrower class lies `WITHIN` a broader; a member is filed\n`IN_CLASS`; the prior goes through `set_prior` with the class as its source. Classes and bundles are left out of\nordinary reading, so no linking pass draws one as a reason for its own members. What it does on the questions still\nopen: the pooling strength is estimated from how much sibling classes differ and never chosen, and a class with no\nsiblings stands alone under a flat prior and says so (Hong Kong's roadworks then read 0.68, not the 0.79 a chosen\nstrength of 20 gave); two filed classes neither within the other write no prior; a class with a narrower filed class\ninside it is set aside, as AM18 rules. Not built: the membership pass (BD2), the band read from a rule's words (his to\nset), leaving a member's own resolved case out of its class's count, and storing any table on the live graph.\n\n**The word *start* became *prior* the same night**, in code, docs and the ruled rows outside his quotes (his words:\n*\"Could we call it priors in the code as well perhaps?\"*, *\"I like to be technically, analytically, philosophically\nand linguistically correct with terms, as you know\"*, and *\"Also go through and rename \"start\" to \"prior\" in\ndocumentation where it's not too cumbersome, and where it has the most impact, in decided rulings in the ledger also,\nyeah?\"*). The stored property is `prior`, `prior_source`, `prior_by` (`docs/graph-properties.md`), set by `set_prior`.\n\n\n\n## 9. Three questions of his, 2026-10-25 (proposals, unruled)\n\n**First, his question on storing classes in the graph**: *\"Why not have kinds/reference classes IN the graph? What's\nthe downside?\"* Three, each with its guard: a class read as a reason for its own members would count one consideration\ntwice (guarded: classes and bundles are left out of ordinary reading, BC8); a class claim carries two numbers, its own\nposterior and the rate it gives its members, which must never be confused (the rate is computed from its bundles, never\nfrom its badge); and a member's own outcome flowing into its class's count feeds back into its own prior (unbuilt:\nleave its own case out). His answer follows.\n\n> *\"If we can diligently guard against the pitfalls and costs, then I'd say it's worth it, for the benefits. Actually,\n> could the feedback you mention not become a feature, since that way... the mind of Deliberus then gets self-updating\n> priors long-term?\"*\n\n**Yes, and it is the case channel doing its job.** When a member resolves (drug X reaches phase III, or does not), it\nis filed under its reference class as a case, and every later member's prior moves with it; each class is then scored\nagainst outcomes (BD5). Three guards: the resolution is itself a claim anyone can attack; a member's own case never\ncounts toward its own prior (its prior stays as written, dated); and a member already inside the counted table (a drug\nin BIO's 2011–2020 count) is not counted again.\n\n> *\"Btw couldn't this whole thing be seen as generalizations and thus a type of *rule* (perhaps rules should be renamed\n> generalizations, at least statistical rules?) that can have cases and exceptions, and both cases and exceptions I\n> guess can be debated/argued about in one addressable place? Isn't this an unexpected feature that helps us\n> potentially?\"*\n\n**A reference class with its rate already is a statistical generalization** (§ 8, point 1), and its cases are filed and\nargued under it, so one home per generalization holds. On the word: philosophy calls the content a *generalization*\n(universal, statistical, normic: Schurz writes *normic laws* and *normic generalizations*), and calls its role in an\ninference the *rule* or *warrant*. Proposed: the claim is a **generalization**, and **rule** names what a generalization\ndoes under a link. That also frees *rule* from colliding with *ruling* in the ledger.\n\n**Other terms used loosely, proposed for his word one at a time:**\n\n- `claim_kind` holds two different things: how a claim came to be (`extracted`, `human`, `implicit`, `machine_step`)\n  and what part it plays (`cq_positive`, `supposed`, `reference_class`, `case_bundle`). Proposed: *origin* and *role*.\n- *case* means an instance under a generalization (AY2), a worked example (*the polls case*) and a test fixture.\n  Proposed: *case* for the first only, *example* for the second, *test case* for the third.\n- *strength* on a claim has no declared meaning (probability, confidence, or the badge's reading; `TODO.md`, the\n  item on declaring what the number means).\n- *pass* and *microtask* name the same thing. Proposed: *machine pass*, with *microtask* for one call.\n\n### 9a. His answers, and three proposals (2026-10-25)\n\nRuled: origin and role told apart (BD6, *\"Blessed\"*, unbuilt) and a generalization with its role as a rule (BD7,\n*\"Ruled, make it so.\"*, applied the same night). Asked:\n\n**Case** (his question: *\"What do you propose here?\"*): *case* keeps one sense, one instance under a generalization\n(AY2). A worked example in a note is an *example* (*the polls example*), and a fixture a test feeds code is a *test\ncase*, the software term, said in full. **Ruled 2026-10-25** (BD9), founder: *\"Agreed about the proposal on case\"*.\n\n**The number on a claim** (his ask: *\"Read up on this in the corpus and through the corpus graph and research it a bit\nand propose the most correct name possible\"*). The corpus's pick (`what-the-number-on-a-claim-means.md` § 4: the\nconfidence the examined case warrants, the best reason as default, probability as the target) and two rows since\n(Q41, open, probability as the target with the record's weight beside the number; Q46, reasons read independent add in\nlog-odds from the prior) make it a prior moved by the examined reasons. Proposed name: **posterior**, with **weight**\n(Keynes's amount examined) beside it. It pairs with *prior*, and where the prior, the rivals and the independence of the\nreasons are recorded it is a probability's posterior exactly. Rejected: *probability*, which promises a calibration the\nnumber lacks where independence is unread; *credence*, an agent's degree of belief, where the number is the record's;\nKyburg's *evidential probability*, the closest in spirit (a probability read from reference classes relative to a body\nof evidence), but interval-valued and built against Bayesian updating, which Q46 uses; *strength*, which stays for a\nlink (Q31) and is the gradual-semantics word. **Ruled 2026-10-25** (BD8): asked *\"Shall I rename the number on a claim to posterior?\"*, the founder answered *\"Yes\"*; the badge's key is `posterior` since that night, in the API and the map's tags.\n\n**Where the posterior departs from Bayes** (his question that night: *\"How do we depart there exactly, is posterior\ndiverging from its theory-use?\"*; he asked to talk it through in depth in the strategy session, which has it). On the\nreasons channel it is Bayes' posterior when the prior is stated and the reasons are read independent. It departs where:\nreasons not yet read for independence count only the best one; news about a claim never flows back to its reasons, so\nnothing is explained away; and a whole's parts cap it at the weakest while lifting it only by a whisper (two parts at\n0.9 and 0.8 give 0.504, where independent Bayes gives 0.72; measured, `what-relevance-is.md` § 3g).\n\n**Microtask and pass** (his words: *\"I like microtask... Is that established outside of Deliberus?\"*). Established:\n*micro-task markets* (Amazon Mechanical Turk) and *microtask crowdsourcing* name small, self-contained units of work\ngiven to many workers who need not talk to each other (Kittur, Smus, Khamkar and Kraut, CrowdForge, UIST 2011, which\nsplits complex work into them). That fits both a model's call and a person's tap. *Pass* is the compiler's word: one\nsweep over every item. Proposed: a **microtask** is one small judgment on one item, by a model or a person; a **pass**\nruns one microtask over many items (a standing pass sweeps the graph); *machine pass* where the actor matters. Bare\n*pass* risks *the tests pass* and *a pass at it*, rarely in context. **Ruled 2026-10-25** (BD10) with *sweep* in place of *pass*, Swedish *svep*: *\"Makes sense. Alrighty. Blessed. Then we can call it a \"svep\" in Swedish also\"*.\n\n### 9b. Where *they are one structure* stops (2026-10-25, his question, unruled)\n\nHis question, on the walkthrough's line that a prior and an inference are one structure: *\"That's a powerful meta aim. What are the limits of this aim?\"* Claude's reading, four limits:\n\n1. **It stops at its own machinery** (already stated as ledger AJ4; the record on arguing the system's own rules, and its other direction, claims that run software, are in `claims-and-code-one-loop.md`; a feature that makes this limit visible, a posterior that lists the rules it leaned on, is `lean-deliberus-analogies.md` § 9.1 point 3). The flat start a class's rate is counted from, and the arithmetic that combines inferences, are form, not claims. Each can be written as a claim and argued, but the code that applies it is not in the graph; a rule turned into a premise needs another rule to apply it (Carroll's tortoise, `philosophy-of-the-unwritten-step.md`).\n2. **It moves the hard question, it does not answer it.** A prior read as an inference puts the difficulty in the case slot: which class this claim belongs to (the reference class problem). The gain is that the choice is now a claim anyone can attack.\n3. **Not everything is an inference.** The experience itself (the person's alone), impact testimony and a value at a horizon enter as claims nobody infers from a rule and a case; forced into the form, a value posit would look like a missing statistic.\n4. **Unifying risks the costliest mistake on record, one word doing two jobs.** Tonight's unification needed a split first (*kind* into kind of inference and reference class). The test before calling two things one: does every operation on one apply to the other, counting, attacking, updating? A member's own case must not count toward its own prior, so a prior and an ordinary inference share a form and keep one difference.\n\n**His question on the third limit, 2026-10-25**: *\"Wait so... Ok so they are claims. But all claims can then be connected\nin arguments/inferences of different kinds. Am I right? Someone could argue I'm a case belonging to a rule after I've\nclaimed something about my experience / personal lived experience. Right?\"* Yes. The limit was about how such a claim\n**enters** (no rule and case produced it), never about what it may **join**. Once in, it is a claim like any other: a\ncase under someone's generalization, a reason for another claim, the target of an attack. Example:\n\n- *I feel uneasy inviting people before the platform is ready* (his report, a claim).\n- *People who have built alone for a long time usually feel uneasy before showing their work* (a statistical\n  generalization) with *he has built alone for a long time* (the case) supports *his unease is the usual pre-launch\n  kind* (a claim about its cause).\n\nThat argument bears on the **cause** of his unease, which anyone, he included, can be wrong about; it cannot overrule\nthe unease itself, which is his alone (the conviction ratified 2026-08-25, *analysis may go anywhere, and owes\nattunement in proportion to depth*). And no machine pass asserts the cause of a named person's belief (ledger AN5, the\nmachine never says why you believe it).\n\n## Cross-references\n\n`priors-as-standing-work.md` (BC1, BC3, BC7, BC8) · `starting-values-and-the-problem-of-the-prior.md` (the prior,\npooling, rec 7 and 8) · `burden-of-proof-and-priors.md` (extraordinary as a prior) · the canon line that a\nmachine's no-answer belongs in a gate · the principle that a person reads a pass before anyone proves its rate.\n"}