{"path":"research/saying-what-kind-a-claim-is/research-report-2026-10-24.md","content":"# Research report: how reference-class assignment is done, and how reliable it is\n\n*A research helper's report (Sonnet, web research, 2026-10-24), written to disk verbatim before it was read\nfurther, for `../saying-what-kind-a-claim-is.md`. Its brief named four uncovered angles and listed the covered\nones to skip. Read-status tags are the helper's own; its numbers are claims until checked against the sources\nit names.*\n\n---\n\nREPORT: how reference-class assignment is done and how reliable it is (four uncovered angles)\n\nRead-status tags: [full text] = I read the tables myself in the PDF; [part] = some sections read; [abstract only]; [secondary] = a search summary or someone else's description. \"Not found\" means I searched and could not find it.\n\nHonest headline: angle 1 (agreement between people on which class to use) has almost no published numbers. Angle 3 (LLMs naming a class and a base rate) is thin and nowhere isolated. Angles 2 and 4 gave usable numbers. Angle 4 gave four real nested cases, below.\n\n## 1. Agreement on class choice, and what the choice does to the estimate\n\n**Agreement rates: not found.** I looked for studies where several analysts pick a class for the same case, with an agreement statistic (kappa, alpha, share picking the same class). I found none. One search summary said the same. If you want the number, you have to measure it, for example with Krippendorff's alpha on your own labelers.\n\n**Lovallo, Clarke & Camerer 2012, \"Robust analogizing and the outside view,\" Strategic Management Journal 33(5):496–512 [abstract only].**\n- In a private-equity study, an outside view (a reference class of analogies) beat a few analogies the decision maker already knew.\n- In a film-revenue study, similarity-based forecasting (reference-class forecasting plus weighting by similarity) beat regression.\n- I could not retrieve sample sizes or error figures. The abstract says existing research gives little guidance on how many analogies to use.\n- [secondary, unverified] A book summary says that using only the single most similar film made forecasts collapse. A full class of 40 analogies was far more accurate.\n- [secondary] A summary says professionals rated their own deal about 50% higher than outside deals that looked alike.\n\n**How much the class definition matters: Theising, \"Reference class selection in similarity-based forecasting of corporate sales growth,\" arXiv 2107.11133 / J. Forecasting 2023, Table 4 [part: tables read, text skimmed].**\n- Setup: out-of-sample backtest on 21,808 US firms, 1950–2019. About 300 candidate ways to define a peer group were ranked by how well the forecast distribution of one-year-ahead sales growth matched actual outcomes.\n- The best class definition was a data-driven one: peers matched on the six-year change in operating margin. It scored KS statistic 1.87 (rank 4 on KS; rank 1 on the quantile-error measure, 0.0155).\n- Two taxonomy-style definitions scored far worse on KS: industry group 10.79 (rank 302) and major group 8.66 (rank 274). Quantile error was 0.0935 for industry group against 0.0155 for the best.\n- The Mauboussin & Callahan (2015) sales-size classes were in between: KS 6.38, rank 213.\n- This is the closest thing I found to a measurement of \"reference class tennis\": the same cases, different class definitions, and a roughly 6-fold gap in distribution-fit error between a taxonomy node and the best neighbour set.\n- Also from that paper: Mauboussin & Callahan's classes were \"neither theoretically derived nor empirically backtested.\"\n\n**Other evidence on class choice, all [secondary] (search summaries, not read):**\n- Batselier & Vanhoucke: reference-class forecasting beats earned-value and Monte Carlo methods only when the similarity of the reference projects is high enough.\n- A US mass-transit study rebuilt classes by unsupervised clustering instead of predefined categories and recommended documenting and testing class formation.\n- Objective peer-group methods can yield classes of about six elements, which brings bias back.\n\n**Do forecasters use reference classes, and does it help?**\n- Good Judgment Project: \"comparison classes\" use correlated with accuracy [secondary, AI Impacts summary].\n- Karvetski, Meinel, Maxwell, Lu, Mellers & Tetlock 2022, Int. J. Forecasting 38(2):688–704: top forecasters' written rationales use more comparison classes [abstract-level, secondary]. No accuracy statistics retrieved.\n\n## 2. Machines assigning items to taxonomy nodes\n\n**Fine versus coarse levels**\n- TaxoGlimpse, Sun et al., VLDB 2024 (arXiv 2406.11131) [part: full-text search, not a table-by-table read].\n  - It tests 18 LLMs on 10 taxonomies, root to leaf.\n  - The question type is \"is X a type of Y\", not assigning a document to a class, so treat it as a proxy.\n  - Common taxonomies (shopping, Schema.org): most LLMs reach about 80% at every level. Accuracy drops from root to leaf in nearly all of them (ICD-10-CM, ACM-CCS, Glottolog and others), and is worst in specialized taxonomies near the leaves.\n  - Giving multiple-choice options cut Llama-3-70B's miss rate from 0.151 to 0.005 and raised its accuracy from 0.694 to 0.791.\n- Many classes at once: gpt-3.5-turbo reaches 32.51% zero-shot choosing among 60 categories, against 94.29% for 2 categories (arXiv 2406.07001) [secondary].\n\n**Medical coding**\n- Soroush et al., NEJM AI 2024, \"Large language models are poor medical coders\" [secondary: several consistent reports]. GPT-4 asked directly for codes got exact-match 45.9% (ICD-9-CM), 33.9% (ICD-10-CM), 49.8% (CPT).\n  - Shorter codes with shorter descriptions matched more often.\n  - GPT-3.5 gave the most generalized-but-correct codes, which is the \"assigned a broader class\" behaviour.\n  - The test set was over 27,000 codes from one hospital system.\n- Kwan 2024 (arXiv 2407.12849) [abstract only]: a retrieve-then-rank system got 100% against 6% for a plain gpt-3.5-turbo. But the set was only 100 simplified single-term conditions, so it is a demonstration, not a rate.\n\n**Retrieve candidates first, then judge each one**\n- Industrial classification of scientific papers (Elsevier/SSRN), COLING-Industry 2025 (aclanthology 2025.coling-industry.14) [full text, tables read].\n  - Taxonomy: 2,778 leaf labels, 477 parents, up to 9 levels. Scored by a subject-matter expert on 70 documents.\n  - A bi-encoder shortlist of 40 leaves, then an LLM judging each leaf and its parent separately (pointwise): 94.3% accuracy.\n  - Same shortlist but the LLM picks in one pass: 58.6%. LLM re-ranking: 70.0%. LLM walking down the tree layer by layer: 50.0%. Previous machine-learning state of the art: 61.5%.\n  - Ablations: dropping the shrinking of the candidate set to a random pick gave 62.9%; dropping label descriptions or parent context gave 85.7%.\n  - The authors report that human labelers' accuracy varied between 65% and 90% depending on time pressure.\n- Biomedical benchmark (Chen et al., Nat. Commun. 2025) [secondary]: GPT-4 on multi-label LitCovid had macro-F1 0.5901 with fixed one-shot examples, 0.6500 with the nearest example retrieved, 0.7055 with the five nearest.\n\n**Abstention / \"none fits\"**\n- Tam et al. 2025, arXiv 2503.01550 [abstract only]: across 28 LLMs on MMLU, accuracy fell about 30–50% when \"None of the Above\" was the correct answer, in every model size. The drop ranged from 14.6% in math to 48.1% in business ethics.\n- arXiv 2507.16199 [abstract only]: adding an \"Unknown\" option caused serious accuracy drops on true/false questions. Replacing \"Unknown\" with an unrelated random word gave a similar effect, so the extra option itself triggers abstention. The effect is tied to instruction tuning, grows with difficulty, and shrinks at larger sizes.\n- No study measured \"none fits\" in an open taxonomy. The only evidence is these general multiple-choice results.\n\n**\"Most specific class with adequate data\"**: not found as a tested prompt. The nearest design evidence is the pointwise leaf-plus-parent judgment above.\n\n## 3. LLMs retrieving or estimating base rates\n\n- Not found: a study where an LLM names a reference class and its rate, with the rate or the resulting forecast scored against outcomes.\n- Not found: an ablation that isolates a base-rate or outside-view step.\n- Halawi et al., NeurIPS 2024 [part]: their prompt tells the model to consider historical base rates, but it is bundled with retrieval and other steps. Their search snippet reports Brier 0.179 for the full system, better than predicting the dataset's base rate.\n- Metaculus Q2 2025 write-up and FutureEval [secondary]: the underlying model matters more than prompt wording, and score correlated with how many distinct research sources a bot used.\n- Selby et al., \"Had enough of experts?\" (arXiv 2402.07770 / Stat 2025) [part: skimmed]: LLM-elicited priors can improve prediction and cut data needs on some datasets, but LLM-based imputation often underperformed a mean/mode baseline on macro-F1. Treat this as caution, not support.\n- A December 2025 prediction-market study (arXiv 2512.16030) [abstract only] reports that most models score worse than simply predicting the base rate.\n- Clinical-trial outcome predictors (LINT, BioBERT-based) [secondary]: ROC-AUC about 0.7. These are trained models, not LLMs asked for a rate.\n\n## 4. Real cases with counts at nested levels (for tests)\n\n**Case A. Drug development, by therapeutic area, in the Wong–Siah–Lo tables**\n- Source: Wong, Siah & Lo, Biostatistics 20(2):273–286 (2019), Table 2 \"POS by therapeutic group,\" 2000–2015, path-by-path method [full text of the advance-article PDF].\n- Caution: the 2019 corrigendum (kxy072) says Table 1 and Table 2's Phase 2→3 columns were wrong because of an Excel error. In the advance PDF I read, the Table 2 stage columns do not multiply to the overall figure, so I am not quoting Phase 2→3 rates. The path counts and the overall figures below are not on the corrigendum's error list, but I did not see the corrected tables.\n- Counts are \"total paths\" starting at Phase 1, with the overall approval probability from Phase 1:\n\n| Level | Phase-1 paths | Overall probability of approval |\n|---|---|---|\n| All indications | 41,040 | 13.8% |\n| Oncology | 17,368 | 3.4% (about 590 approvals, derived) |\n| Vaccines | 1,869 | 33.4% (about 624, derived) |\n| Infectious disease (non-vaccine) | 3,963 | 25.2% |\n| Ophthalmology | 674 | 32.6% |\n| Genitourinary | 757 | 21.6% |\n| Cardiovascular | 2,810 | 25.5% |\n| Metabolic/endocrine | 3,589 | 19.6% |\n| CNS | 4,924 | 15.0% |\n| Autoimmune/inflammation | 5,086 | 15.1% |\n\n- Phase 3 to approval, oncology: 1,236 paths at 35.5%. Overall: 7,532 paths at 59.0%.\n- \"Lead indication\" version, Table 2 lower half: oncology 3,107 Phase-1 paths at overall 11.4%; overall 16,467 at 21.6%.\n- Derived approval counts are rate times paths. They are approximate because the method works over drug paths, not independent trials.\n\n**Case B. Drug development, three nested levels plus a later outcome check, in the BIO/Informa/QLS report**\n- Source: BIO, Informa Pharma Intelligence & QLS Advisors, \"Clinical Development Success Rates and Contributing Factors 2011–2020\" (Feb 2021), Figures 2, 5b, 6, 7 [full text, tables read]. Total: 12,728 phase transitions, 9,704 programs.\n- Here n is the number of \"advanced or suspended\" transitions at that phase. For a phase-to-phase rate it is the denominator of a binomial. For \"likelihood of approval from Phase I\" it is the total transitions of all phases, so it is not a plain binomial denominator.\n- Phase II to III success (Figures 2, 6, 7):\n\n| Level | n | Rate | Approx. successes (derived) |\n|---|---|---|---|\n| All indications | 4,933 | 28.9% | about 1,426 |\n| Oncology | 1,732 | 24.6% | about 426 |\n| Non-oncology | 3,201 | 31.2% | about 999 |\n| Oncology: solid tumors | 1,261 | 23.4% | about 295 |\n| Oncology: hematologic | 449 | 27.8% | about 125 |\n| Oncology: immuno-oncology (overlaps solid and hematologic) | 244 | 40.2% | about 98 |\n\n- Likelihood of approval from Phase I: all 7.9% (n=12,728); oncology 5.3% (n=4,179); non-oncology 9.3% (n=8,549); solid 4.6% (n=2,982); hematologic 7.5% (n=1,094); immuno-oncology 12.4% (n=679).\n- Small narrow cells in Figure 2/5b: urology Phase II 15.0% (n=40), allergy Phase III 64.7% (n=34), hematology Phase I 69.6% (n=92).\n- Phase I to II: oncology 48.8% (n=1,628) against all 52.0% (n=4,414). Phase III to submission: oncology 47.7% (n=495) against all 57.8% (n=1,928).\n- Cross-source \"later outcome\" comparison:\n  - Wong's overall oncology figure (3.4%, 2000–2015) is below BIO's 5.3% (2011–2020).\n  - Thomas et al. (2016) reported 5.1% for oncology [secondary, quoted in Wong's text]; Hay et al. (2014) reported 6.7%.\n  - BIO's own overall (7.9%) is below Wong's 13.8%, but the methods and the end point (approval versus submission) differ. The oncology share of transitions in BIO was 33%, so the broad average also depends on mix.\n\n**Case C. Project cost overrun, with nested class and a leave-one-out check**\n- Flyvbjerg, Holm & Buhl 2002 as reproduced in \"Survival of the Unfittest\" (arXiv 1303.6571), Table 1 [full text]: cost overrun in constant prices.\n\n| Type | n | Mean overrun | SD |\n|---|---|---|---|\n| Rail | 58 | 44.7% | 38.4 |\n| Bridges and tunnels | 33 | 33.8% | 62.4 |\n| Roads | 167 | 20.4% | 29.9 |\n\n- Total 258 projects in 20 nations.\n- Flyvbjerg, Hon & Fok 2016, \"Reference class forecasting for Hong Kong's major roadworks projects,\" Proc. ICE 169(CE6) (arXiv 1710.09419), Table 4 and Table 2 [full text]:\n  - Broad class: 863 road projects from 34 countries, mean overrun +20%, 9 of 10 overran, SD 30%, mean schedule overrun +38%.\n  - Narrow class: 25 Hong Kong roadworks (23 with the comparable \"Category C\" estimate), mean +11%, 7 of 10 overran, SD 38%, mean schedule overrun +58%.\n  - The difference in mean was not significant. The difference in frequency of overruns was (p = 0.006). For time, the mean difference was p = 0.089 and the frequency of delays 8 of 10 against 6 of 10 (p = 0.048).\n  - Leave-one-out inside the narrow class (18 projects with Category C data): a P50 uplift would have covered the overrun in 9 of 18, a P80 uplift in 14 of 18 (78%).\n- This is a natural test of pooling: a narrow rate (+11%, n about 23) against a broad one (+20%, n=863) with the narrow class's variance larger.\n\n**Case D. Replication rates by field and sub-field, with counts**\n- Open Science Collaboration 2015, Science, Table 1 [full text]: studies replicated with p < .05 in the original direction.\n\n| Level | Replicated / original-significant | Rate |\n|---|---|---|\n| Overall | 35 / 97 | 36% |\n| JPSP, social | 7 / 31 | 23% |\n| JEP:LMC, cognitive | 13 / 27 | 48% |\n| Psychological Science, social | 7 / 24 | 29% |\n| Psychological Science, cognitive | 8 / 15 | 53% |\n| All social (combined) | 14 / 55 | 25% |\n| All cognitive (combined) | 21 / 42 | 50% |\n\n- Of the 100 replications, 32 came from JPSP, 28 from JEP:LMC, 40 from Psychological Science; 57 were coded social and 43 cognitive.\n- Later outcomes for different, narrower kinds [secondary]: Camerer et al. 2016 (economics): 11 of 18 replicated. Camerer et al. 2018 (social science in Nature and Science, 2010–2015): 13 of 21 (62%).\n- Class matters here: top-journal social experiments (62%, n=21) differ from the 2008 social psychology sample (25%, n=55).\n\n**Case E. Establishment survival by sector (no counts)**\n- BLS Employment Dynamics, \"34.7 percent of business establishments born in 2013 were still operating in 2023,\" Table [full text].\n- Share still operating, by year since March 2013:\n\n| Sector | 5 years (2018) | 10 years (2023) |\n|---|---|---|\n| Total private sector | 50.6% | 34.7% |\n| Agriculture, forestry, fishing | 66.2% | 50.5% |\n| Utilities | 56.6% | 45.7% |\n| Health care and social assistance | 55.1% | 35.7% |\n| Construction | 53.9% | 40.1% |\n| Professional, scientific, technical | 46.3% | 30.9% |\n| Information | 44.3% | 29.1% |\n| Mining and oil and gas | 40.2% | 24.5% |\n\n- Establishment counts per cell: not found on the page. They would come from the BLS BED data tables.\n- Ranks shift with time: health care is above the total at 5 years but at the total by 10 years. This is a useful warning for a class whose rate is measured at one horizon.\n\n## What a designer of a reference-class pass should take from this\n\n- **Do not trust a taxonomy node to be the best class.** In Theising's backtest, industry group, the standard taxonomy class, ranked last among about 300 ways to define peers. Consider letting the pass propose several classes at different levels and scoring them against outcomes, not just choosing one.\n- **Expect weak agreement on class choice and plan to measure it.** No study gives a number. Run the pass several times, or with several prompts or models, and report the share that agree at each level. Treat disagreement as a signal to widen to the broader class, not as noise.\n- **Give the machine a shortlist, then ask a yes/no question about each candidate and its parent.** In the one large controlled test (2,778 leaves), that pointwise leaf-plus-parent design scored 94.3%, against 50.0% for walking the tree and 58.6% for picking in one pass. Giving multiple-choice options also helped in TaxoGlimpse.\n- **Expect accuracy to fall toward the leaves and with the number of classes.** TaxoGlimpse shows root-to-leaf decline in nearly all taxonomies, and accuracy at 60 categories was a third of that at 2. A pass that must pick the most specific class will be least reliable exactly where pooling depends on it. Pooling should therefore trust the parent more when the leaf assignment is uncertain.\n- **Test \"none fits\" separately and do not assume it helps.** An \"Unknown\" option lowered accuracy on some tasks, and when \"None of the Above\" was correct accuracy dropped 30–50%. Make it a separate gate question, then check how often it fires on your own cases.\n- **Never ask a model to state the rate itself.** I found no evidence that LLM-stated base rates are accurate, and some that models underperform the base rate. Have the pass name classes and let code compute the rates from counted data.\n- **Use these tables as fixtures, with their caveats.** BIO 2021 (all, oncology, solid, hematologic, immuno-oncology), Hong Kong roads against 863 international roads, and OSC 2015 by journal and sub-field give two to three nested levels with counts and a way to compare a narrow rate to a broad one. Wong's Phase 2→3 columns are corrected in the published version, so only use the published tables. The BLS table has the levels but no counts.\n- **Check the end point and period before pooling.** Wong measures approval, BIO measures submission or approval depending on the table, and the sample years differ. Their oncology overall rates (3.4% against 5.3%) show how much that alone moves a rate.\n- **Be careful with overlapping classes.** BIO's immuno-oncology class overlaps both solid and hematologic, so a nested-pooling scheme needs to know which classes are parents of which and which only overlap.\n\nFiles and sources consulted (all web): Lovallo, Clarke & Camerer 2012 (Caltech repository, SMJ); Theising arXiv 2107.11133; Wong, Siah & Lo 2019 advance PDF (gwern.net/doc/biology/2018-wong-2.pdf) and PMC corrigendum PMC6409416; BIO 2021 report (go.bio.org/rs/490-EHZ-999/images/ClinicalDevelopmentSuccessRates2011_2020.pdf); Flyvbjerg arXiv 1303.6571 and 1710.09419; OSC 2015 (discovery.dundee.ac.uk RPP_SCIENCE_2015.pdf); BLS TED 2024 article; COLING-Industry 2025 (2025.coling-industry.14); arXiv 2406.11131, 2503.01550, 2507.16199, 2407.12849, 2402.07770, 2402.18563.\n"}