{"path":"research/what-counts-as-evidence/grading-evidence-in-science-and-policy.md","content":"# Tiers of evidence, and how science and policy grade them\n\n*Research report for Deliberus, 2026-09-26. Scope: evidence hierarchies, GRADE, risk-of-bias\ntools, meta-analysis, causal inference outside RCTs, statistics-as-evidence, grading outside\nmedicine, aggregating heterogeneous evidence, and an assessment of the platform's inert\nmethodology ladder (meta-analysis 0.70 … expert opinion 0.30).*\n\n**Verification convention.** Numbers marked **[verified]** were checked against a primary source or\nits abstract during this session (2026-09-26). Numbers marked **[recalled]** are standard figures I\nam confident of but did not re-open the primary source for today. Anything marked **[unverified]**\nshould be checked before it is cited outward.\n\n---\n\n## 0. The argument in one page\n\n1. **The single ladder is dead in the literature that invented it.** The 2011 Oxford CEBM Levels\n   are not one ladder but a table of ladders, one per *question type* (prevalence, diagnosis,\n   prognosis, treatment benefit, common harms, rare harms, screening), and even that table carries\n   a footnote saying a level \"may be graded down on the basis of study quality, imprecision,\n   indirectness …, inconsistency …, or because the absolute effect size is very small; Level may\n   be graded up if there is a large or very large effect size\" **[verified]**. GRADE (2004–)\n   turned that footnote into the whole method: design sets a *starting point*, and five\n   down-domains and three up-domains move it. Since 2019 even the starting point is not fixed by\n   design label: non-randomised studies assessed with ROBINS-I *start high* and are downgraded\n   for the bias the tool actually finds (GRADE guideline 18) **[verified]**.\n2. **The unit that gets graded is not a study but a *body of evidence for one outcome in one\n   question*.** GRADE rates \"certainty of evidence\" per outcome across studies; RoB 2 and\n   ROBINS-I rate *a specific result* of a study, not the study; EEF padlocks rate *the primary\n   outcome* of an evaluation. The platform's ladder rates a *source* by *design*, which is the\n   level of granularity every current framework explicitly abandoned.\n3. **Certainty and direction are separate axes everywhere mature.** GRADE separates certainty\n   from effect size and from strength of recommendation; IPCC separates *confidence*\n   (evidence × agreement) from *likelihood* (a probability); IARC separates *strength of evidence\n   for a hazard* from *magnitude of risk*. Deliberus's Aug-2026 split of direction from weight\n   (Keynes) is on the right side of this line, and the literature suggests a third axis —\n   *applicability/indirectness to the question asked* — that belongs on the **edge**, not the node.\n4. **Every GRADE domain is already a critical question.** Risk of bias, inconsistency,\n   indirectness, imprecision and publication bias are questions one asks of the inference *from\n   a body of studies to a claim*; large effect, dose-response and confounding-direction are\n   questions that, when answered well, *raise* confidence. They map cleanly onto the CQ\n   mechanic, **provided answers are weighed by content rather than counted** — and counting is\n   exactly the \"vote counting\" method Hedges & Olkin (1980) showed can converge on the wrong\n   answer as studies accumulate.\n5. **\"Expert opinion\" is not an evidence tier.** GRADE's 2019 BMJ position is that expert\n   *opinion* is an interpretation of evidence, not a kind of evidence; what experts report having\n   *observed* is evidence and should be graded like a case series. The Walton expert-opinion\n   scheme already handles appeals to authority as their own inference with their own CQs; putting\n   it on the bottom rung of a design ladder is a category error.\n6. **Mechanistic evidence is a separate stream, not the bottom rung.** OCEBM puts\n   \"mechanism-based reasoning\" at level 5; the EBM+ programme (Clarke, Gillies, Illari, Russo,\n   Williamson) and IARC's 2019 Preamble treat it as a distinct stream that combines with\n   correlational evidence — IARC can reach Group 1 with *limited* human evidence if animal\n   evidence is sufficient and mechanistic evidence in exposed humans is strong **[verified]**.\n7. **Grading is itself noisy.** Trained GRADE raters reach inter-rater reliability of about\n   0.66–0.72, against 0.27–0.31 for intuitive global judgements (Mustafa et al. 2013)\n   **[verified]**; but per-domain agreement for imprecision and risk of bias can fall to\n   κ ≈ 0.06–0.21 (Hartling et al. 2012) **[verified]**. So grading judgements are exactly the kind\n   of contested ground where the machine should propose and people argue.\n\n---\n\n## 1. The evidence pyramid and hierarchies of study design\n\n### 1.1 History\n\n- **Canadian Task Force on the Periodic Health Examination (1979)**, *CMAJ* 121:1193–1254. The\n  first widely adopted grading: evidence levels I (at least one properly randomised trial), II-1\n  (well-designed controlled trials without randomisation), II-2 (cohort or case-control, preferably\n  multi-centre), II-3 (multiple time series or dramatic uncontrolled results), III (opinions of\n  respected authorities, descriptive studies, expert committees); recommendation grades A–E.\n  **[recalled]**\n- **Sackett (1986, 1989)**, *Chest* — levels I–V for antithrombotic recommendations, tying\n  levels to Type I/Type II error: level I = large RCTs with clear results (low risk of both errors),\n  level II = small RCTs with uncertain results, level III = non-randomised concurrent cohort, level\n  IV = historical controls, level V = case series without controls. **[recalled]** Notice that\n  Sackett's original ladder already encoded *precision* (\"large … clear results\" vs \"small …\n  uncertain\"), which later popular pyramids dropped.\n- **Oxford CEBM Levels (1998, revised 2009)** — separate columns for therapy/prevention/aetiology/\n  harm, prognosis, diagnosis, differential diagnosis, and economic analysis, with sublevels 1a–5\n  and \"minus\" modifiers for imprecision and heterogeneity. **[recalled]**\n- **Oxford CEBM 2011 Levels** (OCEBM Levels of Evidence Working Group: Howick, Chalmers, Glasziou,\n  Greenhalgh, Heneghan, Liberati, Moschetti, Phillips, Thornton, Goddard, Hodgkinson)\n  **[verified]**. Organised by clinical question, each row read left to right as \"steps\" in\n  searching for the likely best evidence. Rows **[verified]**:\n\n  | Question | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 |\n  |---|---|---|---|---|---|\n  | How common is the problem? | local, current random-sample surveys or censuses | SR of surveys matchable to local circumstances | local non-random sample | case series | n/a |\n  | Is this diagnostic test accurate? | SR of cross-sectional studies with consistent reference standard and blinding | individual such studies | non-consecutive studies or no consistent reference | case-control, poor/non-independent reference | mechanism-based reasoning |\n  | What will happen without therapy? (prognosis) | SR of inception cohorts | inception cohort | cohort or control arm of RCT | case series/case-control/poor cohort | n/a |\n  | Does this intervention help? | SR of RCTs or n-of-1 trials | RCT or observational study with dramatic effect | non-randomised controlled cohort/follow-up | case series, case-control, historically controlled | mechanism-based reasoning |\n  | Common harms | SR of RCTs, SR of nested case-control, n-of-1 in *this* patient, or dramatic observational | individual RCT or (exceptionally) dramatic observational | non-randomised cohort/post-marketing surveillance *with sufficient numbers* | case series/case-control/historical | mechanism-based reasoning |\n  | Rare harms | SR of RCTs or n-of-1 | RCT or (exceptionally) dramatic observational | as above | as above | mechanism-based reasoning |\n  | Is early detection worthwhile? (screening) | SR of RCTs | RCT | non-randomised controlled cohort | … | … |\n\n  Plus the footnote on grading down/up quoted in §0, and \"As always, a systematic review is\n  generally better than an individual study.\"\n\n  Three features matter for Deliberus. (a) The *question type* picks the ladder; for\n  prevalence the top rung is a local random survey, and an RCT is irrelevant. (b) \"Observational\n  study with dramatic effect\" sits beside the RCT at step 2 — design is overridden by effect size\n  (the parachute/penicillin logic; Glasziou et al. 2007 BMJ \"When are randomised trials\n  unnecessary?\" **[recalled]**). (c) Mechanism-based reasoning appears only at the bottom.\n\n- **The \"new evidence pyramid\"** (Murad, Asi, Alsawas, Alahdab 2016, *BMJ EBM* 21:125–127)\n  **[recalled]**: the dividing lines between designs become *wavy* (a study can move up or down\n  by its quality — GRADE's influence) and systematic reviews are removed from the apex and drawn\n  as a *lens* through which the other studies are viewed. This is the single most useful picture\n  for the platform: a meta-analysis is not a higher-tier study; it is a *reading of* the studies\n  below it, and inherits their quality.\n\n### 1.2 The classic ordering\n\nPopular pyramids (medical libraries, textbooks): systematic review / meta-analysis > RCT > cohort >\ncase-control > case series > case report > expert opinion / mechanistic reasoning / bench research\n> anecdote. This is the ordering the platform's ladder copies, with systematic review and\nmeta-analysis split and \"quasi-experimental\" inserted from social-policy scales.\n\nWhat the ordering is *actually* a proxy for: the probability that confounding, selection and\nmeasurement biases distort the effect estimate for *therapeutic* questions. It is not a measure of\nevidential weight in general, and its authors never claimed it was.\n\n### 1.3 Critiques (philosophy of medicine and methodology)\n\n- **Worrall (2002)**, \"What evidence in evidence-based medicine?\", *Philosophy of Science* 69(S3):\n  S316–S330; and **Worrall (2007)**, \"Why there's no cause to randomize\", *BJPS* 58:451–488.\n  Randomisation does not guarantee balance of unknown confounders in any single trial; the case for\n  RCTs over well-controlled observational studies is weaker than EBM rhetoric implies; the\n  epistemic work is done by *control* of known confounders and blinding, not by randomisation's\n  mystique. **[recalled]**\n- **Howick (2011)**, *The Philosophy of Evidence-Based Medicine* (Wiley-Blackwell / BMJ Books).\n  Defends a *modified* hierarchy: comparative clinical studies are generally better at ruling out\n  alternative explanations, but (i) observational studies with dramatic effects can suffice, (ii)\n  \"placebo-controlled\" and \"double-blind\" are frequently not what they seem, (iii) mechanistic\n  reasoning is rightly low *as usually practised* because mechanisms are complex and poorly known,\n  but high-quality mechanistic knowledge is legitimate evidence. Howick, Glasziou & Aronson (2009,\n  *JRSM* 102:186–194) read the Bradford Hill viewpoints as three evidence *types* — direct\n  (strength, temporality, dose-response), mechanistic (plausibility, coherence), and parallel\n  (consistency, replication, analogy) — and argue hierarchies should combine them rather than rank\n  them. **[recalled]**\n- **Cartwright (2007)**, \"Are RCTs the gold standard?\", *BioSocieties* 2:11–20; **Cartwright &\n  Hardie (2012)**, *Evidence-Based Policy: A Practical Guide to Doing It Better* (OUP). An RCT\n  establishes \"it worked *there*\"; the policy question is \"will it work *here*\", which needs\n  evidence that the causal *capacities* and *support factors* are present in the target. RCTs are\n  strong on internal validity and silent on external validity; the hierarchy ranks only the former.\n  **[recalled]**\n- **Bluhm (2005)**, \"From hierarchy to network: a richer view of evidence for evidence-based\n  medicine\", *Perspectives in Biology and Medicine* 48(4):535–547. Evidence should be seen as a\n  network in which different kinds of study constrain each other, not a ladder. **[recalled]**\n  (This is the closest philosophical statement of the Deliberus graph view.)\n- **Stegenga (2011)**, \"Is meta-analysis the platinum standard of evidence?\", *Studies in History\n  and Philosophy of Biological and Biomedical Sciences* 42(4):497–507 **[verified]**. Meta-analysis\n  is *malleable*: the many choices (inclusion, quality weighting, outcome, model) leave wide\n  latitude, and different meta-analyses of the same evidence reach contradictory conclusions; it\n  therefore fails *constraint* (it does not force a single answer) and *objectivity*.\n- **Stegenga (2018)**, *Medical Nihilism* (OUP). Given the malleability of methods, publication\n  bias and industry influence, and the small effect sizes typical of modern drugs, our confidence\n  in the effectiveness of medical interventions should be *low*; proposes the \"master argument\"\n  (low prior + weak/malleable evidence ⇒ low posterior). **[recalled]**\n- **Concato, Shah & Horwitz (2000)**, *NEJM* 342:1887–1892, and **Benson & Hartz (2000)**, *NEJM*\n  342:1878–1886: well-designed observational studies did not systematically overestimate effects\n  relative to RCTs on the same questions. **[recalled]** Later reviews: Anglemyer, Horvath & Bero\n  (2014, Cochrane MR000034) found on average little difference between RCT and observational\n  estimates (pooled ratio of odds ratios ≈ 1.08) **[recalled — check exact CI before citing]**;\n  and the RCT-DUPLICATE project (Wang et al. 2023, *JAMA*) emulated 32 trials with claims data:\n  Pearson r = 0.82 between emulation and trial, 75% agreement on statistical significance; where\n  design emulation was close, r = 0.93, 94% significance agreement, 88% estimate agreement; where it\n  was difficult, r = 0.53 **[verified]**. The design label mattered less than how closely the\n  observational study reproduced the trial's question.\n- **The EBM+ programme** (Clarke, Gillies, Illari, Russo & Williamson 2014, \"Mechanisms and the\n  evidence hierarchy\", *Topoi* 33:339–360; Parkkinen, Wallmann, Wilde, Clarke, Illari, Kelly,\n  Norell, Russo, Shaw, Williamson 2018, *Evaluating Evidence of Mechanisms in Medicine*, Springer\n  Briefs, open access) **[recalled]**. Built on the **Russo–Williamson thesis** (2007, *Int. Stud.\n  Phil. Sci.* 21:157–170): to establish a causal claim one normally needs evidence *both* that the\n  cause and effect are correlated (difference-making) *and* that there is a mechanism linking them.\n  The two kinds are complementary: correlation alone is confounded or chance; mechanism alone may\n  be masked by counteracting mechanisms. EBM+ proposes evaluating mechanistic evidence with its own\n  quality criteria rather than parking it at level 5. The thesis is contested (e.g. Howick; Broadbent\n  argues it over-reaches for some cases), but IARC's 2019 procedure is an institutional\n  instance of it.\n\n**Consensus vs contested.** Consensus: design alone is insufficient; question type matters; SRs\nare only as good as their inputs; dramatic effects can override design. Contested: how much weight\nmechanistic evidence deserves; whether RCTs deserve their premium; whether any ordinal ladder\nshould survive at all.\n\n---\n\n## 2. GRADE in depth\n\nGRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group, founded\n2000; foundational papers: GRADE Working Group (2004, *BMJ* 328:1490); Guyatt et al. (2008, *BMJ*\n336:924–926, \"GRADE: an emerging consensus on rating quality of evidence and strength of\nrecommendations\"); the *Journal of Clinical Epidemiology* \"GRADE guidelines\" series (2011–,\nnumbered papers now past 36). Used by WHO, Cochrane, NICE, and 100+ organisations. **[recalled]**\n\n### 2.1 Certainty levels and their meanings (Balshem et al. 2011, GRADE guideline 3, *JCE* 64:401–406) [recalled — wording standard]\n\n| Certainty | Meaning |\n|---|---|\n| **High** | We are very confident that the true effect lies close to that of the estimate. |\n| **Moderate** | We are moderately confident: the true effect is likely close to the estimate, but there is a possibility that it is substantially different. |\n| **Low** | Our confidence is limited: the true effect may be substantially different from the estimate. |\n| **Very low** | We have very little confidence: the true effect is likely to be substantially different from the estimate. |\n\nLater GRADE work (Hultcrantz et al. 2017, *JCE* 87:4–13) clarified that certainty is *certainty\nthat the true effect lies on one side of a threshold* (or within a range), and distinguished the\n**minimally contextualised** approach (threshold = no effect or a minimal important difference)\nfrom the **fully contextualised** approach (threshold set by the decision, considering all\noutcomes). **[recalled]** This matters: certainty is always *relative to a claim* (e.g., \"the\neffect exceeds the MID\"), not a free-floating property of studies — exactly the graph's framing.\n\n### 2.2 Starting points\n\n- Body of evidence from **RCTs starts High**.\n- From **observational studies starts Low** (traditional).\n- Since **GRADE guideline 18** (Schünemann et al. 2019, *JCE* 111:105–114) **[verified]**: when\n  non-randomised studies are assessed with a tool that rates bias on an *absolute* scale relative to\n  a hypothetical target trial (ROBINS-I), they **start High** and the tool's findings (typically\n  confounding and selection) downgrade them — usually by two levels, so the destination is often\n  the same, but *the reason is recorded*.\n- **Expert opinion is not a starting point.** Schünemann, Zhang & Oxman (2019, *BMJ* 366:l4606,\n  \"Distinguishing opinion from evidence in guidelines\") **[recalled]**: expert opinion is an\n  interpretation of evidence, not evidence; experts' reported *experience* should be elicited\n  systematically and graded as observational evidence (usually like a case series, i.e. very low).\n\n### 2.3 The five downgrading domains (each −1 \"serious\" or −2 \"very serious\")\n\n1. **Risk of bias** (Guyatt et al. 2011, GRADE 4, *JCE* 64:407–415): limitations in design or\n   execution of the included studies — assessed per study by RoB 2 / ROBINS-I, then judged across\n   the body, weighted toward the studies that contribute most information.\n2. **Inconsistency** (GRADE 7, 2011; updated GRADE guidance 36, 2023 **[verified title]**):\n   unexplained heterogeneity of effect estimates — non-overlapping CIs, point estimates on\n   different sides of the threshold, large I², significant heterogeneity test. Emphasis since the\n   update: judge inconsistency *relative to the decision threshold*, using prediction intervals,\n   not I² alone.\n3. **Indirectness** (GRADE 8, 2011): differences between the studies' PICO (population,\n   intervention, comparator, outcome) and the question's PICO; indirect comparisons (A vs C and B vs\n   C to infer A vs B); surrogate outcomes.\n4. **Imprecision** (GRADE 6, 2011; updated in GRADE guidance 34 and 35, 2022 **[recalled]**):\n   the confidence interval crosses the decision threshold (or includes both important benefit and\n   important harm), or the total sample fails the **optimal information size** (OIS — the sample a\n   single adequately powered trial would need). Imprecision is GRADE's analogue of Keynes's\n   *weight*.\n5. **Publication bias** (GRADE 5, 2011): \"suspected\" or \"undetected\" — suspected from funnel\n   asymmetry, a small number of small industry-funded trials, known unpublished trials in\n   registries, or early small positive studies. Notably graded as a *judgement about the search*,\n   not about any one study.\n\n### 2.4 The three upgrading domains (mainly for observational bodies with no serious limitation; Guyatt et al. 2011, GRADE 9, *JCE* 64:1311–1316) [recalled]\n\n1. **Large effect**: +1 if RR > 2 or < 0.5 with no plausible confounders; +2 if RR > 5 or < 0.2\n   (\"very large\").\n2. **Dose-response gradient**.\n3. **All plausible residual confounding would reduce a demonstrated effect** (or would suggest a\n   spurious effect when none is observed) — i.e., the known biases point *against* the finding.\n\nThe third is the logically most interesting: it is an argument that *the direction of the likely\nbias* strengthens rather than weakens the inference — the same logic as triangulation (§5.2).\n\n### 2.5 Evidence profiles and Summary of Findings (SoF) tables\n\nFor each critical outcome: number of studies and participants, design, the five domain judgements\nwith footnoted reasons, relative effect (95% CI), anticipated absolute effects (baseline risk and\nrisk difference per 1,000), certainty, and a plain-language statement using standard phrases\n(\"probably reduces\", \"may reduce\", \"we are uncertain whether …\" for high / moderate / low / very\nlow). Guyatt et al. 2011 (GRADE 12, 13) and Santesso et al. 2020 (*JCE* 119:126–135, informative\nstatements) **[recalled]**. The plain-language convention is itself a calibrated hedging\nvocabulary tied to certainty — analogous to IPCC's.\n\n### 2.6 Evidence-to-Decision (EtD) frameworks\n\nAlonso-Coello et al. (2016), two papers in *BMJ* 353:i2016 and i2089 **[recalled]**. The core move:\n**certainty of evidence and strength of recommendation are different judgements.** Criteria for a\nclinical recommendation: priority of the problem; desirable effects; undesirable effects;\ncertainty of the evidence; values (how much people value the main outcomes, and variability/\nuncertainty about that); balance of effects; resources required; certainty of evidence about\nresources; cost-effectiveness; equity; acceptability; feasibility. Recommendation is **strong** or\n**conditional (weak)**, for or against.\n\nGRADE lists five paradigmatic situations where a strong recommendation on low/very-low certainty\nis justified (e.g., life-threatening situation; low certainty of benefit but high certainty of\nharm) **[recalled]**. In practice this rule is often broken: Alexander et al. (2014, *JCE*) found\nin 43 GRADEd WHO guidelines (456 recommendations) that 63.4% were strong and **55.5% of strong\nrecommendations rested on low or very-low confidence** **[verified]**.\n\n### 2.7 GRADE-CERQual (qualitative evidence)\n\nLewin et al. (2018), *Implementation Science* 13(Suppl 1), series of papers **[verified]**.\n\"Confidence in the Evidence from Reviews of Qualitative research\" assesses confidence in each\n*review finding* (not each study) via four components:\n\n1. **Methodological limitations** of the contributing studies (e.g., CASP-type appraisal).\n2. **Coherence** — how clear and cogent the fit is between the primary data and the review finding\n   (does the finding explain the data, including contradictory cases?).\n3. **Adequacy of data** — richness and quantity of data supporting the finding.\n4. **Relevance** — the extent to which the contributing studies' context matches the review\n   question (analogous to indirectness).\n\nA fifth, **dissemination bias**, is under development. Overall confidence: high / moderate / low /\nvery low, with the default starting point \"high confidence\" and downgrading for concerns — i.e.,\nthe finding is presumed a reasonable representation until concerns appear. **[recalled for the\ndefault]** This is the most directly transferable framework for Deliberus's *positional / impact\ntestimony* (§9).\n\n### 2.8 GRADE for network meta-analysis\n\nPuhan et al. (2014, *BMJ* 349:g5630) and Brignardello-Petersen et al. (2018, *JCE* 93:36–44)\n**[recalled]**: rate the direct, indirect and network estimates separately; indirect estimates\ninherit the lower of the two contributing direct certainties, then may be downgraded for\n**intransitivity**; the network estimate takes the higher of direct/indirect when both contribute\nand may be downgraded for **incoherence** (disagreement between direct and indirect) and\nimprecision. **CINeMA** (Nikolakopoulou et al. 2020, *PLoS Med* 17:e1003082) **[recalled]**\noperationalises six domains: within-study bias, reporting bias, indirectness, imprecision,\nheterogeneity, incoherence — using each study's *contribution matrix* so bias is weighted by how\nmuch each study actually feeds the network estimate. That contribution-weighting is the direct\nanalogue of the platform's missing \"weight answers by what they contribute\".\n\n### 2.9 Criticisms and reliability\n\n- **Reliability.** Mustafa et al. (2013, *JCE* 66:736–742) **[verified]**: IRR of intuitive global\n  quality ratings 0.31 (students) and 0.27 (GRADE members); with GRADE, **0.66 and 0.72**; three or\n  four raters improved it; consensus between paired reviewers did not. Hartling et al. (2012, *PLoS\n  ONE* 7:e34697) **[verified]**, three reviews: risk-of-bias κ 0.41 / 33% agreement / 0.06;\n  consistency κ 0.84 / 0.37 / 0.79; precision κ 0.18 / 0.19 / 0.21; overall 0.44 / 58% / 0.18.\n  Consistency is judged reliably; precision and risk of bias are not.\n- **Kavanagh (2009)**, *PLoS Med* 6:e1000094, \"The GRADE system for rating clinical guidelines\":\n  argues GRADE lacks demonstrated reliability and validity, that its categories are arbitrary, and\n  that combining judgements into one grade hides disagreements. **[recalled]**\n- **Hierarchy smuggled back in.** Stegenga and others note the RCT-high/observational-low starting\n  points retain the pyramid; guideline 18 partly answers this.\n- **Double counting.** Domains overlap (small-study effects count toward both imprecision and\n  publication bias; inconsistency can stem from indirectness); GRADE guidance tells raters not to\n  downgrade twice for one problem — a manual version of the platform's double-counting taxonomy.\n- **Not additive in the arithmetic sense.** Downgrades are summed ordinal steps, capped at \"very\n  low\" — a floor, not a probability. GRADE explicitly refuses to translate levels into numbers.\n\n---\n\n## 3. Risk-of-bias tools: why design alone is insufficient\n\nThe key conceptual step after 2008 was from \"quality\" (reporting and conduct, often summed into\nscores) to **risk of bias** (would this flaw plausibly distort *this result* in a direction?).\nJüni et al. (1999, *JAMA* 282:1054–1060) **[recalled]** showed that different quality *scores*\napplied to the same 17 trials gave opposite conclusions about whether \"high-quality\" trials showed\neffects — summed scales are not valid; domain-based judgements are.\n\n| Tool | Applies to | Domains | Overall judgement |\n|---|---|---|---|\n| **RoB 2** (Sterne et al. 2019, *BMJ* 366:l4898) [recalled] | a specific *result* of a randomised trial | (1) randomisation process; (2) deviations from intended interventions (effect of assignment vs adherence); (3) missing outcome data; (4) measurement of the outcome; (5) selection of the reported result | low / some concerns / high — overall = **worst domain** (and \"high\" if multiple \"some concerns\" substantially lower confidence) |\n| **ROBINS-I** (Sterne et al. 2016, *BMJ* 355:i4919) [recalled] | a result of a non-randomised study of an intervention, judged against a hypothetical *target trial* | 7: confounding; selection of participants; classification of interventions; deviations from intended interventions; missing data; measurement of outcomes; selection of reported result | low / moderate / serious / critical / no information |\n| **ROBINS-I V2** (released Nov 2024, revised 20 Nov 2025) [verified] | same | 6: confounding (ITT and per-protocol variants); classification of intervention; selection into the study; missing data; measurement of outcome; selection of reported result. Adds algorithms mapping signalling-question answers to judgements, \"strong/weak yes/no\" answers, a triage step, explicit immortal-time questions | same scale |\n| **ROBINS-E** (Higgins, Morgan, Rooney, Taylor, Thayer, Silva et al. 2024, *Environment International*) [verified] | non-randomised follow-up studies of *exposures* | 7 domains via signalling questions (confounding, measurement of exposure, selection, post-exposure interventions, missing data, outcome measurement, reported result) [domain list recalled] | low / some concerns / high / very high; also records *predicted direction* of bias |\n| **AMSTAR 2** (Shea et al. 2017, *BMJ* 358:j4008) [recalled] | a systematic review | 16 items, **7 critical**: protocol registered before review; adequate literature search; justification for excluding individual studies; RoB of included studies assessed; appropriate meta-analytic methods; RoB considered when interpreting results; publication bias assessed | high / moderate / low / **critically low** confidence in the review — driven by critical flaws, *not* a sum score |\n| **QUADAS-2** (Whiting et al. 2011, *Ann Intern Med* 155:529–536) [recalled] | diagnostic accuracy studies | 4: patient selection; index test; reference standard; flow and timing — each rated for risk of bias, first three also for **applicability concerns** | per-domain; no summary score |\n| **Newcastle–Ottawa Scale** (Wells et al., c. 2000) [recalled] | cohort and case-control | selection (up to 4 stars), comparability (up to 2), outcome/exposure (up to 3); max 9 | star count — criticised as arbitrary and unreliable (Stang 2010, *Eur J Epidemiol* 25:603–605) |\n\n**Lessons for the ladder.** (1) Every modern tool rates a *result*, not a source. (2) Overall\njudgement is a **worst-domain (min) rule** in RoB 2 and effectively in AMSTAR 2 — the same\nweakest-link logic the platform uses for necessary parts, and for the same reason (one fatal flaw\ncannot be compensated by strengths elsewhere). (3) ROBINS-E and several environmental-health\ntools also record the **predicted direction of bias**, which is what GRADE's third upgrade domain\nand triangulation need. (4) QUADAS-2 separates *bias* from *applicability*, anticipating the\nindirectness-as-edge-property design recommended in §9. (5) Quality *scores* summed across items\nare discredited (Jüni 1999); a single scalar per design is the extreme case of a summed score.\n\n---\n\n## 4. Meta-analysis and synthesis\n\n### 4.1 Models and heterogeneity\n\n- **Fixed (common) effect**: one true effect; inverse-variance weights; the pooled CI reflects only\n  sampling error.\n- **Random effects** (DerSimonian & Laird 1986; now REML / Paule–Mandel estimators and the\n  Hartung–Knapp–Sidik–Jonkman adjustment recommended for few studies) **[recalled]**: true effects\n  vary with between-study variance **τ²**; weights flatten toward equality as τ² grows.\n- **I²** (Higgins & Thompson 2002; Higgins et al. 2003, *BMJ* 327:557) = proportion of observed\n  variance not attributable to sampling error. Cochrane Handbook rough guide: 0–40% might not be\n  important; 30–60% moderate; 50–90% substantial; 75–100% considerable (overlapping by design)\n  **[recalled]**. I² is *relative* — it rises with study precision even when τ² is fixed —\n  so it is not a measure of how much effects vary in absolute terms (Borenstein et al. 2017,\n  *Res Synth Methods* 8:5–18) **[recalled]**.\n- **Prediction interval** (Riley, Higgins & Deeks 2011, *BMJ* 342:d549; IntHout et al. 2016,\n  *BMJ Open* 6:e010247) **[recalled]**: the range in which the true effect in a *new* setting is\n  expected to lie. Often crosses null when the pooled CI does not — the single most important\n  number for \"will it work here?\" and the core of GRADE's updated inconsistency guidance.\n\n### 4.2 Small-study effects and publication bias\n\n- **Funnel plot**; **Egger's regression test** (Egger et al. 1997, *BMJ* 315:629–634) — tests\n  asymmetry, low power with < 10 studies; asymmetry ≠ publication bias (heterogeneity, chance,\n  artefacts can cause it; Sterne et al. 2011, *BMJ* 343:d4002). **[recalled]**\n- **Trim-and-fill** (Duval & Tweedie 2000, *Biometrics*) — imputes \"missing\" studies; known to\n  under-correct under strong selection and to mis-correct under heterogeneity. **[recalled]**\n- **PET-PEESE** (Stanley & Doucouliagos 2014, *Res Synth Methods* 5:60–78) — meta-regression on SE\n  / variance; biased under heterogeneity and few studies. **[recalled]**\n- **Selection models** (Hedges 1984; Vevea & Hedges 1995; Copas) — model the probability of\n  publication as a function of p-value; three-parameter selection model performs relatively well\n  in simulations (Carter et al. 2019, *AMPPS*) — but no method is reliable across all conditions,\n  and Carter et al. recommend reporting several as a *sensitivity analysis*. **[recalled]**\n- **Registries / comparisons with regulatory data** — Turner et al. (2008, *NEJM* 358:252–260):\n  of 74 FDA-registered antidepressant trials, 94% of published trials appeared positive vs 51% by\n  FDA analysis. **[recalled]** The best evidence of publication bias is *direct* evidence of\n  unpublished studies, not statistical inference from asymmetry.\n\n### 4.3 Dependence, networks, IPD, living reviews\n\n- **Dependent effect sizes / robust variance estimation** (Hedges, Tipton & Johnson 2010, *Res\n  Synth Methods* 1:39–65; Tipton 2015 small-sample correction) and three-level / multilevel models.\n  **[recalled]** Handling dependence is the meta-analytic counterpart of Deliberus's shared-source\n  double counting.\n- **Network meta-analysis** — combines direct and indirect comparisons under transitivity;\n  checks consistency (node-splitting, design-by-treatment interaction).\n- **Individual participant data (IPD) MA** (Riley, Lambert & Abo-Zaid 2010, *BMJ* 340:c221)\n  **[recalled]** — reanalyses raw data; enables consistent outcome definitions and subgroup\n  analysis; avoids ecological bias in effect modification.\n- **Living systematic reviews** (Elliott et al. 2014, *PLoS Med* 11:e1001603; 2017 *JCE* series)\n  **[recalled]** — continually updated as evidence appears; the natural institutional analogue of a\n  graph where edges accumulate.\n\n### 4.4 How meta-analyses mislead\n\n- **Garbage in, garbage out**: pooling biased trials produces precise wrong answers. Classic case:\n  the magnesium-for-MI meta-analyses vs ISIS-4; LeLorier et al. (1997, *NEJM* 337:536–542) found\n  meta-analyses failed to predict subsequent large RCTs about 35% of the time. **[recalled]**\n- **Ioannidis (2016)**, *Milbank Quarterly* 94:485–514 **[verified]**: production of SRs/MAs at\n  \"epidemic proportions\"; many redundant, flawed, or conflicted; his rough estimate is that only\n  about **3%** are decent and clinically useful (an order-of-magnitude estimate, not a measurement).\n- **Wu, Duan, Reed & Tipton (2026)**, \"What can we learn from 1,000 meta-analyses across 10\n  different disciplines?\", *Research Synthesis Methods* 17(1):123–156, doi 10.1017/rsm.2025.10035\n  **[verified]**: median 21 studies per MA (Pharmacy median 11, Business/Economics 53.5); ~57% had\n  more than one estimate per study, but of the 571 MAs with dependent effects **fewer than half\n  (47%) adjusted for dependence**; ~80% used random effects; 72% reported I²; 73% assessed\n  publication bias, 31% of those found evidence of it, and **less than half (47%) of those adjusted\n  their estimates**.\n- **Reproducibility** — Nguyen, McKenzie, Alqaidoom et al. (2025, *BMJ Medicine*) **[verified]**:\n  of 121 meta-analyses for which data or code could be obtained, 86% were fully reproducible\n  (<10% difference). **Replicability** of the whole review — Hamilton, McKenzie, Nguyen et al.\n  (2026, *Res Synth Methods* 17(3):538–556) **[verified]**: in 10 replicated reviews, search\n  retrieval differed by >10% in about half of searches; 8/10 meta-analyses were initially \"not fully\n  replicable\" (1/10 meaningfully different), 5/10 still not fully replicable after resolving\n  discrepancies (0/10 meaningfully different); main causes: under-reported eligibility criteria and\n  result-selection methods. The lesson: re-running the arithmetic is reliable; re-making the\n  *choices* is not — Stegenga's malleability, measured.\n\n---\n\n## 5. Causal inference beyond RCTs\n\n### 5.1 Bradford Hill (1965) and reappraisals\n\nHill, \"The environment and disease: association or causation?\", *Proc R Soc Med* 58:295–300. Nine\n**viewpoints** (he refused to call them criteria): strength, consistency, specificity, temporality,\nbiological gradient, plausibility, coherence, experiment, analogy. Hill explicitly: \"None of my nine\nviewpoints can bring indisputable evidence for or against the cause-and-effect hypothesis and none\ncan be required as a *sine qua non*\" (temporality aside), and warned against over-reliance on\nsignificance tests. **[recalled — quotation standard]**\n\nReappraisals: Rothman & Greenland (2005, *AJPH* 95:S144) — only temporality is necessary; the rest\nare heuristics with counterexamples. Howick, Glasziou & Aronson (2009) — regroup into direct,\nmechanistic, parallel evidence (§1.3). Fedak et al. (2015, *Emerging Themes in Epidemiology* 12:14)\n— molecular and genetic data transform \"plausibility\", \"coherence\" and \"experiment\" (e.g., MR as\nexperiment). **[recalled]**\n\n### 5.2 Triangulation\n\nLawlor, Tilling & Davey Smith (2016), *IJE* 45(6):1866–1886 **[verified]**: integrate results from\napproaches with *different and unrelated key sources of bias*; agreement strengthens confidence,\nespecially when the biases of different approaches would push in *opposite* directions.\nTriangulation requires explicitly stating each approach's expected bias direction — which\nis a claim, and arguable. Contrast with *replication* (same design repeated): replication reduces\nchance, not bias; triangulation reduces shared bias. This is the variety-of-evidence point the\nplatform's reconciliation survey already records (\"a known shared source becomes a node\").\n\n### 5.3 Quasi-experimental designs and the credibility revolution\n\n- **Instrumental variables** — valid only under relevance, independence (as-good-as-random), and\n  exclusion restriction; the last is untestable and always an argument.\n- **Regression discontinuity** (Thistlethwaite & Campbell 1960) — local effect at a threshold;\n  checks: no manipulation of the running variable (McCrary density test), covariate balance.\n- **Difference-in-differences** — parallel-trends assumption; staggered-adoption bias in two-way\n  fixed effects (Goodman-Bacon 2021, *J Econometrics*; Callaway & Sant'Anna 2021).\n- **Natural experiments** (Dunning 2012, *Natural Experiments in the Social Sciences*; Craig et al.\n  2012 MRC guidance, *JECH* 66:1182).\n- **Mendelian randomisation** (Davey Smith & Ebrahim 2003, *IJE* 32:1–22) — genetic variants as\n  instruments; threats: pleiotropy, population stratification, weak instruments.\n- **Target trial emulation** (Hernán & Robins 2016, *AJE* 183:758–764) — specify the protocol of\n  the trial you would run (eligibility, strategies, assignment, time zero, follow-up, outcome,\n  causal contrast, analysis) and emulate it; eliminates self-inflicted biases (immortal time,\n  prevalent users). RCT-DUPLICATE (§1.3) shows the payoff depends on emulation closeness.\n- **Credibility revolution**: Angrist & Pischke (2010), *J Econ Perspectives* 24(2):3–30 —\n  \"design-based\" identification made empirical economics credible; 2021 Nobel to Card, Angrist,\n  Imbens. **[recalled]** Critique: Deaton (2010, *JEL*; Deaton & Cartwright 2018, *Soc Sci Med*) —\n  local average treatment effects answer narrow questions; RCTs are not privileged for\n  generalisation.\n\n### 5.4 Quasi-experimental hierarchies in policy\n\n- **Maryland Scientific Methods Scale** (Sherman et al. 1997 report to Congress; Farrington,\n  Gottfredson, Sherman & Welsh 2002 in *Evidence-Based Crime Prevention*) **[verified]**: 1 =\n  correlation at one point in time; 2 = before–after without comparison, or cross-sectional with\n  comparison; 3 = before–after with comparable control (the usual minimum for \"works\"); 4 = multiple\n  units with and without the programme controlling for other factors; 5 = random assignment. The\n  UK What Works Centre for Local Economic Growth uses a modified version. Critique: Hope (2005,\n  \"Pretend it doesn't work: the 'anti-social' bias in the Maryland Scientific Methods Scale\",\n  *Eur J Crim Policy Res*) **[verified title]** — the scale biases against community interventions\n  that cannot be randomised.\n- **What Works Clearinghouse** (US IES), Procedures and Standards Handbook v5.0 (2022)\n  **[verified]**: ratings *Meets WWC Standards Without Reservations* (well-executed RCTs with low\n  attrition; RDDs can also qualify), *Meets With Reservations* (quasi-experiments with baseline\n  equivalence, RCTs with high attrition), *Does Not Meet*. Attrition judged against\n  \"optimistic\" or \"cautious\" boundaries combining overall and differential attrition; v5.0 has the\n  review team choose and justify the boundary.\n- **EEF padlocks** (Classification of the security of findings from EEF evaluations, v2.0, July\n  2019) **[verified]**. Four criteria: design, minimum detectable effect size (MDES), attrition,\n  threats to internal validity. Steps: rate design, MDES and attrition each 0–5; **the interim\n  rating is the lowest of the three**; then adjust *downward* for threats (confounding, concurrent\n  interventions, experimental effects, implementation fidelity, missing data, outcome measurement,\n  selective reporting). Thresholds: design 5 = randomised, 4 = selection on unobservables handled\n  (RDD, DiD), 3 = selection on all relevant observables, 2 = some confounders, 1 = none, 0 = no\n  comparator; MDES 5 ≤ 0.2, 4 = 0.21–0.29, 3 = 0.30–0.39, 2 = 0.40–0.49, 1 = 0.50–0.59, 0 ≥ 0.6;\n  attrition 5 = 0–10%, 4 = 11–20%, 3 = 21–30%, 2 = 31–40%, 1 = 41–50%, 0 > 50%. Rated for the\n  primary outcome only; \"the ratings take no account of whether the intervention itself was\n  successful\" — i.e., certainty is separate from direction. The Toolkit (synthesis level) awards\n  padlocks by number and quality of studies, with **0 padlocks for strands with fewer than 10\n  qualifying studies** **[verified]**, and effects expressed as months of progress.\n  *This is the cleanest real-world weakest-link-then-downgrade system in the policy literature and\n  is directly transferable.*\n- **Nesta Standards of Evidence** (Puttick & Ludlow 2013) **[verified]**: 1 = can describe what you\n  do and why it matters (theory of change); 2 = data showing change; 3 = evidence of causality (a\n  control group); 4 = independent replication / explanation of why it works; 5 = manuals/systems\n  showing it can be replicated at scale with consistent impact. A *developmental* ladder for\n  innovations, explicitly not a quality ranking of individual studies.\n- **Petticrew & Roberts (2003)**, \"Evidence, hierarchies, and typologies: horses for courses\",\n  *JECH* 57:527–529 **[recalled]**: a matrix of research questions (effectiveness, process, salience,\n  safety, acceptability, cost-effectiveness, appropriateness, satisfaction) × designs, with\n  different designs best for different questions — the social-policy version of OCEBM 2011.\n\n---\n\n## 6. Statistics as evidence\n\n### 6.1 p-values and the ASA statements\n\n- **ASA Statement (Wasserstein & Lazar 2016, *Am Stat* 70:129–133)** **[recalled]**, six principles:\n  (1) p-values can indicate how incompatible the data are with a specified model; (2) p-values do\n  not measure the probability that the hypothesis is true or that the data arose by chance alone;\n  (3) scientific conclusions and decisions should not be based only on whether p crosses a\n  threshold; (4) proper inference requires full reporting and transparency; (5) a p-value does not\n  measure the size of an effect or the importance of a result; (6) by itself a p-value does not\n  provide a good measure of evidence regarding a model or hypothesis.\n- **2019 special issue editorial** (Wasserstein, Schirm & Lazar, \"Moving to a world beyond\n  'p < 0.05'\", *Am Stat* 73(S1):1–19): stop saying \"statistically significant\"; \"ATOM\" — accept\n  uncertainty, be thoughtful, open, modest. **[recalled]** Note: this editorial is *not* an ASA\n  policy statement; the **ASA President's Task Force** (Benjamini et al. 2021, *Annals of Applied\n  Statistics* 15:1084–1085) reaffirmed that p-values and significance testing, properly applied,\n  are valid. The field is split. **[recalled]**\n- **Benjamin et al. (2018)**, *Nat Hum Behav* 2:6–10: redefine significance to p < 0.005 for new\n  discoveries. **Lakens et al. (2018)**: justify your alpha instead. **[recalled]**\n- **Bayes-factor bounds**: Sellke, Bayarri & Berger (2001, *Am Stat* 55:62–71) — under broad\n  conditions p = 0.05 corresponds to a Bayes factor against the null of at most about 2.5\n  (−e·p·ln p bound). **[recalled]** A \"significant\" result is weak evidence.\n\n### 6.2 Other evidence measures\n\n- **Confidence intervals** — the preferred reporting unit (effect + precision); used by GRADE's\n  imprecision domain relative to thresholds.\n- **Likelihood ratios** (Royall 1997, *Statistical Evidence: A Likelihood Paradigm*): the *law of\n  likelihood* — data support H1 over H2 to the degree LR > 1; benchmarks 8 (\"fairly strong\") and 32\n  (\"strong\"). Royall's three questions — what should I believe, what should I do, what does this\n  evidence say — map onto certainty, decision, and evidential support respectively. **[recalled]**\n  In diagnosis, LRs are the operational quantity (a test's positive LR multiplies prior odds).\n- **Bayes factors** (Jeffreys; Kass & Raftery 1995, *JASA* 90:773–795) — ratio of marginal\n  likelihoods; depend on priors over parameters.\n- **E-values (betting-based)** — Shafer (2021, *JRSS-A* \"Testing by betting\"); Vovk & Wang (2021,\n  *Annals of Statistics* 49(3):1736–1754); Grünwald, de Heide & Koolen (2024, \"Safe testing\",\n  *JRSS-B* 86(5):1091–1128) **[verified]**. An e-value is a nonnegative statistic with expectation\n  ≤ 1 under the null; Markov's inequality gives type-I control. Properties relevant here: e-values\n  from independent studies **multiply**; e-values for the same hypothesis can be **averaged**\n  (valid even under arbitrary dependence); they remain valid under **optional continuation** (the\n  decision to run a further study may depend on previous results). This is the only mainstream\n  evidence measure whose combination rules are designed for *accumulating, possibly dependent,\n  sequentially-chosen* evidence — the graph's situation.\n- **Naming collision:** VanderWeele & Ding (2017, *Ann Intern Med* 167:268–274) \"E-value\" is a\n  different thing — the minimum strength of association an unmeasured confounder would need with\n  both exposure and outcome to explain away an observed association. Highly relevant for\n  observational claims (it is a quantitative answer to \"could confounding explain this?\") and\n  should not be confused with betting e-values.\n- **Effect sizes and minimal important differences** — Cohen (1988) small/medium/large\n  (d = 0.2/0.5/0.8) are explicitly *fallback* conventions; the MID (Jaeschke, Singer & Guyatt\n  1989, *Controlled Clin Trials* 10:407–415) is \"the smallest difference in score which patients\n  perceive as beneficial\". Statistical significance ≠ practical significance; GRADE's\n  thresholds are MIDs.\n\n### 6.3 Pre-registration, registered reports, forking paths, many analysts\n\n- **Forking paths** (Gelman & Loken 2013/2014, *American Scientist*): multiple comparisons arise\n  from data-contingent analysis choices even without deliberate fishing. **Simmons, Nelson &\n  Simonsohn (2011)**, *Psych Sci* 22:1359–1366: four common researcher degrees of freedom together\n  raised the false-positive rate to about 60.7%. **John, Loewenstein & Prelec (2012)** QRP survey.\n  **[recalled]**\n- **Registered reports** (Chambers 2013, *Cortex*): peer review of the protocol before results.\n  **Scheel, Schijen & Lakens (2021)**, *AMPPS* 4(2): 44% of registered reports vs 96% of standard\n  psychology reports supported their first hypothesis. **[recalled]**\n- **Many analysts**: Silberzahn et al. (2018), *AMPPS* 1(3):337–356 **[verified]** — 29 teams (61\n  analysts), one dataset (do referees give more red cards to dark-skinned players?): odds ratios\n  0.89 to 2.93 (median 1.31); 20 teams (69%) found a significant positive effect, 9 did not; 21\n  unique covariate combinations. Breznau et al. (2022, *PNAS*) — 73 teams on an immigration/welfare\n  question produced effects spanning both signs, with most variance *unexplained* by measured\n  choices. **[recalled]** Consequence: a single study's estimate is a draw from a distribution over\n  defensible analyses (multiverse analysis, Steegen et al. 2016).\n\n### 6.4 Replication crisis numbers\n\n- **Open Science Collaboration (2015)**, *Science* 349:aac4716: 100 psychology replications;\n  36% significant in same direction (vs 97% originals); 47% of original effects within the\n  replication 95% CI; 39% subjectively judged replicated; mean effect size r ≈ 0.20 vs 0.40\n  originally. **[recalled]**\n- **Camerer et al. (2016)**, *Science* 351:1433 — 11 of 18 economics experiments (61%). **[recalled]**\n- **Camerer et al. (2018)**, *Nat Hum Behav* 2:637–644 — 13 of 21 *Science/Nature* social-science\n  experiments (62%); replication effects ~50% of originals **[verified]**; prediction markets\n  anticipated outcomes well.\n- **Many Labs 1** (Klein et al. 2014): 10 of 13 effects replicated **[recalled]**; **Many Labs 2**\n  (Klein et al. 2018, *AMPPS* 1(4):443–490): 14 of 28 **[verified via secondary source]**; little\n  heterogeneity across sites for effects that did not replicate — failures were not mainly\n  \"context\".\n- Note the 2023 *Nat Hum Behav* paper claiming high replicability with rigour-enhancing practices\n  (Protzko et al.) was **retracted** (2024) **[verified: marked retracted]** — a live example of\n  replication status itself needing a status.\n\n---\n\n## 7. Grading evidence outside medicine\n\n### 7.1 IPCC calibrated language\n\nMastrandrea, Field, Stocker, Edenhofer, Ebi, Frame, Held, Kriegler, Mach, Matschoss, Plattner,\nYohe & Zwiers (2010), *Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on\nConsistent Treatment of Uncertainties* (Jasper Ridge meeting, July 2010); journal version\nMastrandrea et al. (2011), *Climatic Change* 108:675–691. Retained for AR6. **[verified for\nprovenance; scale values recalled]**\n\nTwo metrics:\n\n1. **Confidence** (qualitative): from the *type, amount, quality and consistency of evidence*\n   (summary terms **limited / medium / robust**) and the *degree of agreement* (**low / medium /\n   high**). The 3×3 matrix maps to confidence **very low / low / medium / high / very high**,\n   increasing toward robust-evidence-high-agreement. Authors may also report \"evidence and\n   agreement\" statements directly without assigning confidence.\n2. **Likelihood** (quantitative, when probabilistic assessment is possible): virtually certain\n   99–100%; extremely likely 95–100%; very likely 90–100%; likely 66–100%; more likely than not\n   >50–100%; about as likely as not 33–66%; unlikely 0–33%; very unlikely 0–10%; extremely unlikely\n   0–5%; exceptionally unlikely 0–1%.\n\nRules: likelihood statements implicitly carry high or very high confidence; a finding can be\nassigned confidence without likelihood; low and very low confidence findings are reported only\nfor areas of major concern. The two scales are **orthogonal by design** — confidence is about the\n*evidence base and expert agreement*, likelihood is about *the outcome*.\n\nKnown problems: Budescu et al. (2009, *Psych Sci*; 2014 *Nature Climate Change*) found readers\ninterpret \"very likely\" etc. much less extremely than intended, regressing toward 50%; presenting\nnumeric ranges alongside words improves calibration. **[recalled]** \"Agreement\" in IPCC means\nagreement *among the literature and authors*, assessed deliberatively — not a vote.\n\n### 7.2 Intelligence community (brief — covered elsewhere)\n\nICD 203 (2015) **[recalled]**: seven likelihood terms with ranges (almost no chance 1–5%, very\nunlikely 5–20%, unlikely 20–45%, roughly even chance 45–55%, likely 55–80%, very likely 80–95%,\nalmost certain 95–99%) and a *separate* analytic confidence level (low/moderate/high) reflecting\nsource quality and reasoning. Same two-axis structure as IPCC.\n\n### 7.3 Education, policy, environment, software\n\n- **EEF / UK What Works Network** (launched 2013; ~10 What Works Centres, e.g., EEF, NICE, College\n  of Policing, Early Intervention Foundation) — see §5.4.\n- **Collaboration for Environmental Evidence (CEE)** Guidelines and Standards for Evidence\n  Synthesis in Environmental Management (v5.1, 2022) and **CEESAT** for rating syntheses\n  **[recalled]**; environmental evidence emphasises critical appraisal per study for *internal* and\n  *external* validity and systematic maps (which catalogue evidence without synthesising it — an\n  explicit \"what exists\" layer).\n- **Environmental health**: NTP/OHAT Handbook (Rooney et al. 2014, *EHP* 122:711; 2015/2019\n  handbook) **[recalled]** adapts GRADE: initial confidence set by *study-design features*\n  (controlled exposure, exposure before outcome, individual outcome data, comparison group) rather\n  than design names; rates confidence high/moderate/low/very low; translates to level of evidence\n  (high / moderate / low / evidence of no health effect / inadequate), then integrates human and\n  animal streams into hazard conclusions (known / presumed / suspected / not classifiable / not\n  identified). The **Navigation Guide** (Woodruff & Sutton 2014, *EHP* 122:1007) is similar.\n  Starting by *features* instead of *labels* is a concrete alternative to a design ladder.\n- **Software engineering**: Kitchenham, Dybå & Jørgensen (2004, ICSE) \"Evidence-based software\n  engineering\"; Kitchenham & Charters (2007) SLR guidelines **[recalled]**. SE adopted medical\n  hierarchies, then found RCTs rare and external validity dominant; later work (e.g., Wohlin; the\n  \"grey literature\" debate, Garousi et al. 2019) argues for multivocal reviews including practitioner\n  evidence — an example of a field whose ladder had no rung for its most abundant evidence.\n\n### 7.4 The social-policy critique of hierarchies\n\nBeyond Cartwright and Petticrew & Roberts: Pawson (2006, *Evidence-Based Policy: A Realist\nPerspective*) — \"what works for whom, in what circumstances, and why\" (realist synthesis, RAMESES\nstandards 2013); interventions are theories; effect averages hide mechanism × context\nconfigurations. **[recalled]** Also \"evidence-based\" vs \"evidence-informed\" and the charge that\nhierarchies privilege what is measurable.\n\n---\n\n## 8. Aggregating heterogeneous kinds of evidence\n\n### 8.1 Mixed-methods and cross-design synthesis\n\n- Convergent, sequential and results-based designs for combining quantitative and qualitative\n  syntheses (Noyes et al. 2019, *BMJ Global Health* 4:e000893, WHO complex-interventions series;\n  Cochrane Handbook ch. 21) **[recalled]**; logic models to link them; qualitative evidence explains\n  heterogeneity, acceptability and feasibility — feeding EtD criteria, not the effect estimate.\n- **GRADE across bodies**: when both RCT and NRS bodies exist, rate each; generally rely on the\n  higher-certainty body; NRS can complement (longer follow-up, rare harms, indirectness of RCT\n  populations) or replace. Cuello-Garcia et al. (2022), GRADE guidance 24, *JCE*, \"Optimizing the\n  integration of randomized and non-randomized studies of interventions\" **[unverified details]**.\n  Overall certainty across *outcomes* is the **lowest certainty among critical outcomes** (another\n  weakest-link rule) **[recalled]**.\n\n### 8.2 Weight of evidence in toxicology and regulation\n\n- **EFSA Scientific Committee (2017)**, \"Guidance on the use of the weight of evidence approach in\n  scientific assessments\", *EFSA Journal* 15(8):4971 **[verified]**: weight of evidence is \"a\n  process in which evidence is integrated to determine the relative support for possible answers\n  to a question\"; three steps — **assemble** evidence into lines of evidence of similar type,\n  **weigh** each (three basic considerations: **reliability, relevance, consistency**), **integrate**;\n  qualitative or quantitative (including Bayesian) methods allowed, with uncertainty analysis\n  mandatory (EFSA uncertainty guidance 2018).\n- **US EPA**: Guidelines for Carcinogen Risk Assessment (2005) — weight-of-evidence narrative\n  with five descriptors: *carcinogenic to humans; likely to be carcinogenic; suggestive evidence of\n  carcinogenic potential; inadequate information to assess; not likely to be carcinogenic*\n  **[recalled]**; EPA Weight of Evidence in Ecological Assessment (2016). The US National Academies\n  (2014) review of IRIS pushed EPA toward systematic-review-based integration **[recalled]**.\n- **IARC Monographs Preamble (amended January 2019)** **[verified]**. Title changed from\n  \"Evaluation of carcinogenic *risks*\" to \"Identification of carcinogenic *hazards*\"; classifications\n  relate to the **strength of evidence that an agent is a hazard, not the magnitude of risk** at\n  given exposures. Three evidence streams: **cancer in humans**, **cancer in experimental animals**,\n  **mechanistic evidence** (organised by the **ten key characteristics of carcinogens**, Smith et al.\n  2016, *EHP* 124:713 **[recalled]**). Each stream is rated (humans/animals: sufficient, limited,\n  inadequate, evidence suggesting lack of carcinogenicity; mechanistic: strong, limited,\n  inadequate) and integrated into **Group 1** (carcinogenic to humans), **Group 2A** (probably),\n  **Group 2B** (possibly), **Group 3** (not classifiable). **Group 4** (\"probably not carcinogenic\")\n  was **removed** in 2019 (its sole member, caprolactam, had moved to Group 3) **[verified]**.\n  Group 1 is reachable with sufficient human evidence, *or* with less-than-sufficient human evidence\n  plus sufficient animal evidence plus strong mechanistic evidence in exposed humans **[verified]**.\n  Lessons: (a) multiple streams are *combined by rules* rather than ranked on one ladder; (b) hazard\n  (can it?) is separated from risk (how much, at what exposure?) — processed meat and plutonium\n  share Group 1 because both are established hazards, not because the risks are comparable, a\n  distinction that is routinely lost in public communication; (c) Group 3 is not a negative\n  finding — \"not classifiable\" is IARC's explicit *this is open* status, the category the\n  platform's epistemic-status enum lacks.\n\n### 8.3 The recurring structure\n\nAcross GRADE, IPCC, EFSA, IARC, EEF and OHAT, the same four moves recur:\n\n1. **Separate lines/streams** of evidence by type (RCT vs NRS; human/animal/mechanistic;\n   quantitative/qualitative).\n2. **Appraise each** on a small set of named dimensions (bias/reliability; relevance/indirectness;\n   precision/quantity; consistency/coherence; reporting bias).\n3. **Integrate** by explicit rules that are mostly *ordinal*, often weakest-link or step-down, and\n   that let strong lines compensate for weak ones only in specified ways (IARC's routes to\n   Group 1).\n4. **Report certainty separately** from the direction/size of the effect and from the decision.\n\nNo mature framework produces a single cardinal number from design type.\n\n---\n\n## 9. Assessment of the platform's methodology ladder\n\nThe ladder in code: meta-analysis 0.70, systematic review 0.68, RCT 0.65, quasi-experimental 0.55,\nobservational 0.45, case study 0.38, anecdotal 0.32, expert opinion 0.30; assigned per *source*,\ninert (heterogeneity and study count carry no weight).\n\n### 9.1 Is a single ladder defensible? No — on six independent grounds\n\n1. **Question type.** The ladder is the *therapy/intervention* ladder. For prevalence, prognosis,\n   diagnosis, rare harms, mechanism or meaning, it is wrong in order, not just in magnitude\n   (OCEBM 2011; Petticrew & Roberts 2003). A local random survey outranks an RCT on \"how common is\n   X\"; a qualitative study outranks an RCT on \"why do people refuse X\".\n2. **Unit.** Every current tool grades a *result for an outcome* (RoB 2, ROBINS-I, EEF) or a *body\n   of evidence for an outcome* (GRADE, CERQual). The platform grades a *source by design*, the\n   granularity abandoned after Jüni (1999). The founder's per-claim instinct is exactly the\n   literature's.\n3. **Design is a starting point, not a verdict.** GRADE, OCEBM's footnote, and guideline 18 all\n   treat design as the prior that domains then move. A well-run observational study with a dramatic\n   effect sits at OCEBM step 2; a biased RCT can be very-low certainty.\n4. **Meta-analysis is not a tier.** It is a *reading of* the tier below (the \"lens\" pyramid), and its\n   certainty is the certainty of its inputs adjusted for inconsistency, imprecision and reporting\n   bias. Ranking it above RCTs by fixed margin (0.70 vs 0.65) double-counts the RCTs inside it and\n   ignores that roughly half of meta-analyses do not handle dependence or adjust for detected\n   publication bias (Wu et al. 2026), and that the re-made choices are not replicable (Hamilton et\n   al. 2026; Stegenga 2011).\n5. **Expert opinion is a category error on this axis** (Schünemann et al. 2019). Walton's\n   expert-opinion scheme already represents appeals to authority as a separate inference whose CQs\n   (expertise, domain, consistency with other experts, backing evidence) are the right appraisal.\n6. **Mechanistic evidence and qualitative evidence have no slot**, yet IARC and CERQual show both\n   are evidence streams with their own appraisal criteria.\n\nA seventh, internal to the platform: a cardinal number (0.45 vs 0.55) implies interval-scale\nmeaning that no framework claims; GRADE explicitly refuses numeric translation. And the numbers are\n*uncited house choices* — by the corpus's own sorting test, a commitment that two users could\ndisagree about belongs in the graph, not the ontology.\n\n**What survives of the ladder:** as a *default prior for the starting certainty of an\nintervention-effect claim*, visibly labelled as such and overridable by argued domain claims — which\nis what GRADE's RCT-high/NRS-low convention is.\n\n### 9.2 What should replace it: a GRADE-shaped structure in graph form\n\n**Three nodes, not one.**\n\n- **Study-result claim** (\"Trial T found RR 0.78 [0.65, 0.93] for outcome O in population P\").\n  Empirical, `observed`, carries a locator. Its *internal validity* is appraised by RoB-type CQs on\n  the claim itself (randomisation, deviations, missing data, measurement, selective reporting — or\n  ROBINS-I's confounding and selection for NRS).\n- **Body-of-evidence claim** (\"Across studies, intervention I reduces O in P\", or \"…the effect\n  exceeds the MID\") — the mother claim. Its children are the study-result claims, linked as\n  **corroborative** (never necessary), which is also the fix for the 70 misfiled decomposition\n  edges. The body claim carries the GRADE domains that belong to *bodies*: inconsistency,\n  imprecision, publication bias, and the upgrade domains.\n- **Target claim** (the claim someone is actually arguing: \"we should adopt I in Sweden\"). Linked\n  from the body claim by a SUPPORTS edge whose CQs carry **indirectness** (PICO mismatch) and, for\n  policy, the Cartwright \"will it work here\" support-factor questions. Indirectness is inherently an\n  *edge* property: the same body can be direct for one target and indirect for another.\n\n**Certainty as its own axis, derived.** Keep *direction* (which way) and *weight* (how much examined\ninput) as the badge already does, and add **certainty** as an ordinal reading (high / moderate / low\n/ very low / not yet assessed) computed from the domain claims: start at a stated prior, step down\nfor each domain claim judged serious/very serious, step up for upgrade claims — each step pointing\nat the claim that justifies it, so \"why is this moderate?\" is a graph query, like GRADE's footnotes.\nCertainty is always *relative to a threshold claim* (Hultcrantz 2017): \"certainty that the effect\nexceeds 0.2 SD\" differs from \"certainty that there is any effect\". The platform's \"read, never set\"\nhorizon principle applies: certainty is a *reading*, recomputed when domain claims change.\n\n**Per-question-type starting points.** A small table keyed by the claim's question type\n(intervention effect / harm / prevalence / prognosis / diagnostic accuracy / mechanism /\nexperience-and-meaning) giving the default starting certainty per design, following OCEBM 2011 and\nPetticrew & Roberts. Better still (OHAT): start from **design features** — was assignment\ncontrolled? did exposure precede outcome? is there a comparison group? are outcomes individual-level?\n— each a claim that can be answered and attacked, rather than a label.\n\n### 9.3 Mapping the domains to critical questions\n\n| GRADE / appraisal domain | CQ, phrased for the graph | Where it attaches | What kind of answer counts |\n|---|---|---|---|\n| Risk of bias (per result) | \"Was allocation random and concealed?\" \"Could deviations from intended intervention bias the result?\" \"Is outcome data missing in a way related to outcome?\" \"Could outcome measurement be influenced by knowledge of assignment?\" \"Was the reported result selected from several?\" (RoB 2); for NRS add confounding, selection, classification (ROBINS-I) | study-result claim | a domain judgement with its reason and **predicted direction of bias**; weighed by the study's contribution to the body |\n| Inconsistency | \"Do the study results agree more than chance would allow?\" \"Does the prediction interval cross the decision threshold?\" \"Is heterogeneity explained by a named modifier?\" | body claim | τ², I², prediction interval as claims with values; an explanatory subgroup claim can *resolve* inconsistency (and becomes its own claim) |\n| Imprecision | \"Does the CI cross the threshold?\" \"Is the total information below the optimal information size?\" | body claim | the CI and the OIS computation; this is the formal content of Keynes's weight |\n| Indirectness | \"Does the evidence's population/intervention/comparator/outcome match the target's?\" \"Is the outcome a surrogate?\" \"Is the comparison indirect?\" | SUPPORTS edge body → target | PICO-mismatch claims; the same body can be direct for one target, indirect for another |\n| Publication / reporting bias | \"Were trial registries searched and are registered trials missing?\" \"Is there small-study asymmetry?\" \"Is the body dominated by small sponsor-funded trials?\" | body claim | *search claims* (\"we searched X, Y, Z on date D and found N unpublished trials\") — the missing \"we searched and found nothing\" object |\n| Large effect (up) | \"Is the effect large enough (e.g., RR > 2) that plausible confounding cannot explain it?\" | body claim | effect-size claim + confounding-strength argument (VanderWeele–Ding E-value is a ready quantitative answer) |\n| Dose-response (up) | \"Does the effect increase with dose/exposure?\" | body claim | gradient claim (Hill's biological gradient) |\n| Opposing confounding (up) | \"Would the plausible biases push the estimate *toward* the null?\" | body claim | a bias-direction argument — the triangulation logic |\n| Coherence / mechanism (EBM+, IARC) | \"Is there independent evidence of a mechanism linking I to O?\" | separate mechanistic stream, linked corroboratively to the body claim | mechanistic study claims with their own appraisal |\n| Qualitative findings (CERQual) | methodological limitations; coherence; adequacy (richness, quantity); relevance | review-finding claim | appraisal claims per component |\n| Replication / retraction status | \"Has this result been independently replicated? Retracted? Corrected?\" | study-result claim | dated status claims with sources |\n| Funding / conflict | \"Was the study funded by a party with a stake in the outcome?\" | study-result or source | attribution claims (a *bias* scheme instance, not a downgrade by itself — GRADE treats COI through RoB and publication bias) |\n\n**Critical point: answers must be weighed, not counted.** The platform's current rule (+0.15 per\nanswered CQ) is structurally **vote counting**. Hedges & Olkin (1980, *Psych Bull* 88:359–369)\n**[recalled]** proved that tallying significant vs non-significant studies can have power that\n*decreases* toward zero as the number of studies increases when individual studies are\nunderpowered — i.e., counting can converge on the wrong answer. GRADE, CINeMA and meta-analysis all\nweight by *what each input contributes* (inverse variance; contribution matrix) and judge each\ndomain by *content and severity* (serious vs very serious). In graph terms: an answer to a RoB CQ\nshould carry a *severity* and a *direction*, and should itself be a claim that can be attacked; its\neffect on the edge should depend on the severity claim's own standing, not on the answer's\nexistence. Three one-word answers from one account should move nothing (bare judgement =\ntelemetry), which is consistent with the founder's own principle.\n\n### 9.4 Certainty of a body vs strength of a single study\n\n- **Strength of a single study** = internal validity of *one result* (RoB judgement) × its\n  **precision** (its weight in the body). These are two different things; a large biased trial is\n  precise and wrong. The platform should never collapse them into one scalar per source.\n- **Certainty of a body** = a reading over the set of results *for a given question and threshold*,\n  determined by RoB (weighted by contribution), inconsistency, imprecision, indirectness to the\n  target, and reporting bias, with upgrades. It is not the average of study strengths, and it is\n  not monotone in the number of studies (adding biased or redundant studies can leave certainty\n  unchanged or lower it; adding one large clean trial can raise it by two levels).\n- **Weight (Keynes)** ≈ GRADE's imprecision + OIS + the adequacy component of CERQual. Keynes's\n  point that weight can rise while probability falls is exactly the separation of certainty from\n  direction; the badge's non-saturating count-of-inputs weight is not yet this — it should saturate\n  and should count *independent information* (effective sample size, or number of independent\n  lines), not inputs.\n- **Shared origin**: robust variance estimation (studies from one lab, one dataset, one cohort)\n  and the many-analysts results say: *several analyses of one dataset are one line of evidence*.\n  Represent the dataset as a node; studies share it; certainty computations treat them as\n  dependent (the corpus's \"a known shared source becomes a node\" is exactly right and has direct\n  meta-analytic warrant).\n\n### 9.5 Separating evidence from decision\n\nDeliberus declines verdicts; GRADE's EtD shows the useful shape of what it *can* map without\nissuing one: the certainty reading is one input; values and preferences, resource use, equity,\nacceptability and feasibility are separate claims (many normative or positional). The WHO finding\n(55.5% of strong recommendations on low/very-low certainty) shows that decisions routinely outrun\nevidence certainty — which is legitimate when other EtD claims justify it and illegitimate when they\ndo not; only a graph that holds both makes the difference inspectable. IARC's hazard/risk split is\nthe same move for factual claims: \"can it cause cancer\" and \"how much risk at this exposure\" are\ndifferent claims with different evidence.\n\n---\n\n## 10. Consensus vs contested — summary\n\n| Settled (broad consensus across frameworks) | Contested |\n|---|---|\n| Design alone does not determine evidential quality; appraise results by domains | Whether RCTs deserve a starting premium over well-designed NRS (GRADE 18 vs tradition; Worrall; Deaton) |\n| Question type determines which designs are best | How to weigh mechanistic evidence (EBM+ vs Howick) |\n| Certainty is separate from effect size and from recommendation | Whether ordinal grades should become numbers (GRADE refuses; Bayesian WoE accepts) |\n| Summed quality scores are invalid | The status of p-values (2019 editorial vs 2021 task force) |\n| Heterogeneity must be judged by prediction interval / threshold, not I² alone | Which publication-bias correction to trust (none robust; report several) |\n| Publication bias is real and large in some fields | Whether meta-analysis is \"platinum\" or malleable (Stegenga) |\n| Replication rates in social science are roughly 35–65% | How far GRADE-style grading is reliable (0.66–0.72 overall, poor on imprecision/RoB) |\n| Hazard ≠ risk (IARC) | Whether grading systems privilege the measurable (realist and social-policy critiques) |\n\n---\n\n## 11. Most transferable to a claim-graph argumentation platform like Deliberus\n\nConcrete, in rough priority order.\n\n1. **Retire the per-source design ladder as a strength input; keep it only as a labelled default\n   starting certainty for intervention-effect claims.** Record it as a visible, attackable claim\n   (\"default: RCT bodies start high\"), per the sorting test. Delete *expert opinion* from the ladder\n   and route it through the Walton expert-opinion scheme; route \"expert experience\" through the\n   testimony tiers as observational evidence.\n\n2. **Introduce a `question_type` on empirical claims** (intervention effect / harm / prevalence /\n   prognosis / diagnostic accuracy / mechanism / experience-and-meaning), selecting the default\n   starting certainty per design (OCEBM 2011 table as the seed). Better: start from *design\n   features* (OHAT), each a claim.\n\n3. **Adopt the three-node pattern: study-result claim → body-of-evidence claim → target claim.**\n   Study results are **corroborative** children of the body claim (never necessary/min — which\n   also reclassifies the 70 misfiled \"three studies are witnesses\" edges). RoB CQs live on study\n   results; inconsistency, imprecision, reporting-bias and upgrade CQs live on the body; indirectness\n   lives on the body→target edge. A meta-analysis PDF decomposes into exactly this: its included\n   study results, its pooled estimate, its heterogeneity and bias claims — the founder's \"maximum\n   granularity\" with a known shape.\n\n4. **Add a certainty reading as a third badge axis**, beside direction and weight, computed GRADE-\n   style (start, step down per serious domain, step up per upgrade), always relative to a threshold\n   claim, recomputed on change, with each step linked to its justifying claim. Never display a\n   number; display the four words plus \"not yet assessed\". Borrow GRADE's plain-language verbs\n   (\"probably\", \"may\", \"uncertain whether\") for surfaces.\n\n5. **Weigh CQ answers, never count them.** Each domain answer carries *severity* (none / serious /\n   very serious) and *predicted bias direction*, is itself a claim, and moves the edge only through\n   its own standing. Cite Hedges & Olkin 1980 as the reason: vote counting is a known failure. For\n   bodies, weight studies by contribution (inverse variance or CINeMA-style contribution), not one\n   vote each.\n\n6. **Make \"we searched and found nothing\" a first-class claim type (search claim)**: databases,\n   registries, dates, query, result count, and unpublished-registered trials found. It is the\n   evidence for the publication-bias domain and the only defence of an absence claim.\n\n7. **Represent datasets and analyses explicitly.** A dataset (or cohort, or lab) is a node; study\n   results that share it are dependent and count as one line for weight. For contested single\n   results, allow a \"multiverse/many-analysts\" claim (\"reasonable analyses range from OR 0.89 to\n   2.93\") — Silberzahn gives the template.\n\n8. **Give mechanistic evidence its own stream**, linked corroboratively to the body claim, with\n   EBM+-style appraisal CQs, and an IARC-style integration rule stating when mechanism plus animal\n   or correlational evidence suffices. Do not place it at the bottom of a ladder.\n\n9. **Use GRADE-CERQual's four components for positional / impact testimony**: methodological\n   limitations (how was the testimony gathered?), coherence (does the finding fit all the\n   testimony, including dissent?), adequacy (richness and number of independent testifiers),\n   relevance (does the testifiers' position match the claim's scope?). This gives the proposed\n   testimony tiers a published, reviewed backbone and keeps \"sole-witness\" as an adequacy\n   judgement rather than a gate.\n\n10. **Add an explicit \"open / not classifiable\" epistemic status** modelled on IARC Group 3 and\n    OHAT \"inadequate\" — distinct from \"no effect\" (OHAT \"evidence of no health effect\") — so the\n    graph can say *the evidence does not decide this* without implying a negative.\n\n11. **Separate hazard from risk, and certainty from decision.** For factual claims, split \"can X\n    cause Y\" from \"how much, at what exposure\". For decision-shaped targets, lay out EtD criteria\n    (values, resources, equity, acceptability, feasibility) as sibling claims, so a strong\n    conclusion on weak evidence is visibly justified or visibly not.\n\n12. **Status claims with timestamps**: replication, retraction, correction, registered-report\n    status, funding source — each dated and attributed. The retracted 2023 replicability paper is\n    the anchor case.\n\n13. **Treat grading as contested ground.** Inter-rater reliability for imprecision and RoB can be\n    near chance (κ 0.06–0.21), and FRI's adversarial collaboration traced residual disagreement to\n    different evidence standards. Machine-proposed domain judgements stay proposals; disagreement\n    between two raters' domain claims is itself informative (it locates *which* standard differs),\n    and is exactly the kind of mapped disagreement the platform exists to surface.\n\n14. **If numbers are ever wanted, use evidence measures with principled combination rules**:\n    likelihood ratios for diagnostic-shaped claims; betting e-values for accumulating, possibly\n    dependent evidence (multiply when independent, average under arbitrary dependence, valid under\n    optional continuation); VanderWeele–Ding E-values as the quantitative answer to \"could\n    confounding explain this?\". None of these is a design-tier weight.\n\n15. **Calibrated language on surfaces, with numbers alongside when probabilistic** (IPCC/ICD 203),\n    because readers systematically misread verbal probability terms (Budescu).\n\n---\n\n## References\n\n(Accessed 2026-09-26 where a URL is given.)\n\n- Alexander PE, Bero L, Montori VM, et al. (2014). World Health Organization recommendations are often strong based on low confidence in effect estimates. *J Clin Epidemiol* 67(6):629–634. https://pubmed.ncbi.nlm.nih.gov/24388966/\n- Alonso-Coello P, Schünemann HJ, Moberg J, et al. (2016). GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. *BMJ* 353:i2016; 2: Clinical practice guidelines. *BMJ* 353:i2089.\n- Angrist JD, Pischke J-S (2010). The credibility revolution in empirical economics. *J Econ Perspect* 24(2):3–30.\n- Anglemyer A, Horvath HT, Bero L (2014). Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. *Cochrane Database Syst Rev* MR000034. [numbers unverified today]\n- Balshem H, Helfand M, Schünemann HJ, et al. (2011). GRADE guidelines: 3. Rating the quality of evidence. *J Clin Epidemiol* 64(4):401–406.\n- Benjamin DJ, Berger JO, et al. (2018). Redefine statistical significance. *Nat Hum Behav* 2:6–10.\n- Benjamini Y, De Veaux RD, Efron B, et al. (2021). ASA President's Task Force Statement on Statistical Significance and Replicability. *Ann Appl Stat* 15(3):1084–1085.\n- Benson K, Hartz AJ (2000). A comparison of observational studies and randomized, controlled trials. *NEJM* 342:1878–1886.\n- Bluhm R (2005). From hierarchy to network: a richer view of evidence for evidence-based medicine. *Perspect Biol Med* 48(4):535–547.\n- Borenstein M, Higgins JPT, Hedges LV, Rothstein HR (2017). Basics of meta-analysis: I² is not an absolute measure of heterogeneity. *Res Synth Methods* 8(1):5–18.\n- Brignardello-Petersen R, Bonner A, Alexander PE, et al. (2018). Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. *J Clin Epidemiol* 93:36–44.\n- Budescu DV, Por H-H, Broomell SB, Smithson M (2014). The interpretation of IPCC probabilistic statements around the world. *Nature Climate Change* 4:508–512.\n- Camerer CF, Dreber A, et al. (2016). Evaluating replicability of laboratory experiments in economics. *Science* 351:1433–1436.\n- Camerer CF, Dreber A, Holzmeister F, et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. *Nat Hum Behav* 2:637–644. https://www.nature.com/articles/s41562-018-0399-z\n- Canadian Task Force on the Periodic Health Examination (1979). The periodic health examination. *CMAJ* 121:1193–1254.\n- Carter EC, Schönbrodt FD, Gervais WM, Hilgard J (2019). Correcting for bias in psychology: a comparison of meta-analytic methods. *AMPPS* 2(2):115–144.\n- Cartwright N (2007). Are RCTs the gold standard? *BioSocieties* 2:11–20.\n- Cartwright N, Hardie J (2012). *Evidence-Based Policy: A Practical Guide to Doing It Better*. Oxford University Press.\n- Chambers CD (2013). Registered Reports: a new publishing initiative at Cortex. *Cortex* 49(3):609–610.\n- Clarke B, Gillies D, Illari P, Russo F, Williamson J (2014). Mechanisms and the evidence hierarchy. *Topoi* 33:339–360.\n- Collaboration for Environmental Evidence (2022). *Guidelines and Standards for Evidence Synthesis in Environmental Management*, v5.1. https://environmentalevidence.org/information-for-authors/\n- Concato J, Shah N, Horwitz RI (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. *NEJM* 342:1887–1892.\n- Davey Smith G, Ebrahim S (2003). 'Mendelian randomization': can genetic epidemiology contribute to understanding environmental determinants of disease? *Int J Epidemiol* 32:1–22.\n- Deaton A, Cartwright N (2018). Understanding and misunderstanding randomized controlled trials. *Soc Sci Med* 210:2–21.\n- Duval S, Tweedie R (2000). Trim and fill. *Biometrics* 56:455–463.\n- Education Endowment Foundation (2019). *Classification of the security of findings from EEF evaluations*, v2.0. https://d2tic4wvo1iusb.cloudfront.net/documents/evaluation/peer-review-process/Classifying_the_security_of_EEF_findings_2019.pdf\n- EFSA Scientific Committee (2017). Guidance on the use of the weight of evidence approach in scientific assessments. *EFSA Journal* 15(8):4971. https://efsa.onlinelibrary.wiley.com/doi/10.2903/j.efsa.2017.4971\n- Egger M, Davey Smith G, Schneider M, Minder C (1997). Bias in meta-analysis detected by a simple, graphical test. *BMJ* 315:629–634.\n- Elliott JH, Turner T, Clavisi O, et al. (2014). Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. *PLoS Med* 11(2):e1001603.\n- Farrington DP, Gottfredson DC, Sherman LW, Welsh BC (2002). The Maryland Scientific Methods Scale. In Sherman et al. (eds.), *Evidence-Based Crime Prevention*, 13–21. Routledge. https://www.ojp.gov/ncjrs/virtual-library/abstracts/maryland-scientific-methods-scale-evidence-based-crime-prevention-p\n- Fedak KM, Bernal A, Capshaw ZA, Gross S (2015). Applying the Bradford Hill criteria in the 21st century. *Emerg Themes Epidemiol* 12:14.\n- Gelman A, Loken E (2014). The statistical crisis in science. *American Scientist* 102(6):460.\n- GRADE Working Group (2004). Grading quality of evidence and strength of recommendations. *BMJ* 328:1490.\n- Grünwald P, de Heide R, Koolen W (2024). Safe testing. *JRSS-B* 86(5):1091–1128. https://academic.oup.com/jrsssb/article-abstract/86/5/1091/7623686\n- Guyatt GH, Oxman AD, Vist GE, et al. (2008). GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. *BMJ* 336:924–926.\n- Guyatt GH, et al. (2011). GRADE guidelines 4 (risk of bias), 5 (publication bias), 6 (imprecision), 7 (inconsistency), 8 (indirectness), 9 (rating up). *J Clin Epidemiol* 64 (various pages).\n- Hamilton DG, McKenzie JE, Nguyen P-Y, et al. (2026). Evaluation of the replicability of systematic reviews with meta-analyses of the effects of health interventions. *Res Synth Methods* 17(3):538–556. doi:10.1017/rsm.2025.10064\n- Hartling L, Fernandes RM, Seida J, et al. (2012). From the trenches: a cross-sectional study applying the GRADE tool in systematic reviews of healthcare interventions. *PLoS ONE* 7(4):e34697. https://pmc.ncbi.nlm.nih.gov/articles/PMC3320617/\n- Hedges LV, Olkin I (1980). Vote-counting methods in research synthesis. *Psychol Bull* 88(2):359–369.\n- Hedges LV, Tipton E, Johnson MC (2010). Robust variance estimation in meta-regression with dependent effect size estimates. *Res Synth Methods* 1:39–65.\n- Hernán MA, Robins JM (2016). Using big data to emulate a target trial when a randomized trial is not available. *Am J Epidemiol* 183(8):758–764.\n- Higgins JPT, Thompson SG, Deeks JJ, Altman DG (2003). Measuring inconsistency in meta-analyses. *BMJ* 327:557–560.\n- Higgins JPT, Morgan RL, Rooney AA, et al. (2024). A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E). *Environment International*. https://pubmed.ncbi.nlm.nih.gov/38555664/\n- Hill AB (1965). The environment and disease: association or causation? *Proc R Soc Med* 58:295–300.\n- Hope T (2005). Pretend it doesn't work: the 'anti-social' bias in the Maryland Scientific Methods Scale. *Eur J Crim Policy Res* 11:275–296.\n- Howick J (2011). *The Philosophy of Evidence-Based Medicine*. Wiley-Blackwell/BMJ Books.\n- Howick J, Glasziou P, Aronson JK (2009). The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute? *J R Soc Med* 102:186–194.\n- Hultcrantz M, Rind D, Akl EA, et al. (2017). The GRADE Working Group clarifies the construct of certainty of evidence. *J Clin Epidemiol* 87:4–13.\n- IARC (2019). *Preamble to the IARC Monographs* (amended January 2019). https://monographs.iarc.who.int/wp-content/uploads/2019/07/Preamble-2019.pdf ; Samet JM et al. (2020). The IARC Monographs: updated procedures for modern and transparent evidence synthesis in cancer hazard identification. *JNCI* 112(1):30–37.\n- IntHout J, Ioannidis JPA, Rovers MM, Goeman JJ (2016). Plea for routinely presenting prediction intervals in meta-analysis. *BMJ Open* 6:e010247.\n- Ioannidis JPA (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. *Milbank Q* 94(3):485–514. https://www.milbank.org/quarterly/articles/mass-production-redundant-misleading-conflicted-systematic-reviews-meta-analyses/\n- John LK, Loewenstein G, Prelec D (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. *Psych Sci* 23(5):524–532.\n- Jüni P, Witschi A, Bloch R, Egger M (1999). The hazards of scoring the quality of clinical trials for meta-analysis. *JAMA* 282:1054–1060.\n- Kavanagh BP (2009). The GRADE system for rating clinical guidelines. *PLoS Med* 6(9):e1000094.\n- Kitchenham BA, Dybå T, Jørgensen M (2004). Evidence-based software engineering. *Proc ICSE* 2004, 273–281.\n- Klein RA, Vianello M, Hasselman F, et al. (2018). Many Labs 2: investigating variation in replicability across samples and settings. *AMPPS* 1(4):443–490. https://journals.sagepub.com/doi/10.1177/2515245918810225\n- Lawlor DA, Tilling K, Davey Smith G (2016). Triangulation in aetiological epidemiology. *Int J Epidemiol* 45(6):1866–1886. https://academic.oup.com/ije/article/45/6/1866/2930550\n- LeLorier J, Grégoire G, Benhaddad A, et al. (1997). Discrepancies between meta-analyses and subsequent large randomized, controlled trials. *NEJM* 337:536–542.\n- Lewin S, Booth A, Glenton C, et al. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. *Implement Sci* 13(Suppl 1):2. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0688-3\n- Mastrandrea MD, Field CB, Stocker TF, et al. (2010). *Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties*. IPCC. https://www.ipcc.ch/site/assets/uploads/2018/05/uncertainty-guidance-note.pdf ; Mastrandrea MD et al. (2011). *Climatic Change* 108:675–691.\n- Murad MH, Asi N, Alsawas M, Alahdab F (2016). New evidence pyramid. *BMJ Evid Based Med* 21(4):125–127.\n- Mustafa RA, Santesso N, Brozek J, et al. (2013). The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. *J Clin Epidemiol* 66(7):736–742. https://pubmed.ncbi.nlm.nih.gov/23623694/\n- Nguyen P-Y, McKenzie JE, Alqaidoom Z, et al. (2025). Reproducibility of meta-analytic results in systematic reviews of interventions: meta-research study. *BMJ Medicine*. https://pubmed.ncbi.nlm.nih.gov/41368128/\n- Nikolakopoulou A, Higgins JPT, Papakonstantinou T, et al. (2020). CINeMA: an approach for assessing confidence in the results of a network meta-analysis. *PLoS Med* 17(4):e1003082.\n- Noyes J, Booth A, Moore G, et al. (2019). Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions. *BMJ Glob Health* 4:e000893.\n- OCEBM Levels of Evidence Working Group (2011). The Oxford 2011 Levels of Evidence. Oxford Centre for Evidence-Based Medicine. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence\n- Office of the Director of National Intelligence (2015). Intelligence Community Directive 203: Analytic Standards.\n- Open Science Collaboration (2015). Estimating the reproducibility of psychological science. *Science* 349(6251):aac4716.\n- Parkkinen V-P, Wallmann C, Wilde M, et al. (2018). *Evaluating Evidence of Mechanisms in Medicine: Principles and Procedures*. Springer (open access).\n- Pawson R (2006). *Evidence-Based Policy: A Realist Perspective*. Sage.\n- Petticrew M, Roberts H (2003). Evidence, hierarchies, and typologies: horses for courses. *J Epidemiol Community Health* 57:527–529.\n- Puhan MA, Schünemann HJ, Murad MH, et al. (2014). A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. *BMJ* 349:g5630.\n- Puttick R, Ludlow J (2013). *Standards of Evidence: An Approach that Balances the Need for Evidence with Innovation*. Nesta. https://media.nesta.org.uk/documents/standards_of_evidence.pdf\n- Riley RD, Higgins JPT, Deeks JJ (2011). Interpretation of random effects meta-analyses. *BMJ* 342:d549.\n- Riley RD, Lambert PC, Abo-Zaid G (2010). Meta-analysis of individual participant data. *BMJ* 340:c221.\n- Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA (2014). Systematic review and evidence integration for literature-based environmental health science assessments. *Environ Health Perspect* 122(7):711–718.\n- Rothman KJ, Greenland S (2005). Causation and causal inference in epidemiology. *Am J Public Health* 95(S1):S144–S150.\n- Royall R (1997). *Statistical Evidence: A Likelihood Paradigm*. Chapman & Hall.\n- Russo F, Williamson J (2007). Interpreting causality in the health sciences. *Int Stud Phil Sci* 21(2):157–170.\n- Sackett DL (1986). Rules of evidence and clinical recommendations on the use of antithrombotic agents. *Chest* 89(2 Suppl):2S–3S.\n- Scheel AM, Schijen MRMJ, Lakens D (2021). An excess of positive results: comparing the standard psychology literature with Registered Reports. *AMPPS* 4(2).\n- Schünemann HJ, Cuello C, Akl EA, et al. (2019). GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. *J Clin Epidemiol* 111:105–114. https://pubmed.ncbi.nlm.nih.gov/29432858/\n- Schünemann HJ, Zhang Y, Oxman AD (2019). Distinguishing opinion from evidence in guidelines. *BMJ* 366:l4606.\n- Sellke T, Bayarri MJ, Berger JO (2001). Calibration of p values for testing precise null hypotheses. *Am Stat* 55(1):62–71.\n- Shafer G (2021). Testing by betting. *JRSS-A* 184(2):407–431.\n- Shea BJ, Reeves BC, Wells G, et al. (2017). AMSTAR 2. *BMJ* 358:j4008.\n- Silberzahn R, Uhlmann EL, Martin DP, et al. (2018). Many analysts, one data set. *AMPPS* 1(3):337–356. https://journals.sagepub.com/doi/10.1177/2515245917747646\n- Simmons JP, Nelson LD, Simonsohn U (2011). False-positive psychology. *Psych Sci* 22(11):1359–1366.\n- Stanley TD, Doucouliagos H (2014). Meta-regression approximations to reduce publication selection bias. *Res Synth Methods* 5:60–78.\n- Stang A (2010). Critical evaluation of the Newcastle-Ottawa scale. *Eur J Epidemiol* 25:603–605.\n- Stegenga J (2011). Is meta-analysis the platinum standard of evidence? *Stud Hist Phil Biol Biomed Sci* 42(4):497–507. https://philpapers.org/archive/STEIMT.pdf\n- Stegenga J (2018). *Medical Nihilism*. Oxford University Press.\n- Sterne JAC, Hernán MA, Reeves BC, et al. (2016). ROBINS-I. *BMJ* 355:i4919; ROBINS-I V2 (2024/2025): https://www.riskofbias.info/welcome/robins-i-v2\n- Sterne JAC, Savović J, Page MJ, et al. (2019). RoB 2: a revised tool for assessing risk of bias in randomised trials. *BMJ* 366:l4898.\n- Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. *NEJM* 358:252–260.\n- US EPA (2005). *Guidelines for Carcinogen Risk Assessment*. EPA/630/P-03/001F.\n- VanderWeele TJ, Ding P (2017). Sensitivity analysis in observational research: introducing the E-value. *Ann Intern Med* 167:268–274.\n- Vovk V, Wang R (2021). E-values: calibration, combination and applications. *Ann Stat* 49(3):1736–1754. https://projecteuclid.org/journals/annals-of-statistics/volume-49/issue-3/E-values-Calibration-combination-and-applications/10.1214/20-AOS2020.full\n- Wang SV, Schneeweiss S, Franklin JM, et al. (2023). Emulation of randomized clinical trials with nonrandomized database analyses: results of 32 clinical trials. *JAMA* 329(16):1376–1385.\n- Wasserstein RL, Lazar NA (2016). The ASA statement on p-values: context, process, and purpose. *Am Stat* 70(2):129–133.\n- Wasserstein RL, Schirm AL, Lazar NA (2019). Moving to a world beyond \"p < 0.05\". *Am Stat* 73(S1):1–19.\n- What Works Clearinghouse (2022). *Procedures and Standards Handbook, Version 5.0* (WWC 2022008). https://ies.ed.gov/ncee/wwc/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf\n- Whiting PF, Rutjes AWS, Westwood ME, et al. (2011). QUADAS-2. *Ann Intern Med* 155:529–536.\n- Woodruff TJ, Sutton P (2014). The Navigation Guide systematic review methodology. *Environ Health Perspect* 122:1007–1014.\n- Worrall J (2002). What evidence in evidence-based medicine? *Philosophy of Science* 69(S3):S316–S330.\n- Worrall J (2007). Why there's no cause to randomize. *Br J Philos Sci* 58(3):451–488.\n- Wu W, Duan J, Reed WR, Tipton E (2026). What can we learn from 1,000 meta-analyses across 10 different disciplines? *Res Synth Methods* 17(1):123–156. doi:10.1017/rsm.2025.10035. https://www.cambridge.org/core/journals/research-synthesis-methods/article/what-can-we-learn-from-1000-metaanalyses-across-10-different-disciplines/538E64A39F4F151B387B9DEDF4531840\n\n### Points flagged for verification before outward citation\n\n- Exact CI of Anglemyer et al. 2014 pooled ratio of odds ratios.\n- Cuello-Garcia et al. 2022 (GRADE guidance 24) details on integrating RCT and NRS bodies.\n- ROBINS-E domain list wording (seven domains confirmed; names recalled).\n- LeLorier 1997 \"~35%\" figure (recalled).\n- Hedges & Olkin 1980 power result (recalled; the result is standard in meta-analysis texts).\n- CERQual default \"start at high confidence\" (recalled).\n- IPCC likelihood ranges and ICD 203 ranges (recalled from the guidance documents; source PDFs returned 403 today).\n- Many Labs 2 \"14 of 28\" confirmed only via a secondary summary.\n"}