{"path":"research/what-counts-as-evidence/testimony-experience-and-normative-evidence.md","content":"# The soft and contested kinds of evidence: testimony, lived experience, qualitative evidence, expert judgment, forecasts, and evidence for normative claims\n\n*Research report for the Deliberus evidence programme, 2026-09-26. Written against the briefing at `research-briefing.md`. Scope: the seven areas in the brief, followed by what transfers to a claim-graph argumentation platform. Items marked **[unverified]** were stated from memory or secondary sources and were not checked against the primary text during this pass. Items marked **[contested]** have live, published disagreement.*\n\n---\n\n## 0. Why this report exists, and the one distinction everything else hangs on\n\nThe briefing names positional or \"impact\" testimony (what a policy or state of affairs costs *from where one stands*) as the human contribution machines cannot supply. That puts the platform's reserved human role on the evidential ground where the literatures are least settled and most politically charged. This report treats it as carefully as the hard kinds.\n\nAlmost every section below turns on one distinction, worth stating first because the literature keeps finding that people run the two together:\n\n- **Evidential advantage.** Being in some position makes a person more likely to know, notice, or be able to report something relevant. This is a claim about *reliability* on a specific question, and it comes in degrees, varies by question, and can be checked.\n- **Authority.** Being in some position gives a person's say-so standing to *settle* something: to close a question, to end a line of criticism, or to count as a verdict.\n\nStandpoint epistemology at its most defensible (Wylie 2003), the epistemic-injustice literature (Fricker 2007), the empirical work on introspection and affective forecasting, and the expert-elicitation literature all support **advantage** and all warn against sliding from advantage into **authority**. The critics of \"deference politics\" (Táíwò 2020) are, underneath, objecting to exactly that slide. The platform's rule that \"testimony is evidence for a premise, never a verdict\" is the advantage-without-authority position stated in software. This report tests whether it holds up, and finds that it mostly does, with three amendments (§8).\n\n---\n\n## 1. Epistemic injustice and standpoint\n\n### 1.1 Fricker: testimonial and hermeneutical injustice\n\nMiranda Fricker's *Epistemic Injustice: Power and the Ethics of Knowing* (Oxford UP, 2007) defines two forms:\n\n- **Testimonial injustice.** A hearer gives a speaker's word a *deflated* level of credibility because of identity prejudice. The central case is a **credibility deficit** caused by a *negative identity-prejudicial stereotype*: a stereotype that tracks social identity (race, gender, class) and resists counter-evidence because of an ethically bad affective investment. Fricker's paradigm cases are Tom Robinson in *To Kill a Mockingbird* and Marge Sherwood in *The Talented Mr Ripley*.\n- **Hermeneutical injustice.** A gap in a society's shared interpretive resources puts someone at an unfair disadvantage in making sense of their own social experience. The standard example is sexual harassment before the concept existed (Carmita Wood, 1970s, via Brownmiller's memoir): the experience was real, but there was no public concept to report it under, so it could not be *testified to* as what it was.\n\nFricker's remedy is a set of virtues rather than a procedure: **testimonial justice** (a reflexive correction of credibility judgments for the influence of prejudice) and **hermeneutical justice** (sensitivity to the possibility that a speaker's difficulty in making sense reflects a gap in the collective resources, not a failing in the speaker).\n\n**Credibility excess.** Fricker initially argued that credibility *excess* is not, in the central case, a testimonial injustice, because it does not wrong the speaker in their capacity as a knower. This is **[contested]**. José Medina (*The Epistemology of Resistance*, Oxford UP, 2013; \"The Relevance of Credibility Excess in a Proportional View of Epistemic Injustice\", *Social Epistemology* 25(1), 2011) argues that credibility is **comparative and proportional**: an excess given to one party is, in a zero-sum or near-zero-sum setting (a courtroom, a meeting, a ranked feed), a deficit imposed on another. Emmalon Davis (\"Typecasting, Tokenism, and Tokenism\", *Hypatia* 31(3), 2016) adds that excess can wrong the recipient too: being treated as the spokesperson for one's identity group (\"typecasting as a knower\") is an injustice even when it inflates credibility.\n\nFor a ranked or scored system this matters directly. Any mechanism that raises the weight of one class of contributor lowers the relative standing of every other in any view that ranks, sorts, or truncates. A credibility adjustment is never local.\n\n### 1.2 Dotson: testimonial quieting and smothering\n\nKristie Dotson, \"Tracking Epistemic Violence, Tracking Practices of Silencing\" (*Hypatia* 26(2), 2011, 236–257), identifies two practices of silencing that sit before or beside Fricker's credibility deficit:\n\n- **Testimonial quieting**: the audience fails to recognise the speaker as a knower at all, so the speech is not taken up as testimony. Dotson draws on Patricia Hill Collins's account of controlling images of Black women.\n- **Testimonial smothering**: the speaker truncates their own testimony because they perceive the audience as unwilling or unable to take it up, and the content is \"unsafe and risky\" to share. The testimony that would have been evidence is never given.\n\nDotson's framing is **dependency**: testimony requires an audience to meet the speaker's vulnerability, and silencing is a failure of that reciprocity. Smothering is the one most relevant to platform design, because it is a *supply-side* loss that no scoring rule sees: the platform never learns what was not said. It is a case of what the Deliberus corpus calls **coherent absence**, with a social cause.\n\n### 1.3 Standpoint epistemology\n\n**Lineage.** Nancy Hartsock, \"The Feminist Standpoint: Developing the Ground for a Specifically Feminist Historical Materialism\" (in Harding & Hintikka eds., *Discovering Reality*, 1983), adapting Marx and Lukács; Sandra Harding, *The Science Question in Feminism* (1986) and *Whose Science? Whose Knowledge?* (1991), which introduced **strong objectivity**: objectivity is increased, not decreased, by starting inquiry from marginalised lives, because doing so exposes the background assumptions of dominant groups to scrutiny; Patricia Hill Collins, *Black Feminist Thought* (1990), with the **outsider-within** position (Black women domestic workers seeing white families from inside while excluded from them); Dorothy Smith's sociology \"from the standpoint of women\".\n\n**Wylie's reconstruction.** Alison Wylie, \"Why Standpoint Matters\" (in Figueroa & Harding eds., *Science and Other Cultures*, Routledge, 2003), is the most useful single source for design purposes because it separates three theses:\n\n1. **Situated knowledge thesis**: social location systematically shapes and limits what we know, including tacit, experiential knowledge.\n2. **Inversion thesis**: those subject to structures of domination may be epistemically advantaged *in some crucial respects*.\n3. The inversion thesis is defensible only in a **contingent, non-essentialist** form: epistemic **advantage** (not privilege) that is *relative to a specific epistemic purpose*, and which is not automatic. Wylie is explicit that comprehensive epistemic privilege \"cannot be attributed automatically to those who are oppressed just because they are oppressed.\"\n\nWylie also stresses, following Harding and Hartsock, that a *standpoint* is an **achievement**, not a location: a critical consciousness developed through reflection on one's situation, usually collectively. A location gives access to evidence; turning that access into knowledge is work. This achievement thesis is itself debated (a 2025 *Hypatia* paper, \"Distinguishing Situated Knowledge and Standpoint Theory: Defending the Achievement Thesis\", defends it against collapse into situated-knowledge theory).\n\nThe *location/achievement* distinction maps onto the platform's intended split between *self-attested* positional testimony and *positionally corroborated* testimony, although, as §8 argues, corroboration is not the same thing as achievement.\n\n**Consensus and controversy.** The situated knowledge thesis is close to uncontroversial in philosophy of science: it is continuous with Longino's social epistemology and with ordinary claims about access (§6.5). The automatic privilege thesis is widely rejected, including by standpoint theorists. The live disputes are (a) whether \"achievement\" is a coherent criterion that can be applied without circularity, and (b) how to identify *which* questions a given location advantages.\n\n### 1.4 \"Nothing about us without us\"\n\nThe slogan descends from Central European political tradition (*nihil de nobis sine nobis*) and was popularised by the disability rights movement in the 1990s; James Charlton's *Nothing About Us Without Us: Disability Oppression and Empowerment* (University of California Press, 1998) is the standard reference. It is embedded in the UN Convention on the Rights of Persons with Disabilities (2006), Article 4(3), which requires States Parties to \"closely consult with and actively involve persons with disabilities\" in developing policy affecting them.\n\nIt is a **procedural** principle about participation, not an **evidential** principle about credibility. It says those affected must be present when decisions are made about them. It does not say their reports settle empirical or normative questions. Read carefully, it is compatible with the advantage-without-authority line.\n\n### 1.5 The critique: elite capture and deference politics\n\nOlúfẹ́mi O. Táíwò, \"Being-in-the-Room Privilege: Elite Capture and Epistemic Deference\" (*The Philosopher* 108(4), 2020), later expanded in *Elite Capture: How the Powerful Took Over Identity Politics (and Everything Else)* (Haymarket, 2022). The argument:\n\n- Standpoint epistemology as a *theory* is sound: marginalised positions offer knowledge.\n- **Deference epistemology** as a *practice* (pass the mic, defer to the most marginalised person in the room) goes wrong because **the people in the room are not a representative sample** of the marginalised. Being in the room is itself a privilege. Deferring to whoever is present routes the standpoint's authority to the most advantaged members of a marginalised group: that is elite capture.\n- Deference also shifts attention from *outcomes* (redistributing power and resources) to *interaction* (who speaks in this meeting), and it can be self-undermining by relieving the deferrer of the duty to think.\n- Táíwò's alternative is **constructive**: build and redistribute, rather than defer.\n\nRelated critiques: Linda Martín Alcoff, \"The Problem of Speaking for Others\" (*Cultural Critique* 20, 1991), on the costs of both speaking for and refusing to speak for; and the general worry, in deliberative democracy (Young 2000, *Inclusion and Democracy*), that \"the voice of the affected\" can be instrumentalised by whoever controls which affected voices are heard.\n\n**The design lesson** from Táíwò is statistical before it is political: *the testimony a platform receives is a self-selected sample of the positions it purports to represent.* An unrepresentative sample of positional testimony, given authority, is a biased estimator with a moral halo. This is the ingestion-balance problem (the corpus's residual-error class 2) arriving from the political side.\n\n### 1.6 Evidential advantage versus authority, summarised\n\n| | Supported by | Warned against by |\n|---|---|---|\n| Positional testimony carries evidence others lack | Fricker, Wylie, Collins, Gilbert et al. 2009 (§4.4), disability-paradox data (§4.4) | nobody serious |\n| Credibility judgments should be corrected for identity prejudice | Fricker, Medina, Dotson | nobody serious |\n| Positional testimony should settle questions within its domain | \"strong\" deference practice | Wylie (no automatic privilege), Táíwò, Alcoff, introspection research (§4) |\n| Credibility excess is harmless | Fricker 2007 (initially) | Medina 2011/2013, Davis 2016 |\n\n---\n\n## 2. Lived experience in policy and science\n\n### 2.1 Patient-reported outcomes (PROs/PROMs)\n\nA **patient-reported outcome** is a report of the status of a patient's health that comes directly from the patient, without interpretation by a clinician or anyone else. The **FDA Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims** (December 2009) sets out how PROMs can support label claims, which require evidence of:\n\n- **content validity** (the instrument measures concepts that matter to patients, established through qualitative work with patients: concept elicitation and cognitive interviewing);\n- **reliability** (test–retest, internal consistency);\n- **construct validity** and **ability to detect change**;\n- a defined **meaningful change threshold**.\n\nThe **COSMIN** initiative (COnsensus-based Standards for the selection of health Measurement INstruments; Mokkink et al., from 2010) provides the consensus taxonomy of measurement properties and risk-of-bias checklists for PROM studies. The **FDA Patient-Focused Drug Development** guidance series (2018–2023) extends this, formalising the collection of patient input on what outcomes matter.\n\n**The key structural lesson.** A first-person report becomes *regulatory-grade* evidence when it is: (a) about a construct patients themselves defined as important; (b) collected with an instrument whose psychometric properties are known; (c) aggregated across patients; and (d) interpreted against a threshold for meaningful change. The patient is authoritative about *what matters* (content validity is established by patients) and is the *measuring instrument* for their own state, but the evidential weight comes from the validated instrument and aggregation. No single report settles anything. This is the institutional version of \"incorrigible about the felt state, fallible about everything else\", and its answer is to pair respect for the report with *instrument discipline*.\n\n### 2.2 Patient and public involvement (PPI)\n\nPPI (UK NIHR terminology; \"patient engagement\" in the US, where PCORI requires it) means doing research *with* or *by* patients rather than *to*, *about* or *for* them. Reporting standards: **GRIPP2** (Guidance for Reporting Involvement of Patients and the Public, Staniszewska et al., *BMJ* 2017).\n\n**The evidence on PPI's impact is weak.** Brett et al., \"Mapping the impact of patient and public involvement on health and social care research: a systematic review\" (*Health Expectations* 17(5), 2014), searching 1995–2012, found mostly positive reported impacts (better recruitment, more relevant questions, more acceptable instruments), but the evidence base was dominated by case reports and self-assessment, with little robust comparative evidence. Later reviews reached similar conclusions. PPI is justified mainly on **democratic and relevance** grounds (the \"moral\" or \"rights\" argument) and only secondarily on consequentialist grounds for research quality, where the evidence is thin. This is **[contested]** among PPI researchers, some of whom argue impact measurement misunderstands what PPI is for.\n\n**Design lesson:** the strongest case for involving affected people is that they change **which questions get asked** and **what counts as an outcome** (framing), rather than that they change the answers to a fixed question. That fits the corpus's view that the reframe is one of the highest-value human acts.\n\n### 2.3 Qualitative evidence synthesis\n\nTwo dominant methods:\n\n- **Meta-ethnography** (Noblit & Hare, *Meta-Ethnography: Synthesizing Qualitative Studies*, Sage, 1988): translate the concepts of each study into those of the others. Syntheses can be **reciprocal translation** (studies say similar things), **refutational synthesis** (studies contradict), or **line-of-argument synthesis** (a new interpretation that encompasses the studies). Reporting standard: **eMERGe** (France et al., 2019).\n- **Thematic synthesis** (Thomas & Harden, *BMC Medical Research Methodology* 8:45, 2008): line-by-line coding of findings → descriptive themes → analytical themes that go beyond the primary studies.\n\nOthers: framework synthesis, critical interpretive synthesis (Dixon-Woods et al. 2006), meta-aggregation (JBI).\n\nMeta-ethnography's **refutational synthesis** is the qualitative analogue of an ATTACKS edge between findings, and line-of-argument synthesis is the analogue of a mother claim over child findings. The qualitative-synthesis literature has already worked through the question \"how do you combine findings that are not numbers?\", and its answer is **translation plus explicit handling of contradiction**, not averaging.\n\n### 2.4 GRADE-CERQual\n\n**GRADE-CERQual** (Confidence in the Evidence from Reviews of Qualitative research; Lewin et al., *PLOS Medicine* 2015; full guidance series in *Implementation Science* 13 suppl 1, 2018) assesses confidence in *each individual review finding* against four components:\n\n1. **Methodological limitations** of the contributing studies;\n2. **Coherence**: how clear and cogent the fit is between the primary data and the review finding (ill-fitting data lowers confidence);\n3. **Adequacy of data**: richness and quantity of data supporting the finding (thin data from few participants lowers confidence);\n4. **Relevance**: how applicable the contributing data are to the context of the review question (indirect, partial, or unclear relevance lowers confidence).\n\nThe overall judgment is **high / moderate / low / very low confidence** that the finding is \"a reasonable representation of the phenomenon of interest.\" Two features matter for Deliberus:\n\n- CERQual rates **per finding, not per source**, which is precisely the per-claim versus per-source methodology gap the briefing names.\n- **Coherence** is defined partly by whether contrary cases have been explained. A finding that ignores disconfirming data is marked down. That is a built-in critical question.\n\nCERQual deliberately does **not** rank qualitative evidence below quantitative evidence on a single ladder. It asks a different question (\"is this a reasonable representation of the phenomenon?\") rather than \"how large is the effect?\". Placing \"anecdotal 0.32\" on the same ladder as \"RCT 0.65\" is therefore a category error by CERQual's lights: the methodology ladder is answering the question \"how well does this estimate an effect?\", and qualitative evidence is typically answering a different question.\n\n### 2.5 Participatory research and citizen science\n\n- **Community-based participatory research (CBPR)** (Israel et al., *Annual Review of Public Health* 19, 1998; Wallerstein & Duran): communities are partners across all stages. Evidence for improved relevance and uptake; methodological concerns about rigour and role conflict persist.\n- **Citizen science**: volunteers collect or analyse data. Quality concerns are addressed by **protocol design, training, redundancy, and statistical correction** rather than by credentialing. Examples: Galaxy Zoo (classifications aggregated across many volunteers with per-volunteer weighting learned from agreement), eBird (checklists filtered by expected species/counts, with review of flagged records), Christmas Bird Count. Kosmala et al., \"Assessing data quality in citizen science\" (*Frontiers in Ecology and the Environment* 14(10), 2016), conclude that volunteer data quality is often comparable to professional data when projects are designed for it, and that quality is a property of project design rather than of volunteer status.\n\nThe citizen-science lesson carries directly to an argumentation platform: **quality comes from redundancy, protocol, and measured per-contributor reliability learned from the data, not from credentials**, which is the \"no expert gate\" position with an engineering answer attached.\n\n### 2.6 Testimony in truth commissions\n\nTruth commissions treat testimony as evidence at three levels: as **individual truth** (acknowledgment), as **aggregate data** (patterns of violations), and as **statistical input** (estimating what was not reported). The **South African TRC** (1996–1998) took roughly 21,000 victim statements **[approximate; unverified]** and distinguished four kinds of truth in its *Final Report* (vol. 1, ch. 5): **factual or forensic truth**, **personal or narrative truth**, **social or \"dialogue\" truth**, and **healing and restorative truth**. That taxonomy separates the evidential role of testimony from its acknowledgment role explicitly, and is one of the few institutional attempts to do so.\n\nThe **Peru Truth and Reconciliation Commission** (CVR, final report 2003) collected about 17,000 testimonies. The deaths and disappearances documented *by name* numbered about 24,000; the Human Rights Data Analysis Group (Patrick Ball, Jana Asher and CVR staff) used **multiple systems estimation** (capture–recapture across overlapping lists) to estimate **69,280 deaths (95% interval roughly 61,007–77,552)**, and found that the Shining Path, not the state, was responsible for the largest share, reversing the pattern suggested by raw testimony counts. Similar methods were used in Guatemala (CEH, 1999) and Kosovo.\n\n**This is the single most important institutional case for Deliberus.** Testimony was indispensable (nothing could be estimated without it) and simultaneously *systematically biased as a sample* (who testifies depends on who survived, who was reachable, who trusted the commission). The resolution was neither to discount testimony nor to take counts at face value, but to **model the process that produced the testimony**, including its overlaps. The overlap between independent lists is the evidence about what is missing. This is also a direct method for the briefing's gap \"no source independence / shared-origin correlation\": capture–recapture *requires* knowing which reports refer to the same event and which lists are independent.\n\n### 2.7 How institutions actually weigh lived experience against statistics\n\nAcross the institutions above, the practice converges on a division of labour rather than a weighting:\n\n| Question | Where lived experience leads | Where statistics lead |\n|---|---|---|\n| What outcomes matter? What should be measured? | yes (PROM content validity, PPI, CBPR) | no |\n| What is it like? What mechanisms operate? What barriers exist? | yes (qualitative synthesis, CERQual) | supplementary |\n| How big is the effect? How common is it? | informs the instrument | yes (GRADE for effects) |\n| Is a recommendation acceptable, feasible, equitable? | yes (EtD criteria, §6.6) | contributes |\n| Does the individual case contradict the rule? | can be decisive (§3) | contextualises |\n\nGRADE's Evidence-to-Decision frameworks (§6.6) formalise this: evidence on *effects* is graded with GRADE, evidence on *values, acceptability and feasibility* is often qualitative (graded with CERQual), and the two enter as **separate criteria** rather than being pooled on one scale.\n\n---\n\n## 3. Case studies and anecdotes\n\n### 3.1 Flyvbjerg's five misunderstandings\n\nBent Flyvbjerg, \"Five Misunderstandings About Case-Study Research\" (*Qualitative Inquiry* 12(2), 2006, 219–245). The five misunderstandings and his corrections:\n\n1. *General, theoretical knowledge is more valuable than concrete, practical knowledge.* Correction: in human affairs there is only context-dependent knowledge; expertise (Dreyfus) is built on thousands of concrete cases.\n2. *One cannot generalise from a single case.* Correction: one often can, through **strategic case selection**; and single cases can **falsify** (Popper's black swan).\n3. *The case study is most useful for generating hypotheses.* Correction: it is useful for both generating and testing.\n4. *The case study has a bias toward verification.* Correction: Flyvbjerg reports that case researchers more often have their preconceptions *falsified*, citing Campbell, Ragin, Geertz and Wieviorka.\n5. *It is hard to summarise case studies.* Correction: that is often a property of the reality studied, and the case should sometimes be left as a narrative.\n\nHe distinguishes case selection strategies: **extreme/deviant cases**, **maximum-variation cases**, **critical cases** (\"if it is valid for this case, it is valid for all or many cases\", or the reverse), and **paradigmatic cases**. Critical-case logic: choose the *most likely* case for a hypothesis to hold; if it fails there, it probably fails everywhere (or choose the *least likely* case; if it holds there, it probably holds widely). Flyvbjerg's example is Galileo's refutation of Aristotle's law of gravity, reasoned from a critical case rather than from a sample.\n\nFlyvbjerg's article is highly cited and influential, but his reading of the \"bias toward verification\" evidence relies on researcher self-reports, and his dichotomy between context-dependent and context-independent knowledge is **[contested]**.\n\n### 3.2 When a single case is decisive\n\nA single case decides a question when the claim is **universal** (one counterexample refutes \"all swans are white\"), when the case is **critical** in the sense above, or when it serves as an **existence proof** (one flying machine shows powered flight is possible). The formal point is about likelihood ratios. A case is decisive when *P(case | H)* is near zero, so the likelihood ratio is extreme however small the sample. Anecdote is weak evidence for **frequency or effect-size** claims (where one case barely moves the estimate) and can be strong evidence for **possibility, universality, or mechanism** claims.\n\nTwo refinements:\n\n- Duhem–Quine applies: the black swan refutes \"all swans are white\" only together with auxiliary claims (it is a swan, it is not painted, the observer is reliable). A \"decisive\" case moves the attack to the auxiliaries, which is why a sole witness needs critical questions about the auxiliaries rather than about the conclusion.\n- Existence claims made from a case are only as good as the case's verification. The corresponding critical question is \"has the case been checked?\", not \"is the sample large?\".\n\n### 3.3 Anecdote versus data: the psychology\n\n- **Base-rate neglect** (Kahneman & Tversky, \"On the psychology of prediction\", *Psychological Review* 80, 1973; Bar-Hillel 1980): people underweight base rates when given individuating information. The classic cab problem gives a correct answer of ~41% where the modal response is 80%.\n- **Identifiable victim effect** (Schelling 1968; Jenni & Loewenstein 1997; Small, Loewenstein & Slovic, \"Sympathy and callousness\", *OBHDP* 102, 2007): in Small et al.'s Study 1, donations to an identified girl (Rokia) were roughly twice those prompted by statistics about millions of victims (about $2.38 vs $1.14 per participant **[figures from memory; unverified]**), and adding statistics to the identified victim *lowered* giving (about $1.43 **[unverified]**). Teaching participants about the effect reduced giving to the identified victim without increasing giving to statistical victims. A meta-analysis (Lee & Feeley, *Social Influence* 11(3), 2016) found the effect small overall (r ≈ .05) and moderated by conditions **[effect size from memory; unverified]**, and replication attempts show a real but smaller and more conditional effect **[contested]**.\n- **Vividness** (Nisbett & Ross, *Human Inference*, 1980): vivid, concrete information is hypothesised to be overweighted. Taylor & Thompson's review (\"Stalking the elusive vividness effect\", *Psychological Review* 89, 1982) found the evidence for a general vividness effect surprisingly weak, a useful corrective **[contested]**.\n\nThe design implication is two-sided. Anecdote is overweighted relative to base rates for *frequency* judgments, which argues against letting a vivid story move a frequency claim. But the identifiable-victim literature also shows that *statistics suppress* moral attention, so a platform that renders only aggregates will under-represent what the aggregate is about. The answer is structural: **type the claim the anecdote supports** (possibility, mechanism, or frequency) rather than discounting anecdotes globally.\n\n### 3.4 N-of-1 trials\n\nAn **N-of-1 trial** is a randomised, often blinded, multiple-crossover trial within a single patient (Guyatt et al., \"Determining optimal therapy — randomized trials in individual patients\", *NEJM* 314, 1986). The Oxford Centre for Evidence-Based Medicine 2011 levels place the N-of-1 trial at **Level 1 for treatment-benefit questions about that individual patient**. Aggregated N-of-1 trials can also estimate population effects and heterogeneity (Zucker et al. 1997; CONSORT extension **CENT 2015**, Vohra et al., *BMJ*).\n\nThis is the cleanest counterexample to the idea that single-person evidence is inherently low-tier. The *design* (randomisation, blinding, repeated crossover) decides evidential weight, not the N. The corresponding correction to a methodology ladder is that the ladder ranks *designs for a question*, and the question \"does this work for *me*?\" has a different top rung from \"does this work on average?\". An N-of-1 trial also illustrates first-person authority done carefully: the patient's report is the outcome measure, and randomisation and blinding protect it from the patient's own causal theories.\n\n---\n\n## 4. Introspection and first-person reports\n\nThe briefing's conviction is **fallible self-knowledge**: first-person testimony is incorrigible about the felt experience and fallible about causes, counterfactuals and what would satisfy it. The evidence strongly supports the second half. It does not fully support the first.\n\n### 4.1 Reports about causes: Nisbett & Wilson (1977)\n\nRichard Nisbett and Timothy Wilson, \"Telling More Than We Can Know: Verbal Reports on Mental Processes\" (*Psychological Review* 84(3), 1977, 231–259). Core claim: people often have **no direct introspective access to the higher-order cognitive processes** that produce their judgments. Their reports about *why* they did something are generated from **a priori causal theories** (what would plausibly cause such a response), which is why observers given the same situation produce the same explanations. Classic demonstration: shoppers evaluating four identical pairs of stockings showed a strong position effect (the right-most pair was preferred over the left-most by nearly four to one), and **no participant mentioned position** as a reason. Participants denied that position could matter when asked.\n\nA 2016 retrospective (\"Nisbett and Wilson (1977) Revisited: The Little That We Can Know and Can Tell\", *Social Cognition* 34(3), 2016 **[authorship not checked]**) and philosophical re-readings (e.g., \"Telling Just What We Know: Revisiting Nisbett and Wilson's Stocking Case\", *Philosophy*, Cambridge) argue that some of the original demonstrations are weaker than the headline, but the core finding that causal self-reports are theory-driven is broadly accepted.\n\n### 4.2 Choice blindness (Johansson & Hall)\n\nPetter Johansson, Lars Hall, Sverker Sikström & Andreas Olsson, \"Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task\" (*Science* 310, 2005, 116–119). Participants chose the more attractive of two faces. On manipulated trials the experimenter used sleight of hand to hand them the *non-chosen* face and asked them to explain their choice. Most manipulations went undetected (commonly summarised as only about 13% concurrently detected, and fewer than about a quarter detected overall **[exact figures unverified against the primary paper]**), and participants gave confident reasons for a choice they had not made, sometimes citing features present only in the non-chosen face (earrings, for example). The effect has been extended to taste (jam and tea), moral attitudes (Hall, Johansson & Strandberg, *PLoS ONE* 2012: people defended reversed positions on moral statements), and political attitudes before the 2010 Swedish election (Hall et al., *PLoS ONE* 2013), where manipulated attitude reports shifted later voting intentions.\n\n**Implication:** a person's *stated reasons* for a preference, and even their *stated position* on a moral statement, are partly constructed at the moment of report and are shaped by what they are shown. This is directly relevant to an interface that shows a user a machine rendering of their position and asks them to confirm or reject it: choice blindness says confirmation of a rendering is **weak evidence** that the rendering matches the prior position. It is also relevant to the corpus's \"entrenchment hazard\": the act of articulating reasons can itself move the position.\n\n### 4.3 Reports about current experience: Schwitzgebel\n\nEric Schwitzgebel, \"The Unreliability of Naive Introspection\" (*Philosophical Review* 117(2), 2008, 245–273), and *Perplexities of Consciousness* (MIT Press, 2011). Thesis: \"we are prone to gross error, even in favorable circumstances of extended reflection, about our own ongoing conscious experience, our current phenomenology.\" Examples: the phenomenology of emotion (whether joy has a distinctive felt core, how bodily it is), peripheral vision (people overestimate the region of visual clarity), the phenomenology of thought, and whether we dream in colour (reports shifted historically with the dominant media). Schwitzgebel argues that the foundationalist picture, on which knowledge of one's own experience is the secure base from which knowledge of the world is inferred, is \"almost exactly backward\".\n\nResponses are **[contested]**. Defenders of first-person authority (for example, Declan Smithies; Charles Siewert; the \"transparency\" tradition after Evans and Moran) argue that Schwitzgebel's cases concern **fine-grained structural descriptions** of experience (how wide, how coloured, how bodily) rather than **coarse categorical states** (am I in pain? am I anxious now?), and that reliability for the latter is high. Hurlburt's **Descriptive Experience Sampling** (Hurlburt & Schwitzgebel, *Describing Inner Experience? Proponent Meets Skeptic*, MIT Press, 2007) is the methodological middle ground: momentary, beeper-cued reports with careful interviewing are more reliable than generalised reports (\"I usually think in words\").\n\n**This bears directly on the platform's conviction.** The strongest defensible version is not \"incorrigible about the felt experience\" but something like:\n\n> First-person reports carry **presumptive authority** over *coarse, present, categorical* states (I am in pain; this is distressing to me; I do not endorse that description of my view), are **fallible** about *fine-grained structure* of experience and about *generalisations over time* (I am always anxious at work), and are **weak evidence** about *causes, counterfactuals, and satisfiers*.\n\n\"Incorrigible\" overstates the first tier. \"Presumptive\" (defeasible, but the burden sits with the challenger) better fits the evidence and still protects against credibility deficits. The corpus's own earlier correction (a claim about one's own past intention is a self-interpretation, while the present refusal to endorse is a speech act that cannot be wrong) is consistent with this. The one thing that is genuinely incorrigible is the **speech act** of refusing to endorse a rendering, because it creates the fact it reports. Everything else is a report, and reports can be wrong.\n\n### 4.4 Affective forecasting and remembered experience\n\n- **Affective forecasting** (Wilson & Gilbert, \"Affective forecasting\", *Advances in Experimental Social Psychology* 35, 2003; \"Affective forecasting: Knowing what to want\", *Current Directions in Psychological Science* 14, 2005). People predict the *valence* of future events well but overestimate the **intensity and duration** of their emotional reactions: the **impact bias**. Mechanisms: **focalism** (the focusing illusion — Schkade & Kahneman, \"Does living in California make people happy?\", *Psychological Science* 9, 1998: \"nothing in life is as important as you think it is when you are thinking about it\") and **immune neglect** (Gilbert et al., *JPSP* 75, 1998: people fail to anticipate their own psychological coping). Its size and generality are **[contested]**: Levine et al. (*JPSP* 2012) argue part of the impact bias is a measurement artefact of how questions are posed.\n- **Surrogation beats simulation.** Gilbert, Killingsworth, Eyre & Wilson, \"The Surprising Power of Neighborly Advice\" (*Science* 323, 2009): people predicted their own reaction to a speed date (n = 25) and to a peer evaluation (n = 88) *more accurately* when told only how **another person who had experienced it** had felt than when given full information about the event. Both participants and independent judges wrongly expected the opposite. **This is the single most direct experimental support for impact testimony as evidence.** A report of what something was like, from someone who has been through it, beat imaginative simulation of it, and people systematically undervalue such reports.\n- **The disability paradox.** Albrecht & Devlieger, \"The disability paradox: high quality of life against all odds\" (*Social Science & Medicine* 48, 1999): 54.3% of 153 interviewees with moderate to serious disabilities reported good or excellent quality of life, against an outsider expectation of much lower. Across the health-valuation literature (Ubel, Loewenstein & Jepson, *Quality of Life Research* 12, 2003, \"Whose quality of life?\"), **patients rate the quality of life of a health state higher than the general public imagining it.** Ubel et al. identify three candidate causes: patients and public are not evaluating the same state; they are using different measuring sticks (scale recalibration, \"response shift\"); and there are genuine differences in the state's badness. This is the live example of impact testimony correcting an outsider consensus *and* of it being ambiguous what the correction is evidence *of* (the state's badness, or adaptation, or scale shift). It also bites politically: QALY weights built from public valuations systematically rate disabled lives as worse than disabled people do. Which set of valuations should govern resource allocation is a **normative** question that the empirical discrepancy does not settle (see §6).\n- **Remembered versus experienced utility.** Redelmeier & Kahneman (*Pain* 66, 1996) and Kahneman et al. (*Psychological Science* 4, 1993, cold-pressor): retrospective evaluations follow a **peak-end rule** with **duration neglect**. Extending a painful colonoscopy with a milder tail made it remembered as less bad. Retrospective testimony about an experience is therefore a report of the *remembered* experience, a distinct quantity from the integral of momentary experience.\n- **Experience sampling.** The Experience Sampling Method (Csikszentmihalyi & Larson, *Journal of Nervous and Mental Disease* 175, 1987) and the Day Reconstruction Method (Kahneman, Krueger, Schkade, Schwarz & Stone, *Science* 306, 2004) reduce recall bias by collecting reports close to the moment. Killingsworth & Gilbert (*Science* 330, 2010) used phone sampling of ~2,250 people to find minds wandering 46.9% of the time, and wandering associated with lower happiness.\n\n### 4.5 What first-person authority covers: the evidence for and against the platform's line\n\n| Kind of first-person claim | Evidence on reliability | Suggested standing |\n|---|---|---|\n| \"I refuse to endorse that description of my position\" (present speech act) | Not a report; constitutes the fact | **Authoritative** (sole-author contribution) |\n| Coarse present state (\"this hurts\", \"I'm distressed now\") | High reliability for categorical states; Schwitzgebel's cases mostly concern fine structure | **Presumptive**: defeasible, burden on challenger |\n| Fine-grained structure of experience; generalisations over time | Schwitzgebel 2008; experience-sampling vs retrospective discrepancies | **Fallible**; improved by momentary sampling |\n| Retrospective evaluation (\"it was awful\") | Peak-end, duration neglect | **Evidence of remembered experience**, not of the integral |\n| Causes of one's own state or choice | Nisbett & Wilson; choice blindness | **Weak**; generated from folk theory, same as an observer's |\n| Counterfactuals (\"I would have been fine if…\") | Affective forecasting; immune neglect | **Weak** |\n| Satisfiers (\"what would fix this is X\") | Impact bias; focalism; wanting/liking dissociation | **Weak to moderate**; strengthened by others who tried X (surrogation) |\n| Report of what an experience was like, given to someone who has not had it | Gilbert et al. 2009; disability paradox | **Strong evidence *for others'* forecasts** and outsider valuations |\n\nThe last row is the important addition. The corpus's line is framed as a limit on self-knowledge. The literature shows that positional testimony is *most* valuable in the other direction, as a corrective to **other people's simulations** of a position. Impact testimony's primary evidential job is to *attack outsider forecasts and valuations*, not to establish causes.\n\n---\n\n## 5. Expert judgment and elicitation\n\n### 5.1 Cooke's classical model\n\nRoger Cooke, *Experts in Uncertainty: Opinion and Subjective Probability in Science* (Oxford UP, 1991). Experts give quantiles (typically 5th, 50th, 95th) for **seed (calibration) variables** whose true values are known to the analyst but not to the experts, as well as for the **target variables**. Each expert is scored on:\n\n- **Calibration score**: the p-value of a chi-squared-style test of whether the expert's realisations fall in the inter-quantile bins at the stated rates (5%, 45%, 45%, 5%). This measures statistical accuracy.\n- **Information score**: the relative information (Kullback–Leibler divergence) of the expert's distributions against a background (often uniform or log-uniform) distribution. This measures how narrow, and therefore informative, the expert is.\n- **Weight** proportional to calibration × information, with a **cutoff**: experts whose calibration falls below an optimised threshold get zero weight. This makes the score a **strictly proper scoring rule** in the long run, so experts cannot game it by misreporting.\n\n**Results.** Colson & Cooke, \"Cross validation for the classical model of structured expert judgment\" (*Reliability Engineering & System Safety* 163, 2017), on 33 post-2006 studies: performance weighting beat equal weighting in-sample in all but one; **out-of-sample, performance weighting beat equal weighting on the combined validity index in 26 of 33 studies** (p ≈ 0.001 under a null of no difference), though with an out-of-sample penalty relative to in-sample performance. Colson & Cooke, \"Expert Elicitation: Using the Classical Model to Validate Experts' Judgments\" (*Review of Environmental Economics and Policy* 12(1), 2018), summarises this for a policy audience. Cooke, Marti & Mazzuchi, \"Expert forecasting with and without uncertainty quantification and weighting: What do the data say?\" (*International Journal of Forecasting* 37, 2021), extends to 49 post-2006 studies with similar conclusions.\n\nA common finding across these studies is that **most experts are poorly calibrated** (a large share of individual experts would be rejected as statistical hypotheses at conventional thresholds) and that **calibration does not track seniority or reputation** **[typical finding across Cooke's database; exact proportions unverified]**. Critics (e.g., Clemen, \"Comment on Cooke's classical method\", *RESS* 2008; Bolger & Rowe 2015) question whether seed variables are representative of target variables, how many seeds are needed (typically 10+), and whether the out-of-sample advantage is large enough to justify the cost. The debate over performance versus equal weighting is **[contested]** but the cross-validation evidence currently favours performance weighting when good seed questions exist.\n\n**Transfer:** the classical model is the canonical answer to \"how do we weight experts without an expert gate?\" It weights *demonstrated calibration on checkable questions*, not credentials, and it can assign zero weight to a famous expert and substantial weight to an unknown one. It needs **checkable seed questions in the relevant domain**, which is the oracle gradient again: calibration can only be measured where ground truth arrives.\n\n### 5.2 Delphi, SHELF, IDEA\n\n- **Delphi** (RAND, Dalkey & Helmer, *Management Science* 9, 1963): anonymous, iterated rounds with controlled feedback and statistical aggregation. Rowe & Wright's review (*International Journal of Forecasting* 15, 1999) found Delphi generally more accurate than unstructured groups and than first-round statistical aggregates, but the evidence was modest and heterogeneous. Known risk: convergence can come from **conformity** rather than learning.\n- **SHELF** (Sheffield Elicitation Framework; O'Hagan, Oakley and colleagues; see O'Hagan et al., *Uncertain Judgements: Eliciting Experts' Probabilities*, Wiley, 2006): a facilitated **behavioural aggregation**. Experts first make private judgments, then discuss and agree a single distribution representing what a \"rational impartial observer\" would believe given the discussion. Strong on process documentation. The risk is that dominant voices steer the consensus.\n- **IDEA protocol** (Investigate–Discuss–Estimate–Aggregate; Hemming, Burgman, Hanea, McBride & Wintle, \"A practical guide to structured expert elicitation using the IDEA protocol\", *Methods in Ecology and Evolution* 9, 2018; Burgman, *Trusting Judgements*, Cambridge UP, 2015): private first estimates (with four-point intervals: lowest, highest, best, confidence), then **facilitated discussion focused on reasoning and evidence** (not on converging numbers), then **private second estimates**, then **mathematical aggregation**. Hemming et al. report aggregated judgments almost always outperform individual experts, with 5–12 experts a practical range, and that **status and experience were poor predictors of performance**.\n\nIDEA is the elicitation protocol closest to Deliberus's philosophy: **independence first, then an exchange of reasons, then independent re-estimation, then mechanical aggregation.** Discussion is for surfacing reasons and resolving linguistic ambiguity, not for manufacturing agreement. That is the Delphi lesson (anonymity reduces conformity) combined with the hidden-profile lesson (discussion is needed to share private information).\n\n### 5.3 Wisdom of crowds and its conditions\n\n- **Galton**, \"Vox Populi\" (*Nature* 75, 1907): 787 entries in a weight-judging competition at Plymouth; median estimate 1,207 lb versus the dressed weight of 1,198 lb (under 1% error). A later recomputation puts the mean at 1,197 lb **[unverified]**.\n- **Surowiecki**, *The Wisdom of Crowds* (2004): conditions are **diversity of opinion**, **independence**, **decentralisation**, and an **aggregation** mechanism.\n- **Independence is fragile.** Lorenz, Rauhut, Schweitzer & Helbing, \"How social influence can undermine the wisdom of crowd effect\" (*PNAS* 108, 2011): letting participants see others' estimates reduced diversity without improving accuracy, and increased confidence (**the \"range reduction\" and \"confidence\" effects**). But Becker, Brackbill & Centola, \"Network dynamics of social influence in the wisdom of crowds\" (*PNAS* 114, 2017), found that in **decentralised** networks social influence *improved* accuracy, while centralised networks (with an influential hub) degraded it. So the effect of social information depends on **network structure** **[contested in details]**.\n- **Diversity versus ability.** Hong & Page, \"Groups of diverse problem solvers can outperform groups of high-ability problem solvers\" (*PNAS* 101, 2004), is widely cited for \"diversity trumps ability\". Abigail Thompson, \"Does Diversity Trump Ability? An Example of the Misuse of Mathematics in the Social Sciences\" (*Notices of the AMS* 61(9), 2014), argues the theorem is trivial and the simulation shows that *random* groups do well, not that diversity does. Responses (Singer, \"Diversity, Not Randomness, Trumps Ability\", *Philosophy of Science* 86, 2019; Kuehn in *Critical Review* 29, 2017) partly rehabilitate the result with better-specified diversity measures. This is **[contested]**; the popular slogan should not be cited as established. The robust, uncontroversial part is the **diversity prediction theorem** (Page 2007): collective squared error = average individual squared error − diversity (variance of predictions). It is an identity, so it guarantees that a crowd is never worse than its average member. It does *not* guarantee that diversity is available or independent.\n\n### 5.4 Superforecasters and the Good Judgment Project\n\n- Philip Tetlock, *Expert Political Judgment* (Princeton UP, 2005): ~284 experts, ~28,000 forecasts over two decades; experts on average barely beat simple extrapolation algorithms and did worse than they believed. **Foxes** (many small ideas, self-critical) outperformed **hedgehogs** (one big idea).\n- **Good Judgment Project** (2011–2015, IARPA ACE tournament). Mellers et al., \"Psychological strategies for winning a geopolitical forecasting tournament\" (*Psychological Science* 25, 2014): **training** (probabilistic reasoning, reference classes, averaging) improved Brier scores by roughly 10%; **teaming** helped; **tracking** the top ~2% into superforecaster teams produced the largest gains. Tetlock & Gardner, *Superforecasting* (2015). GJP beat the other ACE teams and reportedly intelligence analysts with access to classified data **[latter claim widely reported; unverified against primary source]**.\n- **Contested:** Hauenstein, Thomas, Illingworth & Dougherty, \"Rethinking the Role of Teams and Training in Geopolitical Forecasting\" (*Psychological Science*, 2025), reanalysed the data and argue that, once uncontrolled method variance is accounted for, the conclusions that teaming and training improved forecasting are not straightforward, raising the possibility that neither did. The Wharton Mack Institute has since run an adversarial collaboration on the question **[outcome not checked]**. The *existence* of stably better forecasters (tracking) is better supported than the *trainability* claims.\n- **Aggregation and extremising.** Baron, Mellers, Tetlock, Stone & Ungar, \"Two reasons to make aggregated probability forecasts more extreme\" (*Decision Analysis* 11(2), 2014): averaged probabilities are too close to 0.5 because (a) individuals shade toward 0.5 under uncertainty and (b) forecasters hold partly *non-overlapping* information, so the mean fails to combine it (Satopää et al., \"Combining multiple probability predictions using a simple logit model\", *IJF* 30, 2014). Pushing the aggregate outward in log-odds improves accuracy. The extremising factor must be **reduced when forecasters share information**, because shared information is already counted: extremising is a correction for *independence*, and applying it to correlated sources double counts.\n\n### 5.5 Proper scoring rules and the Brier score\n\n- **Brier score** (Brier, *Monthly Weather Review* 78, 1950): mean squared difference between probability forecasts and outcomes (0/1). Lower is better; always forecasting 0.5 on binary events scores 0.25. Murphy's decomposition (1973): Brier = reliability (calibration) − resolution + uncertainty.\n- A scoring rule is **strictly proper** if a forecaster maximises expected score only by reporting their true belief (Gneiting & Raftery, \"Strictly proper scoring rules, prediction, and estimation\", *JASA* 102, 2007). Brier (quadratic) and logarithmic scores are strictly proper. Linear scoring is not. Proper scoring rules are the formal version of \"confession must pay\": they make honest reporting of uncertainty the optimal strategy.\n\n### 5.6 Prediction markets\n\nArrow et al., \"The Promise of Prediction Markets\" (*Science* 320, 2008); Wolfers & Zitzewitz, \"Prediction Markets\" (*Journal of Economic Perspectives* 18, 2004). Market prices aggregate information from traders with money at stake; the Iowa Electronic Markets often beat polls. Known issues: favourite–longshot bias, thin markets, manipulation attempts (usually arbitraged away in liquid markets but not in thin ones), and prices reflecting not only probabilities but also risk preferences and budget constraints (Manski 2006). GJP found that prediction markets and team aggregates were competitive with one another **[details unverified]**.\n\n### 5.7 The FRI adversarial collaboration on AI risk\n\nRosenberg, Karger, Morris, Hickman, Hadshar, Jacobs & Tetlock, *Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration* (Forecasting Research Institute, March 2024; published version: \"Belief updating in AI-risk debates: Exploring the limits of adversarial collaboration\", *Risk Analysis*, 2025). Following the Existential Risk Persuasion Tournament (XPT), 22 participants ran in April–May 2023: 11 \"skeptics\" (9 superforecasters, 2 domain experts) and 11 \"concerned\" (domain experts, mostly referred from the AI-safety and EA community). Findings:\n\n- **Median P(AI-caused existential catastrophe by 2100)**: skeptics moved from about 0.10% to 0.12%; the concerned from about 25% to 20%. Essentially no convergence after 25–100 hours of engagement per participant.\n- The disagreement was **not due to misunderstanding**: participants could accurately summarise the other side's arguments.\n- **Differing evidential standards**: skeptics required empirical demonstrations of dangerous capabilities before assigning high probabilities; the concerned weighted theoretical, multi-step arguments more heavily.\n- **Worldview anchors**: skeptics expected gradual change and saw extinction as intrinsically hard to cause; the concerned reasoned from analogies (more intelligent species driving less intelligent ones extinct).\n- The most convergent short-term **crux** (whether AI systems demonstrate autonomous replication, resource acquisition and shutdown evasion by 2030, as tested by METR/ARC Evals) was expected to close only **about 5% of the gap** (≈1 percentage point of a ~20-point gap on a related question). Most of the disagreement concerned long-horizon dynamics that no near-term observation would resolve.\n\n**Why this matters for Deliberus.** It is the best available empirical evidence that, among highly capable and good-faith reasoners who have mapped each other's arguments, **residual disagreement can be about what counts as evidence**, not about the evidence. In the platform's terms this is a disagreement over **which critical questions a scheme should carry and what an adequate answer is**: the concerned side treats argument-from-mechanism with multi-step causal chains as strong, the skeptic side demands something like evidence-to-hypothesis from observations. A graph that renders each side's evidential standard as an attackable claim (rather than hard-coding one methodology ladder) can at least represent this residue. It also warns the convergence wager: full mutual comprehension did not produce convergence here.\n\n---\n\n## 6. Evidence for normative and evaluative claims\n\n### 6.1 Moral intuitions as evidence; reflective equilibrium\n\n- **Sidgwick**, *The Methods of Ethics* (1874/1907), Book III ch. 11: four conditions for a self-evident intuition to have the \"highest degree of certainty\": terms clear and precise; self-evidence ascertained by careful reflection; **mutual consistency** with other such propositions; and **consensus of experts** (\"the absence of such disagreement must remain an indispensable negative condition\"). Sidgwick thus built a *disagreement check* into intuitionism.\n- **Reflective equilibrium** (Goodman 1955 for induction; Rawls, *A Theory of Justice*, 1971; **wide** reflective equilibrium, Daniels, *Journal of Philosophy* 76, 1979): considered judgments at all levels of generality are adjusted against principles and against background theories until they cohere. Intuitions are **defeasible evidence**: they have initial credibility and can be revised. This is the most widely held methodological stance in normative ethics. It is also routinely criticised as conservative (it starts from existing judgments) and as unable to guarantee convergence across people with different starting points.\n- Whether intuitions are *evidence* at all (as opposed to beliefs or seemings) is **[contested]**: Williamson (*The Philosophy of Philosophy*, 2007) argues \"intuitions\" are just judgments and the evidence is their content; Cappelen (*Philosophy without Intuitions*, 2012) argues philosophers do not in fact rely on intuitions as evidence. Bealer and Huemer (phenomenal conservatism) defend intuitions as sui generis seemings with prima facie evidential force.\n\n### 6.2 Debunking arguments\n\n- **Street**, \"A Darwinian Dilemma for Realist Theories of Value\" (*Philosophical Studies* 127, 2006, 109–166): evolution has shaped our evaluative attitudes. A realist must either deny any relation between those forces and the independent evaluative truths (then agreement between our attitudes and the truths would be a coincidence, leading to skepticism) or posit a tracking relation (which is scientifically unmotivated, since the adaptive link explains our attitudes without reference to truth). Realist replies include \"third factor\" views (Enoch; Wielenberg), and a general minimalist reply that debunking generalises too far.\n- **Kahane**, \"Evolutionary Debunking Arguments\" (*Noûs* 45, 2011): global debunking presupposes objectivism and cannot be used selectively in normative ethics without self-undermining.\n- **Greene**, \"The Secret Joke of Kant's Soul\" (2008) and fMRI work (Greene et al., *Science* 293, 2001): deontological judgments are driven by fast emotional responses to \"personal\" harm, so they are **debunkable** as tracking morally irrelevant features (such as personal force). Critics: Berker, \"The Normative Insignificance of Neuroscience\" (*Philosophy & Public Affairs* 37, 2009), argues that the normative work is done by armchair premises about which features are morally relevant, not by the neuroscience. Early imaging findings also had reanalysis problems (McGuire et al., *JESP* 2009, found the original reaction-time effect driven by a few items) **[contested]**.\n\n**Structural point for the platform.** A debunking argument is an **undercutter**, in Pollock's sense: it attacks the link between an intuition and the claim it supports (\"this intuition is explained by a process that does not track moral truth\"), not the claim itself. The briefing lists \"no rebutting vs undercutting distinction typed\" as a gap. In the normative domain almost all serious attacks on intuition-based premises are undercutters, so the gap bites hardest here.\n\n### 6.3 Experimental philosophy's instability findings\n\n- **Order effects**: Schwitzgebel & Cushman, \"Expertise in Moral Reasoning? Order Effects on Moral Judgment in Professional Philosophers and Non-Philosophers\" (*Mind & Language* 27, 2012): professional philosophers showed order effects on trolley-type, action/omission and moral-luck cases **as large as non-philosophers'**, and the order of cases also influenced their endorsement of general principles such as the doctrine of double effect. Schwitzgebel & Cushman, \"Philosophers' biased judgments persist despite training, expertise and reflection\" (*Cognition* 141, 2015), found that neither reflection nor familiarity eliminated the effects.\n- **Framing and demographic variation**: Swain, Alexander & Weinberg (*Philosophy and Phenomenological Research* 76, 2008) on order effects in epistemic cases; Weinberg, Nichols & Stich (2001) on cultural variation in Gettier intuitions, which **did not replicate**: Machery et al., \"Gettier Across Cultures\" (*Noûs* 51, 2017), found Gettier intuitions robust across cultures.\n- **Replicability**: Cova et al., \"Estimating the Reproducibility of Experimental Philosophy\" (*Review of Philosophy and Psychology* 12, 2021): 40 studies from 2003–2015 replicated about 70% of the time, better than psychology's ~36–39% in the Open Science Collaboration (2015). So x-phi's *findings* are fairly robust; which findings show *instability of intuitions* versus *robustness* varies by case.\n\n**Consensus:** some moral intuitions are unstable under irrelevant manipulations (order, framing), expertise does not immunise against this, and others are robust across cultures and procedures. Instability is **case-specific**, not global. The practical upshot for a platform is that an intuition's evidential weight should depend on **whether it survives variation in presentation** (order, framing, perspective), which is testable.\n\n### 6.4 Thought experiments\n\nThought experiments are sometimes treated as evidence from imagined cases (Sorensen, *Thought Experiments*, 1992; Gendler, *Thought Experiment*, 2000). On Williamson's account they are counterfactual reasoning whose reliability is continuous with ordinary modal judgment. The main evidential worries are (a) the imagined case may import features the author did not intend, (b) intuitions about far-fetched cases may be unreliable (Parfit's worry about bizarre cases; Dennett's \"intuition pumps\"), and (c) the order and framing effects above. Thought experiments are best treated as **arguments with an intuition premise**, whose critical questions include \"is the case coherent?\", \"is the verdict stable under redescription?\", and \"does the intuition track the feature the argument needs?\"\n\n### 6.5 Public reason, deliberative standards, and value-laden science\n\n- **Public reason** (Rawls, *Political Liberalism*, 1993; \"The Idea of Public Reason Revisited\", 1997): in justifying coercive political decisions on constitutional essentials, citizens should offer reasons others could reasonably accept, not reasons that depend on comprehensive doctrines others reasonably reject. Critics: Wolterstorff, Eberle, and others argue this unfairly burdens religious citizens. **[contested]**\n- **Habermas** (*Between Facts and Norms*, 1992/1996): the discourse principle, under which valid norms are those all affected could agree to as participants in rational discourse. **Validity claims** are distinguished: **truth** (empirical), **normative rightness**, **sincerity** (of self-expression), and comprehensibility. That is a four-way distinction the platform should know. First-person testimony about one's own state raises a **sincerity** claim, which is redeemed by consistency of conduct over time, not by argument. Normative claims raise **rightness** claims, redeemed by argument about generalisable interests. Empirical claims raise **truth** claims, redeemed by evidence. Habermas's scheme says impact testimony is primarily a sincerity-claiming expressive act that becomes *input* to rightness discourse about whose interests are affected.\n- **Gutmann & Thompson**, *Democracy and Disagreement* (1996) and *Why Deliberative Democracy?* (2004): **reciprocity** (justify in terms others can accept), **publicity**, and **accountability**, and an **economy of moral disagreement** (seek the rationale that minimises rejection of the opposing position).\n- **Inductive risk** (Rudner, \"The Scientist qua Scientist Makes Value Judgments\", *Philosophy of Science* 20, 1953; Douglas, \"Inductive Risk and Values in Science\", *Philosophy of Science* 67, 2000; *Science, Policy, and the Value-Free Ideal*, 2009): because accepting a hypothesis risks error, and errors have non-epistemic consequences, **non-epistemic values legitimately set evidential thresholds** (how much evidence is enough), and even influence data characterisation and methodology choice. Douglas's example: classifying ambiguous rat liver slides in dioxin studies. Douglas distinguishes a legitimate **indirect** role for values (setting the threshold) from an illegitimate **direct** role (values as reasons to believe).\n- **Longino**, *Science as Social Knowledge* (1990) and *The Fate of Knowledge* (2002): **contextual empiricism**. Evidence bears on hypotheses only relative to background assumptions, so objectivity is a property of communities, secured by **transformative criticism** under four norms: recognised **venues** for criticism, **uptake** of criticism, **public standards**, and **tempered equality** of intellectual authority. This is arguably the most directly transferable philosophy-of-science framework for an argumentation platform: it locates objectivity in the *structure of criticism*, not in any individual's reliability. Its \"tempered equality\" is exactly the balance between no expert gate and not treating all voices as equal on all questions.\n\n### 6.6 Evidence that a policy works versus evidence that it is right\n\nThe GRADE **Evidence-to-Decision (EtD) frameworks** (Alonso-Coello et al., \"GRADE Evidence to Decision frameworks\", *BMJ* 353:i2016 and i2089, 2016; Moberg et al., *Health Research Policy and Systems* 16, 2018, for health-system and public-health decisions) separate a recommendation into criteria including: priority of the problem; desirable effects; undesirable effects; **certainty of the evidence**; **values** (how much people value the main outcomes, and the uncertainty or variability in that); balance of effects; resources required; cost-effectiveness; **equity**; **acceptability**; **feasibility**. Each criterion gets a judgment plus the research evidence and \"additional considerations\" behind it. Two structural features matter:\n\n1. **Certainty of evidence about effects** (GRADE) and **evidence about values** are separate rows. Evidence on values is often qualitative or preference-elicitation research, graded separately (for example with CERQual).\n2. **The recommendation is a judgment made by a panel across criteria**, not a function computed from them. The framework makes the judgment inspectable (each row has a stated basis) without mechanising it.\n\nThat is the practical answer to \"how do you handle normative evidence without pretending it is empirical\": **type the premise** (effect, value, equity, acceptability), grade each with the method appropriate to its type, and make the weighing a visible, attributable act. That maps directly onto Deliberus's argument-from-weighing scheme.\n\n### 6.7 What counts as a good reason in ethics versus evidence in science\n\nA defensible synthesis of the above:\n\n- **Empirical premises** have evidence in the ordinary sense: observations with likelihoods under competing hypotheses.\n- **Normative premises** have **reasons**: considerations whose force is assessed by coherence (reflective equilibrium), stability under variation (x-phi), resistance to debunking (Street, Greene), reciprocity and acceptability (Rawls, Gutmann & Thompson), and survival of transformative criticism (Longino). There is no oracle; there is a **discipline of criticism**.\n- **Mixed premises** (most policy claims) decompose into empirical and normative parts, and **inductive risk** is the principled bridge: normative stakes legitimately set how much empirical evidence is enough, but must not stand in for it.\n\n### 6.8 Is impact testimony evidence for an empirical premise, a normative premise, or neither?\n\nThe literature supports a three-part answer.\n\n1. **Empirical premises, directly.** \"This policy imposes cost C on people in position P\" is an empirical claim, and a report from someone in P is **testimonial evidence** for it, under Walton's position-to-know scheme. Its strength is governed by the critical questions of that scheme plus sample-representativeness (Táíwò, truth commissions). The surrogation result (Gilbert et al. 2009) and the disability paradox show that such reports can be **better evidence than outsiders' simulations** for claims about what a state is like.\n2. **Normative premises, indirectly.** Impact testimony is not itself evidence that a policy is wrong. It supplies the **inputs** normative reasoning needs: whose interests are affected and how (Habermas's \"all affected\"), what the *values* row of an EtD framework should contain, and whether a proposed justification passes reciprocity (can it be justified *to this person*, given what it costs them?). It can also serve as **evidence about intuitions**: a considered judgment formed from inside a position is a datum for reflective equilibrium that outsiders lack, as Wylie's contingent advantage would predict.\n3. **Neither, in one role.** In its **acknowledgment** role (the TRC's \"personal or narrative truth\"), testimony is not offered as evidence for any premise. It is a speech act whose uptake is owed as recognition. The platform should not force every testimony into an evidential role: doing so can itself be a hermeneutical injustice, recasting \"this is what happened to me\" as \"here is a data point for your argument.\"\n\nSo the answer to the briefing's contradiction (\"whether impact testimony is weighed as evidence\") is: **it is weighed as evidence for the empirical premise it bears on, it is an input to normative premises without being weighed as proof of them, and it can also be registered as an act of testimony that bears on nothing in the strength layer.** These are three different edges.\n\n---\n\n## 7. Trust and credibility calibration of sources\n\n### 7.1 Source credibility research\n\n- **Hovland and the Yale programme** (Hovland & Weiss, \"The Influence of Source Credibility on Communication Effectiveness\", *Public Opinion Quarterly* 15, 1951; Hovland, Janis & Kelley, *Communication and Persuasion*, 1953): high-credibility sources produced more immediate attitude change. Credibility was decomposed into **expertise** and **trustworthiness**. The **sleeper effect**, where a low-credibility message gains persuasive impact over time as the source is dissociated from the message, was reported; a meta-analysis (Kumkale & Albarracín, *Psychological Bulletin* 130, 2004) found it real but conditional, occurring mainly when the discounting cue arrives *after* the message and the message is initially impactful.\n- **Elaboration likelihood model** (Petty & Cacioppo, 1986): source credibility acts as a peripheral cue under low elaboration and can bias processing under high elaboration.\n\nThe sleeper effect has a direct platform implication: **provenance must stay attached to the claim at every point of display**, because a claim detached from its source is remembered without the discount.\n\n### 7.2 Epistemic vigilance\n\nSperber, Clément, Heintz, Mascaro, Mercier, Origgi & Wilson, \"Epistemic Vigilance\" (*Mind & Language* 25(4), 2010, 359–393). Humans depend massively on communication, which exposes them to misinformation, accidental or intentional. So they evolved mechanisms of **epistemic vigilance**, directed at both the **source** (competence and benevolence) and the **content** (coherence with background beliefs, and the quality of any argument offered). Mercier & Sperber's argumentative theory (*The Enigma of Reason*, 2017) extends this: reasoning evolved partly as vigilance, to evaluate others' arguments and to persuade. Mercier, *Not Born Yesterday* (2020), argues humans are **not gullible** on the whole; mass persuasion mostly fails, and the problem is more often excessive distrust of reliable sources than excessive trust.\n\nThe two dimensions (competence and benevolence) correspond to Walton's critical questions for position to know (\"is *a* in a position to know?\" and \"is *a* honest?\") and to Hovland's expertise and trustworthiness. Three independent literatures converge on the same two-factor decomposition of source reliability, and it should be treated as settled.\n\n### 7.3 Bayesian models of testimony and source reliability\n\n- **Bovens & Hartmann**, *Bayesian Epistemology* (Oxford UP, 2003): Bayesian networks for witnesses of **partial reliability** (with some probability the witness is a truth-teller, otherwise a randomiser). Key results: (a) agreement among **independent** partially reliable witnesses can confirm strongly even when each is weak; (b) the impact of coherence on confidence depends on reliability, and no coherence measure can be both a complete ordering and truth-conducive in all cases (their impossibility result, related to Olsson, *Against Coherence*, 2005); (c) **dependence among witnesses** sharply reduces the confirmatory value of agreement. The value of a report depends on the witness's reliability *and on how surprising the report is given priors*.\n- **Hahn, Harris & Corner**, \"Argument content and argument source: An exploration\" (*Informal Logic* 29(4), 2009, 337–367): in Bayesian terms, **source reliability and content strength combine multiplicatively, not additively.** A strong argument from an unreliable source and a weak argument from a reliable one cannot be traded off linearly. Harris & Hahn (2009) found that participants rated arguments supported by more agreeing witnesses as stronger even when individual reliability was held constant, which matches Bayesian norms *if witnesses are independent*. Hahn & Oaksford, \"The rationality of informal argumentation: A Bayesian approach to reasoning fallacies\" (*Psychological Review* 114, 2007), show that argument from ignorance, circularity, and slippery slope can each be strong or weak under Bayesian analysis depending on parameters: they are not fallacies by form. Hahn & Hornikx, \"A normative framework for argument quality: argumentation schemes with a Bayesian foundation\" (*Synthese* 193, 2016), link Walton schemes to Bayesian parameters: **each critical question corresponds to a parameter** of the likelihood ratio.\n- **Harris, Hahn, Madsen & Hsu**, \"The appeal to expert opinion: quantitative support for a Bayesian network approach\" (*Cognitive Science* 40, 2016): people's judgments of expert testimony track a Bayesian network model with separate expertise and trustworthiness nodes.\n- **Biased sources.** Bayesian treatments distinguish a source's **reliability** (does their report covary with the truth?) from its **bias** (do they report H more often regardless of truth?). A source biased toward H who reports H provides a weak likelihood ratio; the same source reporting *against* H provides a strong one (the \"admission against interest\" in law, **statement against interest**, Federal Rules of Evidence 804(b)(3)). The informative quantity is always P(report | H) / P(report | ¬H), which requires modelling what the source would have said in each case.\n\nThis is the formal answer to the briefing's gap \"answers to CQs are counted, not weighed\": Hahn & Hornikx show CQ answers should set parameters of a likelihood ratio, and Hahn, Harris & Corner show source and content combine multiplicatively.\n\n### 7.4 Crowdsourced testimony: manipulation and design\n\n- **Community Notes** (formerly Birdwatch; Wojcik et al., \"Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation\", arXiv:2210.15723, 2022). Notes are scored by **matrix factorisation** on the rater × note matrix: each rater and note get a latent \"viewpoint\" factor, and a note's helpfulness is its **intercept after the viewpoint factor is removed**. So a note rates as helpful only when raters who usually disagree both rate it helpful. The **bridging** criterion makes brigading by one faction ineffective by construction, because one faction's ratings load on the factor, not the intercept. Weaknesses reported in later work: a low fraction of notes reach \"helpful\" status; notes often appear after most views of a viral post; the one-dimensional factor can miss manipulation along other axes; and there are concerns about sustainability of rater participation (e.g., \"Threats to the sustainability of Community Notes on X\", arXiv:2510.00650) **[figures not checked]**.\n- **Wikipedia**: **Verifiability** (content must be attributable to a reliable published source), **No Original Research** (no novel analysis or synthesis, including \"synthesis of published material that advances a position\" — WP:SYNTH), and **Reliable Sources** (sources with editorial oversight and a reputation for fact-checking; self-published sources generally excluded except for claims about the author). Wikipedia deliberately **does not accept first-person testimony as a source**, even from the subject. That is a sharp contrast with Deliberus's aims. Wikipedia made this choice because it cannot verify identity or positional access and chose to outsource verification to publishers. It works for an encyclopaedia of *settled published knowledge* and systematically excludes *unpublished positional knowledge*: a hermeneutical-injustice-shaped gap in the world's largest reference work, which is itself documented (the gender gap in biographies, for example).\n\nThe two systems represent the two coherent answers to crowdsourced testimony: **Wikipedia verifies sources externally and excludes the unverifiable; Community Notes verifies agreement across viewpoints and excludes the one-sided.** Deliberus wants to admit positional testimony that neither would admit, so it needs a third mechanism, which §8 proposes.\n\n---\n\n## 8. Most transferable to a claim-graph argumentation platform like Deliberus\n\n### 8.1 Where positional/impact testimony enters: three different edges\n\nReplace \"impact testimony is weighed as evidence\" (ambiguous) with three typed roles:\n\n1. **Testimonial evidence for an empirical premise** (\"policy X costs people in position P in way C\"). Edge: SUPPORTS or ATTACKS via **argument from position to know** (or witness testimony). This edge **moves strength**, subject to its critical questions.\n2. **Input to a normative or weighing premise** (the *values* row of an argument from weighing; \"can this be justified to people in P?\"). Edge: SUPPORTS a *value premise* or populates a weighing's option/cost set. It moves strength only through the weighing scheme's own CQs, never as proof of the normative conclusion. It should be typed so that a reader can see it is a stakeholder input, not a derivation.\n3. **Registered testimony (acknowledgment)**. Minted as reported speech via the non-propagating REPORTS edge (\"A testified that…\"). It moves no strength and exists to be heard and to be available for (1) and (2). The platform should **not force** every testimony into role 1, because recasting \"this happened to me\" as a data point is itself a documented harm (TRC's four truths; hermeneutical injustice).\n\nThe same utterance can occupy several roles; each is a separate edge and is separately attackable.\n\n### 8.2 Critical questions for positional testimony\n\nBuild on Walton's position to know (Is *a* in a position to know? Is *a* honest? Did *a* assert it?) with CQs drawn from the literatures above. Each should be rendered as an answerable claim:\n\n1. **Position**: Is the testifier in the position claimed, and does that position give access to *this* claim (Wylie: advantage is relative to a question)?\n2. **Kind of first-person claim** (§4.5): Is it about a present state, a retrospective evaluation, a cause, a counterfactual, or a satisfier? Weight falls steeply in that order.\n3. **Representativeness** (Táíwò; truth commissions): Is this position's testimony a representative sample, or are the positions most affected the least able to testify here (smothering; being-in-the-room privilege)?\n4. **Independence** (Bovens & Hartmann; extremising): Are corroborating testimonies independent, or do they share an origin (the same community post, the same organiser, the same prompt)?\n5. **Sincerity/consistency** (Habermas): Is the testimony consistent with the testifier's other contributions and conduct over time?\n6. **Rendering fidelity** (choice blindness): If the testimony was confirmed against a machine rendering, was it the testifier's own words first?\n7. **Scale shift** (Ubel et al.): For comparative claims about badness, is the testifier using the same scale as the outsider claim it contradicts?\n8. **Presentation stability** (x-phi): For intuition-bearing testimony, does the judgment survive reordering and reframing?\n9. **Stake** (source bias, §7.3): Does the testifier benefit from the claim being accepted, and would they have said this regardless? A statement *against* interest is strong evidence.\n\n### 8.3 Avoiding both credibility deficit and credibility excess\n\n- **Deficit** is avoided by giving positional testimony a *presumptive* standing for claims within its access (burden on the challenger), by never gating contribution on credentials, and by the surrogation finding: display positional testimony *beside* outsider forecasts about the same state, because it is systematically better evidence for them and people systematically undervalue it (Gilbert et al. 2009).\n- **Excess** is avoided by **never letting testimony close a question** (advantage, not authority), and by recognising that credibility is **comparative** (Medina): any ranking that lifts one class of testimony lowers another's relative visibility. So tier-based weights should affect *strength*, never *existence or visibility*, which is already the corpus's \"weight rises with tier; nothing gated on tier\" line, now with a philosophical warrant.\n- **Sole witness**: label it (as proposed), and add CQ 3. A sole witness is the modal situation for smothered testimony, so the label must read as an *invitation for corroboration*, never as a demotion.\n- **Tiers** (unverified / self-attested with record / positionally corroborated): the literature supports them with one amendment. \"Positionally corroborated\" should require **independent** corroboration, because correlated corroboration is the brigading channel (§7.3–7.4). Borrow Community Notes' logic: corroboration from people in the same position who *otherwise disagree* is far stronger than from a single faction.\n\n### 8.4 Representing expert judgment without an expert gate\n\n- **Weight calibration, not credentials** (Cooke; IDEA; Hemming et al.: status and experience predicted performance poorly). Where the platform ever gets ground truth (resolved empirical claims, forecasting questions), score contributors' probability statements with a **strictly proper scoring rule** (Brier or log) and let a *per-domain calibration record* become a claim attached to the contributor. It remains attackable and never gates.\n- **Expert opinion as a scheme, not a tier.** Walton's appeal to expert opinion carries CQs (expertise, field, what was said, consistency with other experts, evidence basis). Treat the \"expert opinion 0.30\" ladder rung as an **unanswered-CQ default**, not a property of the claim.\n- **Independence first** (IDEA, Lorenz et al. 2011). Where the platform elicits estimates, collect private judgments *before* showing others', then share reasons, then allow private revision, then aggregate mechanically. Show aggregates in log-odds and extremise only when estimates are independent.\n- **Evidential-standards disagreement is data** (FRI 2024). When two sides disagree about whether a class of argument (multi-step mechanism versus empirical demonstration) is strong, that disagreement should be mintable as a claim about the scheme's CQs, rather than resolved by a fixed methodology ladder.\n\n### 8.5 Handling normative evidence without pretending it is empirical\n\n- **Type the premise, then choose the grading method** (EtD): effect claims get effect-evidence CQs; value claims get normative CQs (coherence, presentation stability, debunkability, reciprocity); weighing claims keep the weighing visible and attributed.\n- **Add undercutting as a typed attack.** Debunking arguments, bias arguments, and \"this intuition is order-dependent\" are undercutters. Typing them separately from rebutters is especially important for normative premises, where almost all serious attacks are undercutters.\n- **Inductive risk as a bridge claim.** When a normative stake is used to set how much empirical evidence is enough (\"given the harm of being wrong, we need more than X\"), mint it as its own claim linking the two. That keeps Douglas's legitimate indirect role for values and blocks the illegitimate direct one.\n- **Longino's four norms as platform properties**: venues (every claim can be attacked), uptake (an attack visibly changes something), public standards (schemes and CQs are published), and tempered equality (no gate, but calibration and position can weight). Most of these are already built. The one to watch is **uptake**: an attack that never changes the display is a venue without uptake.\n\n### 8.6 Reconciling \"testimony is evidence for a premise, never a verdict\" with epistemic-injustice concerns\n\nThe rule is defensible and should be kept. The literature supports it: Wylie (advantage, not privilege), Táíwò (deference fails as practice), Schwitzgebel and Nisbett & Wilson (self-reports are fallible), and Longino (objectivity lives in criticism, not in any one voice). But it is exposed to three epistemic-injustice objections, each of which has a design answer:\n\n1. **\"Never a verdict\" can become testimonial quieting** if testimony is admitted but never visibly *moves* anything. Answer: testimony must be able to move strength on the empirical premise it bears on, and that movement must be visible. Being admitted but inert would be quieting.\n2. **Fixed CQs can become hermeneutical injustice** if the available schemes and CQs have no slot for what the testifier is trying to say (the harassment-before-the-concept case). Answer: `does_not_fit` with a free-text reason on the testimony itself, and the reframe as a first-class contribution. The corpus's \"frame gap gets a contribution path\" ruling is exactly the hermeneutical-justice remedy.\n3. **Smothering is invisible** to every instrument, because it is testimony not given. Answer: it can only be addressed by **lowering the cost and risk of testifying** (attribution control, visibility defaults, no public demotion of sole witnesses) and by **measuring representation** (which positions affected by a claim have contributed at all), the per-cluster ingestion-balance instrument applied to positions.\n\nOne refinement to the conviction wording is recommended by the evidence (§4.3–4.5): replace \"incorrigible about the felt experience\" with **\"authoritative about one's present refusal or endorsement, presumptive about one's present state, fallible about fine structure, and weak about causes, counterfactuals and satisfiers\"**. This keeps the ethical protection while fitting Schwitzgebel, experience sampling and choice blindness. (This is a proposal and has not been ruled on.)\n\n### 8.7 Concrete buildables, ranked by fit\n\n1. **Split impact testimony into three edges** (empirical evidence / normative input / non-propagating registration). Resolves a live contradiction in the corpus.\n2. **Per-kind first-person CQ** (present state / retrospective / cause / counterfactual / satisfier) on position-to-know edges whose source is a first-person report.\n3. **Independence CQ and shared-origin node** on corroboration, with Community-Notes-style cross-faction corroboration as the strongest tier.\n4. **Undercutter as a typed attack**, prioritised for normative premises.\n5. **Likelihood-ratio reading of CQ answers** (Hahn & Hornikx): each CQ maps to a parameter, and source and content combine multiplicatively, which replaces counting answers with weighing them.\n6. **Surrogation display**: next to an outsider's forecast of what a state is like, show testimony from people in that state.\n7. **Private-first elicitation** (IDEA) for any numerical estimates the platform collects.\n8. **Evidential-standard claims**: allow a side to assert \"this scheme's CQs should be X\" as an attackable claim, so FRI-style disagreements about evidence become representable.\n9. **Representation instrument**: for a contested claim, which affected positions have contributed? Counting only, no model.\n\n---\n\n## Unverified and contested points, collected\n\n- Small, Loewenstein & Slovic 2007 donation figures ($2.38 / $1.14 / $1.43) and Lee & Feeley 2016 r ≈ .05 are from memory. **[unverified]**\n- Johansson & Hall 2005 exact detection rates (13% concurrent; under a quarter overall) are from secondary summaries. **[unverified against primary]**\n- The authorship of \"Nisbett and Wilson (1977) Revisited\" (*Social Cognition* 34(3), 2016) was not checked. **[unverified]**\n- South African TRC number of statements (~21,000). **[approximate]**\n- Galton mean recomputation (1,197 lb). **[unverified]**\n- GJP beating intelligence analysts with classified data. **[widely reported, primary not checked]**\n- Typical share of poorly calibrated experts in Cooke's database. **[unverified proportion]**\n- Community Notes display-rate and timeliness figures. **[not checked]**\n- Mack Institute adversarial collaboration on GJP teams/training: outcome not checked.\n- Contested: credibility excess as injustice; performance vs equal weighting; Hong–Page; GJP training/teaming (Hauenstein et al. 2025); vividness effect; impact-bias magnitude; Greene's debunking; Schwitzgebel's scope; public reason's fairness to religious citizens; the standpoint achievement thesis.\n\n---\n\n## References\n\n**Epistemic injustice and standpoint**\n- Alcoff, L. M. (1991). The problem of speaking for others. *Cultural Critique* 20, 5–32.\n- Charlton, J. I. (1998). *Nothing About Us Without Us: Disability Oppression and Empowerment*. University of California Press.\n- Collins, P. H. (1990). *Black Feminist Thought*. Unwin Hyman.\n- Davis, E. (2016). Typecasting, tokenism, and tokenism: credibility excess as epistemic injustice. *Hypatia* 31(3), 485–501.\n- Dotson, K. (2011). Tracking epistemic violence, tracking practices of silencing. *Hypatia* 26(2), 236–257. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1527-2001.2011.01177.x\n- Fricker, M. (2007). *Epistemic Injustice: Power and the Ethics of Knowing*. Oxford UP.\n- Harding, S. (1991). *Whose Science? Whose Knowledge?* Cornell UP.\n- Hartsock, N. (1983). The feminist standpoint. In Harding & Hintikka (eds.), *Discovering Reality*. Reidel.\n- Medina, J. (2011). The relevance of credibility excess in a proportional view of epistemic injustice. *Social Epistemology* 25(1), 15–35. — (2013). *The Epistemology of Resistance*. Oxford UP.\n- Táíwò, O. O. (2020). Being-in-the-room privilege: elite capture and epistemic deference. *The Philosopher* 108(4). https://www.thephilosopher1923.org/post/being-in-the-room-privilege-elite-capture-and-epistemic-deference — (2022). *Elite Capture*. Haymarket.\n- UN Convention on the Rights of Persons with Disabilities (2006), Art. 4(3).\n- Wylie, A. (2003). Why standpoint matters. In Figueroa & Harding (eds.), *Science and Other Cultures*. Routledge. https://philarchive.org/archive/WYLWSM\n- Young, I. M. (2000). *Inclusion and Democracy*. Oxford UP.\n\n**Lived experience, qualitative evidence, participation**\n- Brett, J., et al. (2014). Mapping the impact of patient and public involvement on health and social care research: a systematic review. *Health Expectations* 17(5), 637–650. https://onlinelibrary.wiley.com/doi/10.1111/j.1369-7625.2012.00795.x\n- FDA (2009). *Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims*.\n- France, E. F., et al. (2019). Improving reporting of meta-ethnography: the eMERGe reporting guidance. *BMC Medical Research Methodology* 19:25.\n- HRDAG, Peru project. https://hrdag.org/peru/ ; Truth and Reconciliation Commission of Peru, *Final Report* (2003).\n- Israel, B. A., et al. (1998). Review of community-based research. *Annual Review of Public Health* 19, 173–202.\n- Kosmala, M., et al. (2016). Assessing data quality in citizen science. *Frontiers in Ecology and the Environment* 14(10), 551–560.\n- Lewin, S., et al. (2015). Using qualitative evidence in decision making for health and social interventions (GRADE-CERQual). *PLOS Medicine* 12(10), e1001895. https://journals.plos.org/plosmedicine/article?id=10.1371%2Fjournal.pmed.1001895\n- Lewin, S., et al. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series (and papers 3–6). *Implementation Science* 13(Suppl 1). https://pubmed.ncbi.nlm.nih.gov/29384079/\n- Mokkink, L. B., et al. (2010). The COSMIN checklist. *Quality of Life Research* 19, 539–549.\n- Noblit, G. W., & Hare, R. D. (1988). *Meta-Ethnography: Synthesizing Qualitative Studies*. Sage.\n- Staniszewska, S., et al. (2017). GRIPP2 reporting checklists. *BMJ* 358:j3453.\n- Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. *BMC Medical Research Methodology* 8:45.\n- Truth and Reconciliation Commission of South Africa (1998). *Final Report*, vol. 1, ch. 5.\n\n**Case studies and anecdotes**\n- Flyvbjerg, B. (2006). Five misunderstandings about case-study research. *Qualitative Inquiry* 12(2), 219–245. https://journals.sagepub.com/doi/10.1177/1077800405284363\n- Guyatt, G., et al. (1986). Determining optimal therapy — randomized trials in individual patients. *NEJM* 314, 889–892.\n- Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. *Psychological Review* 80, 237–251.\n- Lee, S., & Feeley, T. H. (2016). The identifiable victim effect: a meta-analytic review. *Social Influence* 11(3), 199–215.\n- Small, D. A., Loewenstein, G., & Slovic, P. (2007). Sympathy and callousness. *Organizational Behavior and Human Decision Processes* 102, 143–153. https://www.sciencedirect.com/science/article/abs/pii/S0749597806000057\n- Taylor, S. E., & Thompson, S. C. (1982). Stalking the elusive \"vividness\" effect. *Psychological Review* 89, 155–181.\n- Vohra, S., et al. (2015). CONSORT extension for reporting N-of-1 trials (CENT). *BMJ* 350:h1738.\n\n**Introspection and first-person reports**\n- Albrecht, G. L., & Devlieger, P. J. (1999). The disability paradox. *Social Science & Medicine* 48, 977–988. https://pubmed.ncbi.nlm.nih.gov/10390038/\n- Gilbert, D. T., Killingsworth, M. A., Eyre, R. N., & Wilson, T. D. (2009). The surprising power of neighborly advice. *Science* 323, 1617–1619. https://www.science.org/doi/10.1126/science.1166632\n- Gilbert, D. T., et al. (1998). Immune neglect. *JPSP* 75, 617–638.\n- Hall, L., Johansson, P., & Strandberg, T. (2012). Lifting the veil of morality: choice blindness and attitude reversals on a self-transforming survey. *PLoS ONE* 7(9), e45457.\n- Hurlburt, R. T., & Schwitzgebel, E. (2007). *Describing Inner Experience? Proponent Meets Skeptic*. MIT Press.\n- Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to detect mismatches between intention and outcome in a simple decision task. *Science* 310, 116–119.\n- Kahneman, D., Krueger, A. B., Schkade, D., Schwarz, N., & Stone, A. (2004). A survey method for characterizing daily life experience: the day reconstruction method. *Science* 306, 1776–1780.\n- Killingsworth, M. A., & Gilbert, D. T. (2010). A wandering mind is an unhappy mind. *Science* 330, 932.\n- Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know. *Psychological Review* 84(3), 231–259. https://web.mit.edu/curhan/www/docs/Articles/15341_Readings/Social_Cognition/Nisbett_Wilson_1977_Telling_more_than_we_can_know.pdf\n- Redelmeier, D. A., & Kahneman, D. (1996). Patients' memories of painful medical treatments. *Pain* 66, 3–8.\n- Schkade, D. A., & Kahneman, D. (1998). Does living in California make people happy? *Psychological Science* 9, 340–346.\n- Schwitzgebel, E. (2008). The unreliability of naive introspection. *Philosophical Review* 117(2), 245–273. https://faculty.ucr.edu/~eschwitz/SchwitzAbs/Naive.htm — (2011). *Perplexities of Consciousness*. MIT Press.\n- Ubel, P. A., Loewenstein, G., & Jepson, C. (2003). Whose quality of life? *Quality of Life Research* 12, 599–607. https://link.springer.com/article/10.1023/A:1025119931010\n- Wilson, T. D., & Gilbert, D. T. (2003). Affective forecasting. *Advances in Experimental Social Psychology* 35, 345–411; (2005) *Current Directions in Psychological Science* 14, 131–134.\n\n**Expert judgment and forecasting**\n- Arrow, K. J., et al. (2008). The promise of prediction markets. *Science* 320, 877–878.\n- Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., & Ungar, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. *Decision Analysis* 11(2), 133–145. https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015---two-reasons-to-make-aggregated-probability-forecasts_1.pdf\n- Becker, J., Brackbill, D., & Centola, D. (2017). Network dynamics of social influence in the wisdom of crowds. *PNAS* 114, E5070–E5076. https://www.pnas.org/doi/10.1073/pnas.1615978114\n- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. *Monthly Weather Review* 78, 1–3.\n- Colson, A. R., & Cooke, R. M. (2017). Cross validation for the classical model of structured expert judgment. *Reliability Engineering & System Safety* 163, 109–120. https://www.sciencedirect.com/science/article/pii/S0951832017302090\n- Colson, A. R., & Cooke, R. M. (2018). Expert elicitation: using the classical model to validate experts' judgments. *Review of Environmental Economics and Policy* 12(1), 113–132. https://www.journals.uchicago.edu/doi/full/10.1093/reep/rex022\n- Cooke, R. M. (1991). *Experts in Uncertainty*. Oxford UP.\n- Cooke, R. M., Marti, D., & Mazzuchi, T. (2021). Expert forecasting with and without uncertainty quantification and weighting: what do the data say? *International Journal of Forecasting* 37, 378–387. https://www.sciencedirect.com/science/article/pii/S0169207020300959\n- Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. *Management Science* 9, 458–467.\n- Galton, F. (1907). Vox populi. *Nature* 75, 450–451.\n- Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. *JASA* 102, 359–378.\n- Hauenstein, C. E., Thomas, R. P., Illingworth, D. A., & Dougherty, M. R. (2025). Rethinking the role of teams and training in geopolitical forecasting. *Psychological Science*. https://doi.org/10.1177/09567976241266481\n- Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C. (2018). A practical guide to structured expert elicitation using the IDEA protocol. *Methods in Ecology and Evolution* 9, 169–180. https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12857\n- Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. *PNAS* 101, 16385–16389.\n- Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. *PNAS* 108, 9020–9025. https://www.pnas.org/doi/10.1073/pnas.1008636108\n- Mellers, B., et al. (2014). Psychological strategies for winning a geopolitical forecasting tournament. *Psychological Science* 25, 1106–1115. https://journals.sagepub.com/doi/10.1177/0956797614524255\n- O'Hagan, A., et al. (2006). *Uncertain Judgements: Eliciting Experts' Probabilities*. Wiley.\n- Rosenberg, J., Karger, E., Morris, A., Hickman, M., Hadshar, R., Jacobs, Z., & Tetlock, P. E. (2024). *Roots of Disagreement on AI Risk*. Forecasting Research Institute. https://forecastingresearch.org/ai-adversarial-collaboration ; published as \"Belief updating in AI-risk debates: exploring the limits of adversarial collaboration\", *Risk Analysis* (2025). https://onlinelibrary.wiley.com/doi/10.1111/risa.70023\n- Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool. *International Journal of Forecasting* 15, 353–375.\n- Singer, D. J. (2019). Diversity, not randomness, trumps ability. *Philosophy of Science* 86, 178–191.\n- Surowiecki, J. (2004). *The Wisdom of Crowds*. Doubleday.\n- Tetlock, P. E. (2005). *Expert Political Judgment*. Princeton UP. — & Gardner, D. (2015). *Superforecasting*. Crown.\n- Thompson, A. (2014). Does diversity trump ability? *Notices of the AMS* 61(9), 1024–1030. https://www.ams.org/notices/201409/rnoti-p1024.pdf\n\n**Normative evidence**\n- Alonso-Coello, P., et al. (2016). GRADE Evidence to Decision (EtD) frameworks. *BMJ* 353:i2016 and i2089.\n- Berker, S. (2009). The normative insignificance of neuroscience. *Philosophy & Public Affairs* 37, 293–329.\n- Cova, F., et al. (2021). Estimating the reproducibility of experimental philosophy. *Review of Philosophy and Psychology* 12, 9–44. https://link.springer.com/article/10.1007/s13164-018-0400-9\n- Daniels, N. (1979). Wide reflective equilibrium and theory acceptance in ethics. *Journal of Philosophy* 76, 256–282.\n- Douglas, H. (2000). Inductive risk and values in science. *Philosophy of Science* 67, 559–579. — (2009). *Science, Policy, and the Value-Free Ideal*. Pittsburgh UP.\n- Greene, J. D., et al. (2001). An fMRI investigation of emotional engagement in moral judgment. *Science* 293, 2105–2108.\n- Gutmann, A., & Thompson, D. (1996). *Democracy and Disagreement*. Harvard UP.\n- Habermas, J. (1996). *Between Facts and Norms*. MIT Press.\n- Kahane, G. (2011). Evolutionary debunking arguments. *Noûs* 45, 103–125.\n- Longino, H. (1990). *Science as Social Knowledge*. Princeton UP. — (2002). *The Fate of Knowledge*. Princeton UP.\n- Machery, E., et al. (2017). Gettier across cultures. *Noûs* 51, 645–664.\n- Moberg, J., et al. (2018). The GRADE EtD framework for health system and public health decisions. *Health Research Policy and Systems* 16:45. https://link.springer.com/article/10.1186/s12961-018-0320-2\n- Rawls, J. (1971). *A Theory of Justice*; (1993). *Political Liberalism*. Harvard/Columbia UP.\n- Rudner, R. (1953). The scientist qua scientist makes value judgments. *Philosophy of Science* 20, 1–6.\n- Schwitzgebel, E., & Cushman, F. (2012). Expertise in moral reasoning? *Mind & Language* 27, 135–153. — (2015). Philosophers' biased judgments persist despite training, expertise and reflection. *Cognition* 141, 127–137.\n- Sidgwick, H. (1907). *The Methods of Ethics*, 7th ed., III.xi.\n- Street, S. (2006). A Darwinian dilemma for realist theories of value. *Philosophical Studies* 127, 109–166. https://link.springer.com/article/10.1007/s11098-005-1726-6\n- Swain, S., Alexander, J., & Weinberg, J. (2008). The instability of philosophical intuitions. *Philosophy and Phenomenological Research* 76, 138–155.\n- Williamson, T. (2007). *The Philosophy of Philosophy*. Blackwell.\n\n**Source credibility and testimony**\n- Bovens, L., & Hartmann, S. (2003). *Bayesian Epistemology*. Oxford UP.\n- Hahn, U., Harris, A. J. L., & Corner, A. (2009). Argument content and argument source: an exploration. *Informal Logic* 29(4), 337–367. https://informallogic.ca/index.php/informal_logic/article/download/2903/2340\n- Hahn, U., & Hornikx, J. (2016). A normative framework for argument quality: argumentation schemes with a Bayesian foundation. *Synthese* 193, 1833–1873. https://link.springer.com/article/10.1007/s11229-015-0815-0\n- Hahn, U., & Oaksford, M. (2007). The rationality of informal argumentation. *Psychological Review* 114, 704–732.\n- Harris, A. J. L., Hahn, U., Madsen, J. K., & Hsu, A. S. (2016). The appeal to expert opinion: quantitative support for a Bayesian network approach. *Cognitive Science* 40, 1496–1533.\n- Hovland, C. I., & Weiss, W. (1951). The influence of source credibility on communication effectiveness. *Public Opinion Quarterly* 15, 635–650.\n- Kumkale, G. T., & Albarracín, D. (2004). The sleeper effect in persuasion: a meta-analytic review. *Psychological Bulletin* 130, 143–172.\n- Mercier, H. (2020). *Not Born Yesterday*. Princeton UP.\n- Sperber, D., Clément, F., Heintz, C., Mascaro, O., Mercier, H., Origgi, G., & Wilson, D. (2010). Epistemic vigilance. *Mind & Language* 25(4), 359–393. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-0017.2010.01394.x\n- Walton, D., Reed, C., & Macagno, F. (2008). *Argumentation Schemes*. Cambridge UP.\n- Wikipedia policies: Verifiability (WP:V), No original research (WP:NOR, WP:SYNTH), Reliable sources (WP:RS).\n- Wojcik, S., et al. (2022). Birdwatch: crowd wisdom and bridging algorithms can inform understanding and reduce the spread of misinformation. arXiv:2210.15723. https://arxiv.org/abs/2210.15723\n"}