{"path":"research/long-form-sources-and-meta-analysis-weighing.md","content":"# Long-Form Sources and Meta-Analysis Weighing: One Shape, Two Names\n\n*Aug 13, 2026. Founder brought a two-hour video review of a thousand studies and asked two things: that content like it be analyzable \"both as a whole after submitting, and as its constituent parts\", and whether weighing evidence and interpreting meta-analyses had been discussed before — adding \"this whole video could be seen as a meta analysis I guess?\"*\n\n*Both answers turn out to be yes, and they are the same answer. The meta-analysis question was settled in April, in the founder's own words. The video question is that settled question arriving in a new medium, which is why it feels familiar.*\n\n---\n\n## 1. The prior art, quoted rather than re-derived\n\n`scheme-bounded-decomposition-and-evidence-as-subgraph.md` (April 2026) opens with the founder's own framing:\n\n> \"I want people and AI agents surfing around in Deliberus to see exactly what the best minds on our planet think in detail and what the best evidence shows — evidence that also is decomposed into its constituent parts/arguments. This is similar to the conviction I have that value premises are often decomposable and evidence is often too. What actually constitutes un-decomposable, more atomic evidence is an open question to me.\"\n\nAnd immediately after, the meta-analysis case by name:\n\n> \"uploading a meta-analysis as a PDF to Deliberus as support of a claim should be allowed. But then that PDF should be decomposed as far as possible — maximum granularity that is helpful to the rest of humanity.\"\n\nThat document then sketches the decomposition concretely — a meta-analysis resolving into individual studies, each into its methodology, each methodology into its assumptions, with `[CQ] Effect size heterogeneity? → \"I²=73%, high heterogeneity\"` hanging off it — and states the principle that makes it cheap: **\"A meta-analysis IS text. Run the pipeline on it.\"** Elsewhere it classifies a meta-analysis as *a causal evidence claim*, which is what licenses treating it as a claim with children rather than as an attachment.\n\nSo the conceptual work exists and does not need repeating. What follows is what has been *built* since, what the video adds that the April thinking did not cover, and the one place where the built behaviour is now demonstrably wrong.\n\n## 2. What is already built, which is more than it feels like\n\nVerified in code rather than recalled:\n\n| Piece | Where | State |\n|---|---|---|\n| Methodology ladder feeding QBAF base weight | `extraction/models.py` `METHODOLOGY_BASE_WEIGHT` | **Live.** `meta_analysis` 0.70 · `systematic_review` 0.68 · `rct` 0.65 · `quasi_experimental` 0.55 · `observational` 0.45 · `case_study` 0.38 · `anecdotal` 0.32 · `expert_opinion` 0.30 |\n| Tier overrides an explicit base weight | `graph/qbaf.py` `compute_edge_strength` | **Live**, and tested |\n| Quantitative claim metadata | `extraction/models.py` | **Live**: `effect_size`, `effect_size_type`, CI bounds, `sample_size`, **`study_count`**, **`heterogeneity`** |\n| That metadata reaching the synthesis prompt | `truth_graph._evidence_metadata_suffix` | **Live** — emits `[rct; cohens_d=0.4; CI=0.1-0.7; N=240; studies=12; heterogeneity=0.73]` |\n| Exact source quote per claim | `AtomicClaim.source_span` | **Live** — every claim carries the stretch of source it came from |\n| Mother-claim decomposition with transitive strength | `DECOMPOSES_INTO` + QEM | **Live** |\n| Audio → text | `POST /transcribe` (Gemini, `inline_data`) | **Live**, used for voice input |\n| PDF read natively, not text-scraped | Gemini `inline_data`, no pymupdf | **Live** |\n\n**So `heterogeneity` and `study_count` are already extracted, already reach the model, and are already displayed — and they do not affect anything.** They are recorded facts, not weights. A meta-analysis of twelve studies with I² = 0.73 gets exactly the same 0.70 base weight as a meta-analysis of forty studies with I² = 0.05.\n\nThat is the gap, and it is narrower and more concrete than \"we should think about meta-analyses.\"\n\n**⚠ Superseded three days later, on the diagnosis rather than the observation (2026-08-16).** The measurement above holds — those two fields are inert. What is wrong is the sentence that follows from it in §7 below, which reads the inertness as *a missing weighting formula*. `scheme-bounded-decomposition-and-evidence-as-subgraph.md` § What evidence currently *is* names this doc's status note explicitly and answers it: **it is a property that should be a claim.** `heterogeneity: 0.73` is a number nobody can argue with; *\"the twelve studies disagree substantially with each other\"* is attackable, decomposable, and can carry evidence of its own. Mint it with an attack edge on the parent and the shipped strength computation handles it with **no new formula and no new weights**. So the *goal* of recommendation 1 survives intact — heterogeneity should move strength — while its *method* does not, and the cheaper route was invisible from inside the weighting frame. The same reading applies to `sample_size` (a contestable claim about generalisability) and to `methodology_tier` (a claim about what that design can establish), which makes it the general answer rather than a fix for one field.\n\n## 3. The video is the same shape with a third level\n\nThe specimen: *\"I analyzed 1,000 Health Studies: Here are 10 Things I Learned\"* (Physionic / Nicolas Verhoeven, PhD — a channel whose whole format is breaking down primary literature for lay readers).\n\nIts structure, taken from the author's own public per-lesson posts rather than from the video body:\n\n- **The whole**: one aggregate claim about what a body of literature shows.\n- **Ten parts**: numbered lessons, each a separate substantive claim on a different topic — resistant starch and visceral fat, omega-3 and muscle strength, a brain receptor responding to an amino acid, vitamin K2 in liver-cancer care, microbiome intervention in oncology, and so on.\n- **Each part's own parts**: the studies beneath each lesson.\n\n**And the evidential standing differs sharply between lessons, stated by the author himself.** One rests on several randomized controlled trials. Another rests on mechanistic work while clinical evidence remains thin — the author's own hedge is that the mechanism is compelling but the clinical research is limited. Another rests on early-phase oncology trials. In the project's own vocabulary those are three different methodology tiers inside one source.\n\n**This is what makes whole-source extraction actively wrong here, not merely coarse.** The pipeline assigns methodology at the *source* level — `list_extractions_from_db` computes a `dominant_type` per extraction, and the truth-graph design says the base weight of a new edge comes from \"the methodology tier of the source extraction.\" Feed this video in as one blob and ten claims of wildly different standing inherit one strength. The RCT-backed lesson and the mechanism-only lesson come out looking equally supported.\n\nThat is a flattening mechanism, and it is a *new member of a family this project has been cataloguing all week*: extraction flattens by paraphrasing opposing claims into near-identical text; a synthesis flattens by dropping a side of a conflict; **an aggregate source flattens by averaging the evidential standing of its parts.** Each needs its own instrument, because each destroys something the others preserve.\n\nThe clean statement: **a source is not the unit of evidential quality. A claim is.** Methodology belongs on the claim, and the source-level tier is at best a prior.\n\n## 4. What \"as a whole and as its parts\" requires\n\nFour things, in dependency order. None of them is a rewrite.\n\n**A segment is a first-class thing between source and claim.** Today the graph has Source → Claim. A long-form source has an intermediate: a chapter, a lesson, a section, a segment. `Argument` already plays a near-identical role in extraction output (claims are grouped by argument structure), which suggests the cheapest path is to map segments onto argument groups rather than to introduce a new node type. Worth checking before adding anything: the ontology's one-node-type discipline is load-bearing, and a new node type needs to justify itself against the existing grouping.\n\n**Methodology moves to the claim.** Per-claim `methodology_tier` already exists on the model. What has to change is the *edge-strength derivation*, which currently reaches for the source's tier. This is an ontology change to shipped QBAF semantics and therefore a founder decision, not an implementation detail.\n\n**Provenance needs a locator, not just a quote.** `source_span` gives the exact text a claim came from. For a two-hour video, the useful locator is a timestamp — the thing that lets a reader verify in eight seconds instead of ninety minutes. A transcript with timings makes this mechanical: the span maps back to a time offset. For a PDF the analogue is a page. **Neither exists today**, and the video case is where its absence hurts most, because scanning a video for an unlocated quote is far more expensive than scanning a document.\n\n**Ingestion needs a fetch step the product does not have.** Audio-to-text is already live via `/transcribe`. The missing piece is only *getting* the audio, which is a fetch-and-extract problem, not an intelligence problem.\n\n## 5. Weighing: five specific things the built ladder gets wrong\n\nRecorded as candidates, all of them founder decisions because they change shipped QBAF semantics.\n\n**A flat top tier is too generous.** `meta_analysis: 0.70` treats every meta-analysis alike. A 2025 study in *Research Synthesis Methods* coded 1,000 meta-analyses across ten disciplines and found that 57% contained dependent effect sizes, and of those, **53% analysed them as if independent** — a practice that inflates significance. Medical meta-analyses in that sample were often built on only three to five studies. So the tier that sits at the top of the ladder is the tier whose internal quality varies most, and the ladder cannot see any of it. `study_count` and `heterogeneity` are already extracted and would carry most of the signal.\n\n**Heterogeneity is contestation, and it is currently silent.** High I² means the pooled studies disagree with each other. An aggregate over disagreeing studies is precisely the shape this project treats as a flattening alarm everywhere else — and here it arrives pre-quantified by the researchers themselves. **A meta-analysis with high heterogeneity should read as a weighing claim with a live dispute inside it, not as strong evidence.** The April sketch already had this as a critical question; nothing consumes it.\n\n**Mechanistic evidence has no tier at all.** The ladder runs from meta-analysis to expert opinion with nothing for \"the mechanism is well characterised, the clinical outcome is not.\" The video's own hedges make this concrete, and a mechanism claim mapped onto `observational` or `expert_opinion` is mis-tiered in both directions — stronger than opinion about its mechanism, weaker than an RCT about its outcome. This may be an argument that *one* tier per claim is the wrong shape, and that a claim needs a tier per thing-it-is-claiming.\n\n**Aggregation of an aggregate is not addressed.** A video reviewing a thousand studies is a third-order object: a person's reading of many meta-analyses which each pool many trials. Where does its aggregate claim get its strength — from its own tier, or from the propagated strength of its ten lessons? QEM already propagates through `DECOMPOSES_INTO`, so **the honest answer may be that an aggregate source should have no independent tier at all and should inherit only what its decomposition supports.** That is a clean and testable position.\n\n**Publication bias, replication status, and funding conflicts have no representation.** They are among the most decision-relevant properties of a study and appear nowhere in the model. They are also each a critical question, which is the natural home for them.\n\n## 6. The video could not be transcribed here, and why that matters more than it looks\n\n`yt-dlp` installs and runs fine in this container. YouTube refuses it: *\"Sign in to confirm you're not a bot\"*, the datacenter-IP wall, which needs cookies exported from a signed-in browser — a human-only credential of exactly the same class as the Google OAuth session that blocks live extraction from cloud sessions. The public oEmbed endpoint gave title and author; the author's own posts gave the per-lesson structure this analysis needed. **The structure was the useful part, and it came from public sources rather than from the video body.**\n\nTwo consequences worth carrying:\n\n**The local command is one line** (`yt-dlp --write-auto-subs --skip-download --cookies-from-browser <browser> <url>`), so this is a five-minute local-session step, not a project.\n\n**And a full third-party transcript must not land in `sources/`.** That tree is git-tracked and becomes readable on GitHub when the repo flips public, so committing a verbatim transcript of someone else's video would publish their content. The precedent already exists and is the right one: **PDFs are stored in Postgres as bytes and never in git.** A transcript is a PDF-shaped object, not a `sources/` specimen. Short attributed quotes in analysis are a different matter and are what extraction produces anyway.\n\n## 7. What this changes about the roadmap\n\nNothing here is decided. Ranked by evidence-over-effort:\n\n1. ~~**Consume `heterogeneity` and `study_count` in edge strength.**~~ **Superseded 2026-08-16 — read the note at the end of §2 before acting on this.** The observation stands and the prescription does not: mint the property as a claim and attach it with an attack edge, which needs no weighting change at all. This entry is kept rather than deleted because the correction is the useful part — an inert number read as a missing formula, when it was a missing node.\n2. **Move methodology from source to claim in the edge-strength derivation.** The mis-tiering it currently causes is demonstrated rather than hypothetical, by a source the founder brought.\n3. **A locator alongside `source_span`** — timestamp for audio and video, page for documents. Cheap at extraction time, expensive to retrofit, and the thing that makes verification affordable for long sources.\n4. **Segment-level structure**, checked against the existing argument grouping before any new node type is added.\n5. **Video ingestion**, which is only a fetch step away given that transcription already ships.\n\nThe one-line version, which is also the reason the founder's two questions were one question: **an aggregate hides the variance of its parts, and Deliberus exists to expose exactly that.** A meta-analysis and a two-hour review video are the same object at different scales, and the project already has the mechanism — the mother claim with children whose strength flows upward. What it lacks is the willingness to let an aggregate's *stated* internal disagreement, which researchers hand over as a number, actually count against it.\n\n---\n\n**See also**: [scheme-bounded-decomposition-and-evidence-as-subgraph.md](scheme-bounded-decomposition-and-evidence-as-subgraph.md) (the April prior art, with the founder's framing verbatim and the meta-analysis decomposition sketch) · [truth-graph-evidence-system.md](truth-graph-evidence-system.md) (methodology-adjusted QEM as designed and shipped) · [self-similar-decomposition-and-claim-ontology.md](self-similar-decomposition-and-claim-ontology.md) (the mother-claim pattern) · [synthesis-build-plan.md](synthesis-build-plan.md) (the other flattening mechanisms in this family) · `../extraction-pipeline.md` (what the pipeline accepts today)\n\n**Sources for §5**: Hedges et al. / *Research Synthesis Methods* (2025), \"What can we learn from 1,000 meta-analyses across 10 different disciplines?\" — the dependence-handling and study-count figures. Video metadata via YouTube oEmbed; per-lesson structure via the author's public posts. Read depth: abstract and highlights for the *RSM* paper, not the full text.\n"}