{"path":"research/first-pipeline-run-analysis.md","content":"# First Pipeline Run: Analysis and Improvement Plan\n\n**Date**: March 28, 2026\n**Source text**: Wikipedia \"Universal basic income\" — Economics/Costs + Philosophy/Morality sections (~4600 chars)\n**Pipeline**: Three-pass hybrid (scout → focused extraction → cross-structure analysis)\n**LLM**: Gemini 2.0 Flash via google-genai SDK\n**Output**: `data/extractions/ubi_wikipedia/`\n\n---\n\n## Results Summary\n\n| Metric | Count | Comparison to virtual hand-extraction |\n|--------|-------|--------------------------------------|\n| Argument structures (Pass 1) | 12 | — (hand-extraction didn't use this pass) |\n| Atomic claims (Pass 2) | 44 | 22 (hand-extraction was less granular) |\n| Relationships (Pass 3) | 29 | 10 (pipeline found more fine-grained connections) |\n| Contested concepts (Pass 3) | 3 | 4 (pipeline missed \"fairness\") |\n\n---\n\n## Pass 1 (Scout) — Assessment: Strong, but needs a structural adjustment\n\n**What worked:**\n- 12 argument structures identified — good coverage of a ~4600 char text\n- Type classification mostly correct: Straubhaar's \"tax reform\" correctly tagged `definitional`, \"UBI is socially just\" and \"UBI fairness\" correctly `normative`, \"UBI is a human right\" correctly `value_premise`\n- Meta section captures key tensions accurately: \"Economic growth vs degrowth\", \"Fairness and deservingness\", \"Definition of work and contribution\"\n- Contested concepts flagged early (economic growth, basic income, fairness, responsibility, deserving)\n\n**What needs fixing:**\n\n**The scout is mixing \"argument discovery\" with \"argument summarization.\"** Example: the \"UBI fosters risk-taking\" argument has its conclusion rewritten as a compressed multi-hop causal chain: \"guaranteed financial security → increased risk-taking → culture of inventiveness + entrepreneurial spirit.\" That's not a single conclusion — it's a compressed mini-graph. When Pass 1 produces these \"bundled\" conclusions, Pass 2 is forced either to over-decompose or accept non-atomic claims.\n\n**Fix**: Pass 1 should output **argument spans + candidate thesis sentences** (pointing to the text), not LLM-authored paraphrased conclusions. Think of Pass 1 output as indices/anchors into the source text, not a rewriting of the arguments. The scout should POINT, not PARAPHRASE.\n\n---\n\n## Pass 2 (Focused Extraction) — Assessment: Detailed, but has classification and granularity issues\n\n**What worked:**\n- 44 atomic claims — comprehensive decomposition\n- Good decontextualization — most claims pass the \"stranger test\" (understandable without the source)\n- Source spans preserved, linking claims back to original text\n- Four-type classification applied to every claim\n\n**Issue 1: \"Empirical\" type is overused**\n\nMany claims tagged `empirical` are actually predictions, conjectures, or feasibility claims:\n\n| Claim | Tagged as | Should be |\n|-------|----------|-----------|\n| \"An increased willingness to take risks would create a culture of inventiveness\" | empirical | prediction/forecast |\n| \"A basic income could increase economic growth\" | empirical | causal hypothesis |\n| \"By 2020, it would have been nearly a trillion dollars less expensive\" | empirical | counterfactual |\n| \"Could indeed be financed through redistribution\" | empirical | feasibility claim |\n\n**Root cause**: The four types (empirical/normative/definitional/value_premise) conflate the KIND of claim with its EPISTEMIC STATUS. A claim can be empirical in kind (about facts) but predictive in status (not yet verified).\n\n**Solution: Add epistemic modality as an orthogonal annotation, not a new type.** Keep the four types but add:\n\n```\nepistemic_status: observed | causal_hypothesis | prediction | counterfactual | feasibility\nevidence_type: citation | quantitative | anecdotal | none\n```\n\nThis directly fixes the overuse without exploding the taxonomy. \"UBI would create a culture of inventiveness\" stays `empirical` in type but gets `prediction` in status, which enables appropriate confidence scoring (predictions should carry lower confidence than observations).\n\n**Issue 2: Over-decomposition**\n\nSome claims are split too finely:\n- \"The cost of a basic income is one of the biggest questions in the public debate\" and \"The cost of a basic income is one of the biggest questions in research\" — these are trivially parallel. A reader would not treat them as distinct commitments.\n- This over-decomposition cascades into Pass 3, where relationships exist ONLY because of the artificial split (e.g., \"supports public debate question\" and \"supports research question\" as separate edges that add no insight).\n\n**Root cause**: Partly induced by Pass 1's bundled conclusions (forcing Pass 2 to split), partly the prompt not distinguishing \"rhetorical parallelism\" from \"genuinely distinct claims.\"\n\n**Fix**: Add a \"merge if only rhetorical parallelism differs\" heuristic. LLM-judged: \"Would a reader treat these as distinct commitments?\" If not, merge.\n\n**Issue 3: Confidence scores are undiscriminating**\n\nAll claims scored 0.8-0.95. This provides no useful signal. A verified statistic ($200 billion less expensive) and a speculative prediction (culture of inventiveness) receive nearly identical confidence.\n\n**Fix**: Confidence should be computed FROM epistemic modality + evidence type, not guessed by the LLM in isolation. Observed + citation = high. Prediction + no evidence = low. The formula can be simple:\n\n```\nbase_confidence = {observed: 0.9, causal_hypothesis: 0.6, prediction: 0.4, counterfactual: 0.5, feasibility: 0.5}\nevidence_bonus = {citation: +0.1, quantitative: +0.05, anecdotal: 0, none: -0.1}\nconfidence = base_confidence[status] + evidence_bonus[evidence]\n```\n\n**Issue 4: value_premise vs normative distinction needs documentation**\n\nThe boundary between these types will drift unless explicitly defined:\n- **value_premise** = asserts a value/rights principle (\"every person deserves a subsistence floor\")\n- **normative** = asserts what should be done GIVEN values (\"UBI should be implemented because it is a human right\")\n\nA normative claim DECOMPOSES INTO value premises + empirical premises. A value premise is a currently-undecomposed foundation (but per the \"No Copout Axioms\" principle, always decomposable further).\n\n---\n\n## Pass 3 (Cross-Structure) — Assessment: The star of the pipeline, with one structural issue\n\n**What worked excellently:**\n\n- **29 relationships** with good variety across all types: supports (most common), attacks, qualifies, reframes, decomposes_into\n- **The Straubhaar REFRAME correctly detected** (claim 14→23, strength 0.9) — redefining UBI as \"tax reform\" is a genuinely novel relationship that traditional argument mining doesn't capture\n- **DECOMPOSES_INTO** used correctly for premise→conclusion chains (e.g., the \"naive calculation\" argument: definition of UBI as a tax system → gross costs are misleading → naive multiplication is wrong)\n- **QUALIFIES** relationships used appropriately (rich/poor distribution qualifying the definition of UBI as a tax system)\n- **Contested concepts are genuine signal**, not pattern-matching:\n  - \"cost\": monetary expense vs net financial impact (two genuinely different senses)\n  - \"work\": gainful employment vs all value-creating activity (THE definitional dispute)\n  - \"deserving\": earned through labor vs superficial classification (directly opposed framings)\n\n**Issue: \"Fairness\" not flagged as contested**\n\nThe virtual hand-extraction identified \"fairness\" as the CORE contested concept (two definitions: responsibility-based vs contribution-recognition-based). The pipeline missed it despite having all the raw material: \"Responsibility is central to fairness\" (opponents) vs implied fairness-as-non-discrimination (proponents).\n\n**Root cause**: The pipeline detects **polysemy** (word has two dictionary senses) but misses **normative criterion contestation** (people share the word but disagree on which PRINCIPLE defines it). \"Fairness\" doesn't have two dictionary senses — it has competing criteria for what satisfies it.\n\n**Fix**: Detect two subtypes of conceptual contestedness:\n1. **Semantic contestedness**: different senses/definitions (cost, work, deserving) — detectable via \"X means Y\" patterns\n2. **Criterion contestedness**: different principles compete for the same concept (fairness, justice) — detectable via \"X is central to fairness\", \"fair because...\", \"unfair because...\" patterns showing incompatible GROUNDS for the same evaluative term\n\n**Issue: Pass 3 is doing two jobs at once**\n\nRelationship detection and contested concept detection interfere with each other. They require different analytical lenses: relationships need clean entailment/attack edges between atomic claims, while contestedness requires comparing incompatible senses or normative criteria across the whole claim set.\n\n**Fix**: Split into Pass 3a (relationships) and Pass 3b (contested concepts). Pass 3b can look at the full graph (claims + types + modalities + relationships) rather than just the flat claim list.\n\n**Issue: Some relationship explanations reverse premise/conclusion**\n\nExample: from_claim 1 → to_claim 0 with explanation \"Getting higher-skilled jobs supports people while they invest in education\" — this reverses the actual logic (education leads to better jobs, not the other way around). This is a symptom of Pass 1's paraphrased conclusions cascading into Pass 3's reasoning.\n\n**Issue: decomposes_into edges appearing in Pass 3**\n\nDecomposition should be settled in Pass 2. If Pass 3 is still finding decomposition relationships, the claim inventory from Pass 2 isn't stable enough for graph-building. This suggests Pass 2 needs to be more thorough about premise chains.\n\n---\n\n## Comparison: Pipeline vs Claimify (Quantified)\n\nOn the same ~4600 chars of UBI text:\n\n| Dimension | Claimify (estimated) | Deliberus Pipeline | Ratio |\n|-----------|---------------------|-------------------|-------|\n| Claims extracted | 3-4 (only empirical, verifiable) | 44 (all types) | ~11x |\n| Types captured | 1 (factual only) | 4 (empirical, normative, definitional, value_premise) | 4x |\n| Relationships | 0 (no relation detection) | 29 | ∞ |\n| Contested concepts | 0 (no concept tracking) | 3 | ∞ |\n| Normative claims | 0 (discarded at Selection) | Multiple (N1-N6 in virtual run) | ∞ |\n| Definitional claims | 0 (discarded) | Multiple (Straubhaar reframe, work definition) | ∞ |\n\nThe wider mouth is validated on real data: Deliberus extracts an order of magnitude more material than Claimify from the same text, including the normative, definitional, and value-premise claims that are the most interesting for deliberation.\n\n---\n\n## The \"Aha Moment\" — Validated\n\nThe pipeline's output, even in this first rough run, makes visible what the source text hides:\n\n**\"They're not really arguing about whether UBI is affordable. They're arguing about what 'fairness' means. And they're using the same word ('deserving', 'work', 'cost') to mean different things.\"**\n\nThis is the Deliberus thesis in action: structural clarity dissolves false disagreements. The contested concepts detection — even with its current limitations — surfaces the semantic layer that drives the debate. With the normative criterion detection added, the \"fairness\" dispute would surface too.\n\n---\n\n## Improvement Priorities (Ordered by Impact)\n\n### 1. Add epistemic modality (highest impact)\nAdd `epistemic_status` and `evidence_type` as orthogonal annotations. Fixes empirical overuse, enables meaningful confidence, improves relationship quality.\n\n### 2. Fix Pass 1 to point rather than paraphrase\nScout should output argument spans + candidate thesis sentences, not rewritten multi-hop conclusions. Reduces cascading over-decomposition in Pass 2.\n\n### 3. Add criterion contestedness detection\nSplit contested concepts into semantic (polysemy) and criterion (competing principles for evaluative terms). Catches \"fairness\" and similar normative disputes.\n\n### 4. Split Pass 3 into 3a (relationships) + 3b (contested concepts)\nSeparates the two analytical lenses. Contested concepts can use the full graph (types, modalities, relationships) as input.\n\n### 5. Reduce over-decomposition\nAdd \"merge if only rhetorical parallelism\" heuristic. LLM-judged: \"Would a reader treat these as distinct commitments?\"\n\n### 6. Compute confidence from modality + evidence\nReplace LLM-guessed confidence with a formula based on epistemic status and evidence type.\n\n---\n\n## What Works and Should Be Preserved\n\n- The three-pass architecture is fundamentally sound\n- The four-type classification (with value_premise renamed per \"No Copout Axioms\") works on real data\n- The REFRAMES relationship type is a genuinely novel detection that traditional argument mining misses\n- The contested concepts detection produces real signal, not noise\n- Source span preservation enables human verification\n- Each pass persisted independently enables inspection and iteration\n- The stranger test decontextualization is mostly working\n\n---\n\n## Cross-References\n\n- [extraction-pipeline-design.md](extraction-pipeline-design.md) — pipeline architecture decisions\n- [extraction-virtual-run.md](extraction-virtual-run.md) — hand-extraction benchmark (22 claims, 10 relationships, 4 concepts)\n- [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md) — four node types, concept tracking\n- [embeddings-tension-and-ai-slop.md](embeddings-tension-and-ai-slop.md) — automation vs authenticity\n- [../vision.md](../vision.md) §No Copout Axioms — value premises are depth-relative\n- Source code: `deliberus/extraction/scout.py`, `focused.py`, `cross_structure.py`\n- Pipeline output: `data/extractions/ubi_wikipedia/pass_*.json`\n"}