{"path":"research/extraction-improvement-research.md","content":"# Extraction Pipeline Improvement Research\n## State-of-the-Art Argument Mining Techniques (2025–2026)\n\n**Date**: March 28, 2026\n**Researcher**: Claude Sonnet 4.6\n**Scope**: Academic and engineering research across 7 domains to guide improvement of the Deliberus three-pass extraction pipeline\n**Current pipeline**: Gemini 2.0 Flash → three passes (scout/extract/cross-structure) → FalkorDB → 35 claims, 12 relationships, 4 contested concepts from ~4600 chars Wikipedia UBI text\n\n---\n\n## Summary of Key Actionable Findings\n\nBefore the deep dives: these are the findings with the highest immediate impact on the Deliberus pipeline.\n\n1. **Self-consistency voting is worth doing** — A 7-point macro-F1 gain from majority-voting across five diverse reasoning traces (IBM-Rank benchmark). Run Pass 2 three times at temperature=0.7 and take 2/3 agreement. Cost: 3x API calls, but Gemini 3 Flash is cheap enough.\n\n2. **Chain-of-thought rationales should be part of the extraction prompt** — The reasoning trace, not the final label, accounts for most of the performance gain in LLM argument mining. Ask the model to explain WHY it classified a claim as normative before emitting the label.\n\n3. **Fine-tuned models beat prompting on known benchmarks but fail on new domains** — The ACL 2025 \"learns datasets not arguments\" finding means that for a novel task like Deliberus (four-type classification of argumentative claims), prompt-based extraction with a large general LLM is likely MORE robust than a fine-tuned specialized model.\n\n4. **Use `instructor` + Pydantic for structured output** — More reliable than raw `response_mime_type: \"application/json\"`. Automatic validation and retry on parse failure. Gemini 3 Flash SDK has a known schema-ordering bug that breaks CoT; `instructor` works around it.\n\n5. **MArgE architecture is directly relevant** — Build multi-LLM voting into Pass 2: three Gemini 3 Flash calls, take 2/3 consensus. This is precisely what MArgE does (multiple LLMs vote on argumentative structure) and what Claimify's majority voting does (3x same LLM).\n\n6. **Add `epistemic_status` annotation** — This solves the \"empirical overuse\" problem identified in the first run. It's already under-researched in NLP (most papers ignore it), making it a genuine Deliberus differentiator.\n\n7. **For criterion contestedness** — No NLP paper directly addresses it. The closest technique is the \"incompatible grounds\" detection heuristic: find evaluative terms (\"fairness\", \"justice\") co-occurring with incompatible normative criteria using pattern matching + LLM judgment.\n\n---\n\n## 1. Argument Mining SOTA (2025–2026)\n\n### The paradigm shift: from supervised classifiers to LLM-driven pipelines\n\nThe most important finding from the 2025 literature is a fundamental paradigm shift documented in the comprehensive survey **\"Large Language Models in Argument Mining: A Survey\"** (arXiv:2506.16383, Jun–Nov 2025, ~250 papers reviewed). The field has moved from pipelines of supervised, task-specific classifiers (fine-tuned BERT variants on specific datasets) to prompt-driven, retrieval-augmented, and reasoning-oriented paradigms.\n\nKey shift: \"Even naïve declarative prompts ('Is the following sentence a claim?') enable GPT-3 to match feature-rich baselines on component segmentation and stance detection without gradient updates.\" This was early evidence. By 2025, chain-of-thought prompting has taken it further: the reasoning trace, not the final label, accounts for most of the performance gain.\n\n**The generalization crisis in fine-tuned AM** (ACL 2025): A critical finding from \"Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments\" (arXiv:2505.22137, ACL 2025) is directly relevant to Deliberus. Evaluating four transformer models across 17 English sentence-level datasets, the paper found that state-of-the-art models rely on lexical shortcuts tied to content words — they learn dataset-specific cues, not true argument understanding. Performance drops markedly on unseen datasets. For Deliberus (a novel four-type classification with normative/definitional/value claims), this means: a fine-tuned model trained on existing AM datasets (PE, AbstRCT, CDCP) would have poor transferability. Large general LLMs with structured prompting are more robust for novel tasks.\n\n### COLING 2025: Fine-tuned LLMs as SOTA on known benchmarks\n\n\"Argument Mining with Fine-Tuned Large Language Models\" (ACL Anthology: 2025.coling-main.442) fine-tuned LLaMA-3, LLaMA-3.1, Gemma-2, Mistral, Phi-3, Qwen-2 on the PE, AbstRCT, and CDCP datasets, modeling AM sub-tasks as text generation tasks. Results achieved SOTA across all AM sub-tasks and datasets. **Implication**: Fine-tuning on a Deliberus-specific annotated corpus would be the path to highest precision, but requires building that corpus first. The three-pass pipeline generating data that gets human-corrected is the right path to this.\n\n### IJCNN 2025: Autoregressive end-to-end architecture\n\n\"End-to-End Argument Mining through Autoregressive Argumentative Structure Prediction\" (arXiv:2510.16363, accepted IJCNN 2025) proposes AASP: a conditional language model that jointly solves Argument Component Identification, Span Boundary Pairing, Type Labeling, and Relation Classification in a single autoregressive pass. Achieves average Micro-F1 improvement of 3.3% and 3.81% on AAE and AAE-FG datasets, with strongest gains in relational tasks (+4.17/5.74% for ARI and ARC).\n\n**Actionable for Deliberus**: The autoregressive constraint enforcement — using constrained generation to enforce structural consistency — is a technique worth exploring for Pass 2. Rather than free-form JSON generation, constrain the decoder to produce structurally valid argument trees.\n\nAlso notable: \"Argument Mining as a Text-to-Text Generation Task\" (arXiv:2603.23949) achieves Component-F1 of 80.15 at essay level and Relation-F1 of 61.19 using FLAN T5-XXL, eliminating task-specific postprocessing. The framing of argument mining as seq2seq generation rather than classification has matured.\n\n### Normative vs. empirical classification\n\nThe comprehensive LLM argument mining study (arXiv:2603.19253, Feb 2026) \"A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2\" tests advanced prompting strategies — Chain-of-Thought, prompt rephrasing, voting, and certainty-based classification — on argument classification corpora. This is directly relevant to Pass 2's four-type classification problem. The finding is that CoT + voting > CoT alone > direct classification. No existing dataset has normative/value_premise as a target class — Deliberus is pioneering this space.\n\n**Gap in the literature**: No published paper specifically addresses the normative/empirical/definitional/value_premise four-way classification that Deliberus uses. The field focuses on claim/premise detection and support/attack relation classification. This is a genuine research gap that the Deliberus pipeline can contribute to.\n\n### Real-world F1 scores\n\nOn real-world (not curated benchmark) text, argument mining performance is significantly lower than benchmark numbers suggest:\n- Legal argument mining (Czech legal data): 82.6% macro-F1 for span detection, 77.5% for type classification\n- Relation classification (Mixtral-8x7B): 81% micro-F1 on structured debate datasets\n- On unstructured online comments: lower (see the multi-agent debate paper below)\n- The benchmark collapse finding suggests: headline numbers (80%+ F1) reflect dataset-specific pattern learning, not generalizable argument understanding\n\nFor Deliberus's Wikipedia/Reddit use case, expect effective performance in the 60–75% range for claim classification and 55–70% for relation detection on first run, improving significantly with prompt iteration and self-consistency voting.\n\n---\n\n## 2. Structured Output from LLMs\n\n### The reliability landscape in 2026\n\n**Function calling > structured output mode > free-text-then-parse** — this ranking has become the consensus in 2025.\n\n- **Function calling** (\"tool use\") is the most reliable path because the model is trained specifically to populate function parameters. Well-defined function signatures with typed, named parameters + docstrings produce the most consistent output.\n- **Native structured output** (`response_mime_type: \"application/json\"` with a JSON Schema) is reliable for simple schemas but has documented issues with complex nested schemas and reasoning tasks.\n- **Free-text-then-parse** is fragile for production use; justifiable only for rapid prototyping.\n\n### Gemini-specific known issues\n\nFrom the Gemini API GitHub issues and developer blogs, active issues as of late 2025 / early 2026:\n\n1. **Schema key ordering not preserved** in the google-genai Python SDK. The order of keys in your Pydantic model or dict schema is not guaranteed in the output. This matters for chain-of-thought because if you ask the model to reason in `reasoning_trace` BEFORE filling `claim_type`, but the schema delivers them in the wrong order, the model may fill `claim_type` before reasoning. **Workaround**: Use `instructor` library (see below), which wraps the call and enforces schema ordering via prompt engineering.\n\n2. **Function calling + structured output incompatibility** in Gemini 2.5: \"Function calling with a response mime type 'application/json' is unsupported\" when tool call history is present in the message. **Workaround**: Use one or the other per request, not both.\n\n3. **`additionalProperties` rejected by SDK** until November 2025 update. SDK client-side validation rejected schemas containing `additionalProperties: false` even though the API supported it. Fixed in November 2025 but may affect older SDK versions.\n\n4. **Decreased quality with constrained decoding** on complex generation tasks. Using strict JSON Schema constraining can reduce output quality compared to unconstrained generation with post-processing. For complex argument extraction with long reasoning traces, consider: generate reasoning in unconstrained text first, then extract structured data in a second call.\n\n### The `instructor` library — recommended solution\n\n`instructor` (Python, 3M+ monthly downloads, 11k GitHub stars) wraps any LLM provider with Pydantic model validation and automatic retry. It handles the schema ordering issue, validates output against the Pydantic schema, and retries with the validation error message fed back to the model. Key advantages for Deliberus:\n\n- **Automatic retry with error context**: If the model returns `claim_type: \"descriptive\"` (not a valid enum), instructor feeds \"validation error: 'descriptive' is not in ['empirical', 'normative', 'definitional', 'value_premise']\" back to the model for self-correction.\n- **Semantic validation**: Beyond type checking, allows custom validators (e.g., \"if `claim_type = 'empirical'` then `epistemic_status` must not be 'None'\").\n- **Provider-agnostic**: Works with Gemini, Anthropic, OpenAI — same code.\n- **Chain-of-thought integration**: Pydantic field with `description=\"Think step by step before answering\"` reliably induces reasoning before the answer field.\n\n**Implementation sketch for Pass 2**:\n\n```python\nfrom pydantic import BaseModel, Field\nfrom enum import Enum\nimport instructor\nimport google.generativeai as genai\n\nclass ClaimType(str, Enum):\n    empirical = \"empirical\"\n    normative = \"normative\"\n    definitional = \"definitional\"\n    value_premise = \"value_premise\"\n\nclass EpistemicStatus(str, Enum):\n    observed = \"observed\"\n    causal_hypothesis = \"causal_hypothesis\"\n    prediction = \"prediction\"\n    counterfactual = \"counterfactual\"\n    feasibility = \"feasibility\"\n\nclass AtomicClaim(BaseModel):\n    reasoning: str = Field(description=\"Step-by-step reasoning for classification\")\n    text: str = Field(description=\"Decontextualized claim text (stranger test)\")\n    claim_type: ClaimType\n    epistemic_status: EpistemicStatus\n    confidence: float = Field(ge=0, le=1)\n    source_span: str\n\nclient = instructor.from_gemini(genai.GenerativeModel(\"gemini-3-flash-preview\"))\n```\n\nThis pattern forces `reasoning` to be populated before `claim_type`, which the model cannot skip (Pydantic enforces it). The reasoning trace is preserved in the output and can be inspected.\n\n### The DnDScore insight: decomposition + decontextualization tension\n\n\"DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation\" (arXiv:2412.13175, EMNLP 2025) identifies a fundamental tension the Deliberus pipeline currently ignores:\n\nDecomposition isolates atomic facts → makes claims shorter, strips context\nDecontextualization inserts relevant context → makes claims longer, adds back context\n\nThese work in opposite directions. The insight is to do BOTH explicitly, in sequence: first decompose into atomic claims (Pass 2 focused extraction), then separately decontextualize each claim (stranger test pass). Currently in Deliberus these are done in a single prompt, which is why some claims are \"almost standalone\" but still have residual context-dependency. Separating them into sub-steps within Pass 2 would improve decontextualization quality.\n\n---\n\n## 3. Claim Decontextualization\n\n### The academic foundation\n\nThe foundational paper \"Decontextualization: Making Sentences Stand-Alone\" (TACL, MIT Press, 2021, Choi et al.) defines the task and establishes that good decontextualization requires resolving pronouns, expanding noun phrases, and adding context for time/location dependencies. The paper has become the standard reference.\n\n### Molecular Facts (2024): two criteria\n\n\"Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification\" (arXiv:2406.20079, EMNLP 2024 Findings) establishes two criteria for high-quality standalone claims:\n\n1. **Decontextuality**: can be verified outside the original context (no pronouns, no \"this\", no context-dependent references)\n2. **Minimality**: contains exactly the information needed for decontextuality — not more (no injected extra content), not less (no dangling references)\n\nThese criteria are in tension: achieving decontextuality sometimes requires violating minimality. **For Deliberus**: apply both criteria as a two-pass within Pass 2. First extract, then apply a separate decontextualization step that tries to satisfy both criteria.\n\n### Claimify's evaluation framework\n\nClaimify (arXiv:2502.10855, Microsoft Research, Mar 2025) achieved:\n- 99% entailment rate (extracted claim entailed by original text)\n- 87.6% coverage of verifiable content\n- 96.7% precision\n\nThe evaluation framework itself is valuable for Deliberus: entailment rate + coverage measures the quality of claim extraction independently. An automatic evaluator using NLI (Natural Language Inference) models can check whether each extracted claim is entailed by the source text. This is a self-evaluation pass that requires no human annotation (see Section 7).\n\n**Key Claimify divergence**: Their Selection stage filters to \"specific, verifiable propositions\" — discarding ~96% of text in normative sources. This is correct for factuality evaluation but wrong for deliberation analysis. Deliberus's \"wider mouth\" design is the right approach for the platform's goals.\n\n### Evaluating decontextualization quality automatically\n\nNo standard benchmark exists specifically for argumentative claim decontextualization quality. The relevant proxy metrics used in the literature are:\n\n- **NLI entailment** (claim entailed by source): measures correctness but not completeness\n- **Self-containedness score** (does the claim require external context to understand?): can be LLM-judged with \"rate from 1-5 how well this claim stands alone\"\n- **Round-trip test**: extract claim → try to identify source text — if impossible, claim is over-decontextualized (lost essential specificity)\n\n---\n\n## 4. Relationship Detection Between Claims\n\n### Support/attack classification: LLMs vs fine-tuned models\n\n\"Can Large Language Models perform Relation-based Argument Mining?\" (arXiv:2402.11243, COLING 2025) is the most direct comparison on this sub-task. Key findings: LLMs with carefully crafted, taxonomy-guided prompts can outperform specialized fine-tuned models for relation classification without task-specific training. The pattern holds for support/attack/neutral three-way classification.\n\nHowever, the generalization finding from ACL 2025 applies here too: fine-tuned models score higher on known benchmarks but fail on new domains. For Deliberus's novel relation types (QUALIFIES, REFRAMES, DECOMPOSES_INTO), LLM prompting is the right approach — no fine-tuned model exists for these types.\n\n**REFRAMES as a Deliberus innovation**: Of the five relation types in the Deliberus schema, only SUPPORTS and ATTACKS correspond to standard AM relation classes. QUALIFIES (conditional support), REFRAMES (frame shift), and DECOMPOSES_INTO (premise-conclusion decomposition) are novel. The first pipeline run successfully detected the Straubhaar REFRAMES relationship — demonstrating LLMs can handle non-standard relation types when the prompt defines them clearly.\n\n### Multi-agent debate for relation classification\n\n\"Multi-Agent LLM Debate Unveils the Premise Left Unsaid\" (ACL 2025 ArgMining Workshop, 2025.argmining-1.6) introduces a multi-agent framework where two LLM agents debate which implicit premise is most contextually appropriate, converging on the most coherent interpretation. Key finding: accuracy gains stem from agents refining predictions in response to opposing views, not just repeated generation. Forcing models to defend assigned stances DEGRADES performance.\n\n**Actionable for Deliberus**: For ambiguous relationships (especially between normative claims), a lightweight two-agent mini-debate (each agent argues for a different relationship type, then converge) could improve classification. This is heavier than simple voting but appropriate for high-stakes relationships.\n\n### Cross-document argument linking\n\n\"Mining Inter-Document Argument Structures in Scientific Papers for an Argument Web\" (Dagstuhl TGDK, 2025) extends the Sci-Arg corpus with inter-document argumentative relations and evaluates three automated approaches for predicting these cross-document relations. Baselines are established.\n\n\"Argument centric causal intervention for cross document event coreference\" (Nature Scientific Reports, 2025) uses argument-centric causal intervention (ACCI) to mitigate bias in cross-document coreference resolution, integrating counterfactual reasoning to reduce spurious lexical correlations.\n\n**Key insight**: Cross-document linking requires a claim normalization step — two claims from different documents that express the same proposition must be identified as equivalent before linking. This is where embeddings come in (dedup by cosine similarity). The pipeline design notes this for future work, but it should be treated as Pass 4 rather than an afterthought.\n\n### Walton argument schemes\n\n\"Large Language Models in Argument Mining: A Survey\" (arXiv:2506.16383) specifically notes: \"Symbolic scaffolding, in which an LLM populates a Walton-style template before emitting a verdict (Lee et al., 2024; Borg and Bex, 2024), has shown promise.\" The approach: give the LLM a Walton scheme template (e.g., \"Appeal to Expert Opinion: [Expert E] claims P. E is an expert in field F. Therefore P\") and ask it to check whether a relationship matches the scheme structure.\n\nFor Deliberus, Walton schemes can serve as **relation classifiers**: if claim A and B match \"Appeal to Consequences\" (arguer says: if we believe P, bad thing Q will happen), it gets that scheme tag. Schemes are more semantically precise than bare SUPPORTS/ATTACKS and could improve the classification of QUALIFIES and REFRAMES relationships.\n\n---\n\n## 5. Contested Concept Detection\n\n### No direct academic work on Gallie's \"essentially contested concepts\"\n\nThe search found no NLP paper that directly operationalizes Gallie's (1956) notion of essentially contested concepts — concepts whose proper application is \"inevitably a matter of dispute\" not because of semantic ambiguity but because of deep value conflicts. This is a genuine gap.\n\nThe closest academic tradition is **Lexical Semantic Change Detection** (LSCD), which detects how word meanings shift over time or across corpora. Key tools: the GWSD dataset (2025, graded word sense disambiguation for diachronic tasks), semantic change detection methods using word sense distribution comparison. These are temporal in focus (meaning change over decades), not synchronic disputational (same text, different senses used by different speakers).\n\n### Word sense disambiguation in argumentative text\n\nThe relevant technique for Deliberus's contested concept detection is **synchronic polysemy detection**: finding cases where the same word is used with materially different meanings by different speakers in the same debate. The pipeline currently detects this for semantic contestedness (\"cost\" = monetary expense vs. net financial impact) but misses criterion contestedness (\"fairness\" = responsibility-based vs. non-discrimination-based).\n\n**Two subtypes (reinforcing the first-run analysis)**:\n\n1. **Semantic contestedness** (different dictionary senses):\n   - Detection pattern: find claims where a key term is defined differently (\"A means B\" vs \"A means C\")\n   - Existing NLP tools: WSD models can identify which sense is being used\n   - Current pipeline: catches this via \"X means Y\" pattern matching\n\n2. **Criterion contestedness** (competing principles for evaluative terms):\n   - Detection pattern: find evaluative terms (\"fairness\", \"justice\", \"merit\") co-occurring with incompatible normative criteria — \"fair because [responsibility premise]\" vs \"unfair because [discrimination premise]\"\n   - No existing NLP tool addresses this directly\n   - Proposed detection method: extract all sentences containing evaluative terms + their context clauses, cluster by the criterion being applied, flag clusters with incompatible criteria\n\n### A practical approach for Pass 3b\n\nNo published system implements criterion contestedness detection. The approach that emerges from the literature:\n\n1. **Extract evaluative terms**: find normative/value_premise claims that use evaluative language (\"fair\", \"just\", \"deserving\", \"right\", \"legitimate\")\n2. **Extract the grounding structure**: for each evaluative claim, extract the normative criterion it appeals to (the \"because\" clause or the implicit premise)\n3. **Compare across speakers/claim sources**: if the same evaluative term has incompatible groundings across claims, flag as criterion-contested\n4. **Tag as CONTESTED_CRITERION** vs **CONTESTED_SENSE** in the concept node\n\nThis can be fully implemented in Pass 3b (contested concept detection) as an LLM prompt: \"Find all uses of [evaluative term] across these claims. Do they appeal to compatible or incompatible principles? If incompatible, explain the dispute.\"\n\n---\n\n## 6. Multi-Pass vs. Single-Pass Extraction\n\n### The academic consensus: multi-pass with self-consistency wins\n\nThe 2025 survey (arXiv:2506.16383) reports: \"Performance improves further when prompts elicit chain-of-thought rationales and apply self-consistency voting, with a 7-point macro-F1 gain on IBM-Rank after majority-voting across five diverse reasoning traces.\"\n\nThis is a significant result: from 7-point macro-F1 gain by varying the reasoning traces (not just running the same prompt multiple times). The implication: diverse prompts producing the same answer in 2/3 or 3/5 trials is more reliable than a single precise prompt.\n\n**Single-pass failures**: While existing approaches typically generate the entire reasoning chain in a single forward pass, this leads to \"CoT derailment\" where reasoning drifts due to compounding errors. The MPPA paper (arXiv:2510.11620) proposes generating multiple candidate plans and aggregating them — directly analogous to running multiple scout passes and taking the consensus argument structure map.\n\n### Pipeline approach vs end-to-end for Deliberus\n\nThe autoregressive AASP approach (IJCNN 2025) achieves better benchmarks on end-to-end extraction. However, Deliberus has a different constraint: each pass output is persisted and must be inspectable. This is a design choice that prioritizes debuggability and human review over raw accuracy, and it's the right choice for the platform's epistemic goals.\n\nThe key insight from comparing approaches is: **consistency between passes is the main weakness of multi-pass pipelines**. When Pass 1 produces bundled/paraphrased conclusions, Pass 2 is forced to over-decompose, and Pass 3 builds relationships on shaky foundations. The fix identified in the first-run analysis (Pass 1 should POINT not PARAPHRASE) is exactly what the literature supports: scout passes should produce structured indices (spans, candidate theses) not authored summaries.\n\n### Self-consistency for Pass 2 specifically\n\nFor the Deliberus pipeline, the highest-impact application of self-consistency is in Pass 2 (focused extraction), not Pass 1:\n\n- Run Pass 2 with 3 different prompt framings (varied examples, varied CoT instructions)\n- Take 2/3 consensus on: claim text, claim type, epistemic status\n- When 3 passes disagree on claim type, log as `classification_uncertain: true` — these are the cases for human review\n\nCost analysis: Pass 2 currently runs once at Gemini 3 Flash pricing (~$0.004/article). 3x = ~$0.012/article. Still negligible. At Wikipedia-corpus scale (millions of articles), 3x becomes relevant — apply voting only to claims flagged as `classification_uncertain` on first pass.\n\n### Hybrid pipeline architecture: pipeline-first, end-to-end as future upgrade\n\nThe right architecture is: begin with the three-pass pipeline (debuggable, inspectable, extensible), then train a specialized end-to-end model on the human-corrected pipeline output once enough annotated data exists. The pipeline generates its own training data. This is the path that preserves the platform's epistemic goals while working toward higher accuracy.\n\n---\n\n## 7. Evaluation\n\n### LLM-as-judge for argument extraction\n\nThe 2025 literature on LLM-as-judge has matured significantly. Key findings:\n\n- **Sage** (self-consistency-based judge evaluation): measures LLM judge quality via local self-consistency (pairwise preference stability) and global logical consistency (transitivity). A judge that contradicts itself across similar inputs is unreliable.\n- **Human annotation is not reliable gold standard**: \"Substantial inconsistency in human judgments indicates that human annotation may not be a reliable gold standard\" (search result summary from PMC paper). This is philosophically important for Deliberus — contested claims have inherently contested \"correct\" classifications.\n- **Bias warning**: LLM judges inherit topic and sentiment biases. Zero-shot sufficiency scores from LLMs rival supervised baselines for argument quality assessment but with systematic topic bias. For Deliberus's political/normative content, this matters: models trained on Western political discourse may misclassify claims from other value systems.\n\n### Self-evaluation pass (Pass 4): what can be built now\n\nA fourth pass checking the extraction output against the source text is feasible without human annotation:\n\n**Entailment check** (correctness):\nFor each claim, run an NLI check: is this claim entailed by the source text? A claim that is NOT entailed by the source was either hallucinated or over-decomposed into something not actually stated. Entailment rate (target: >95%, matching Claimify's 99%) gives a correctness score for each run.\n\n**Coverage check** (completeness):\nReverse question: given the source text, ask the LLM \"which claims in this list cover which parts of the source? Are there significant argument positions in the source not represented in the claim set?\" This is a qualitative completeness check.\n\n**Internal consistency check**:\nDo the relationships form a consistent argument graph? Specific checks:\n- No claim can both SUPPORTS and ATTACKS the same target claim\n- DECOMPOSES_INTO edges should form a DAG (no cycles)\n- A claim of type `value_premise` should not SUPPORTS an `empirical` claim (wrong direction of argument logic)\n\nThese consistency checks are rule-based (no LLM needed) and can catch Pass 3 errors deterministically.\n\n**Self-repair via reflection**:\n\"S2R: Teaching LLMs to Self-verify and Self-correct\" (ACL 2025) shows that models can improve their own outputs when asked to verify specific properties. A reflection prompt for Pass 2: \"Review this extracted claim. Does it pass the stranger test? Could a reader who has not read the source text understand it? If not, rewrite it.\" Running this as a post-processing step improves decontextualization without a full re-extraction.\n\n### A proposed evaluation pipeline\n\n```\nPass 1: Scout (holistic, produces spans/theses)\nPass 2: Focused extraction ×3 (self-consistency voting)\nPass 3a: Relationship detection\nPass 3b: Contested concept detection (semantic + criterion subtypes)\nPass 4a: NLI entailment check (correctness, no human needed)\nPass 4b: Reflection/stranger-test check (decontextualization quality)\nPass 4c: Graph consistency check (rule-based, no LLM)\n```\n\nTotal cost estimate (Gemini 3 Flash): ~$0.02–0.05 per Wikipedia article with full self-consistency + evaluation. Still under $0.10 per article at this scale.\n\n---\n\n## 8. Additional High-Value Papers Not Fitting the Seven Categories\n\n### Unified framework: argumentation theory + formal logics\n\n\"LLM-based Argument Mining meets Argumentation and Description Logics: a Unified Framework for Reasoning about Debates\" (arXiv:2603.02858, March 2026) proposes extracting a **fuzzy argumentative knowledge base** from debate text where arguments are linked by attack/support relations with initial fuzzy strength scores reflecting plausibility. Quantitative argumentation semantics then compute final argument strengths by propagating support/attack effects.\n\nThis is conceptually adjacent to the Deliberus vision: a formal reasoning layer atop extracted argument structures. The paper demonstrates it's feasible to compute \"which position wins the argument, given the strength of supporting/attacking claims\" automatically. Worth tracking as the platform's reasoning layer matures.\n\n### Implicit premises\n\n\"Multi-Agent LLM Debate Unveils the Premise Left Unsaid\" (ACL 2025 ArgMining Workshop, 2025.argmining-1.6): most real arguments have unstated premises (enthymemes). The paper achieves SOTA on implicit premise recovery via structured two-agent debate. **For Deliberus**: the pipeline currently extracts explicit claims only. Adding implicit premise recovery (even as an optional post-pass) would surface the hidden assumptions that drive many debates — the \"because\" clauses that speakers consider so obvious they don't state them.\n\n### Claim matching for fact-checking\n\n\"Zero-shot and Few-shot Learning with Instruction-following LLMs for Claim Matching in Automated Fact-checking\" (arXiv:2501.10860, Jan 2025) addresses claim matching: given two claim texts, determine if they express the same proposition. This is the deduplication problem for the Deliberus graph — before storing a claim, check if it already exists under a paraphrase. Few-shot examples significantly improve claim matching accuracy over zero-shot.\n\n### Argumentative Human-AI Decision-Making\n\n\"Argumentative Human-AI Decision-Making: Toward AI Agents That Reason With Us, Not For Us\" (arXiv:2603.15946, March 2026) proposes that AI agents should reason with humans through explicit argumentative frameworks rather than producing opaque conclusions. Closely aligned with Deliberus's core thesis about transparent collective reasoning.\n\n---\n\n## 9. Structured Output: Specific Recommendations for the Deliberus Pipeline\n\nBased on the research, here are concrete technical recommendations:\n\n### Drop `response_mime_type: \"application/json\"` for complex schemas\n\nSwitch to `instructor` + Pydantic for all three passes. The benefits:\n- CoT-before-label ordering guaranteed (use field order in Pydantic model)\n- Automatic retry with validation error context\n- Semantic validation (custom validators for cross-field constraints)\n- Works with Gemini 3 Flash via `instructor.from_gemini()`\n\n### Prompt engineering for extraction\n\nThe survey identifies that \"the reasoning trace, not the final label, accounts for most of the performance gain.\" Apply this to every extraction call:\n\n```\nBAD:  \"Classify this claim: [claim text]. Output: {claim_type: ...}\"\nGOOD: \"Analyze this claim: [claim text].\n       First, identify what kind of proposition this makes (factual,\n       normative, definitional, or value-based). Then consider the\n       epistemic status (observed fact, prediction, etc.).\n       Finally, classify it. Output: {reasoning: ..., claim_type: ..., epistemic_status: ...}\"\n```\n\n### Handle JSON failures gracefully with `instructor`\n\nThe current retry approach (re-run entire pass on JSON parse failure) is wasteful. With `instructor`, failures trigger targeted retries where the error message is fed back to the model. This reduces failures from ~5-10% to <1% in practice.\n\n### Schema design for CoT ordering\n\nIn Pydantic models, always put the reasoning field FIRST. Python dicts preserve insertion order, and Pydantic models enumerate fields in definition order. The model fills fields in schema order — reasoning trace before answer ensures the model cannot \"skip to the answer.\"\n\n---\n\n## 10. Key Papers Reference List\n\n| Paper | Venue | ArXiv | Relevance |\n|-------|-------|-------|-----------|\n| Large Language Models in Argument Mining: A Survey | Preprint | 2506.16383 | Comprehensive SOTA survey, ~250 papers |\n| Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments | ACL 2025 | 2505.22137 | Critical warning: fine-tuning overfits benchmarks |\n| Argument Mining with Fine-Tuned LLMs | COLING 2025 | — | SOTA on known benchmarks (LLaMA/Qwen/Mistral) |\n| End-to-End Argument Mining through Autoregressive Structure Prediction | IJCNN 2025 | 2510.16363 | End-to-end AASP architecture |\n| Argument Mining as a Text-to-Text Generation Task | — | 2603.23949 | Seq2seq framing, strong baselines |\n| MArgE: Multi-LLM Argumentative Evidence | Preprint | 2508.02584 | Multi-LLM voting for argument extraction |\n| Multi-Agent LLM Debate Unveils the Premise Left Unsaid | ACL 2025 ArgMining | — | Implicit premise recovery, debate agents |\n| A comprehensive study of LLM-based argument classification | Preprint | 2603.19253 | GPT-5.2, DeepSeek, Llama4 comparison |\n| Claimify: Towards Effective Extraction and Evaluation of Factual Claims | Microsoft Research | 2502.10855 | Factual claim pipeline, evaluation framework |\n| DnDScore: Decontextualization and Decomposition | EMNLP 2025 | 2412.13175 | Tension between decomposition and decontextualization |\n| Molecular Facts: Desiderata for Decontextualization | EMNLP 2024 | 2406.20079 | Decontextuality + minimality criteria |\n| LLM-based Argument Mining meets Argumentation and Description Logics | Preprint | 2603.02858 | Formal reasoning over extracted arguments |\n| Mining Inter-Document Argument Structures | TGDK/Dagstuhl 2025 | — | Cross-document argument linking baselines |\n| Zero-shot and Few-shot Claim Matching | Preprint | 2501.10860 | Claim deduplication across documents |\n| Argumentative Human-AI Decision-Making | Preprint | 2603.15946 | AI-human joint argumentation vision |\n| Decontextualization: Making Sentences Stand-Alone | TACL 2021 | — | Foundational paper for stranger test |\n| Walton, Reed, Macagno: Argumentation Schemes | Cambridge UP, 2008 | — | Scheme library for relation classification |\n\n---\n\n## 11. Synthesis: Priority Improvements for the Deliberus Pipeline\n\nRanked by impact/effort ratio:\n\n### Tier 1 — Implement now (high impact, low effort)\n\n1. **Switch to `instructor` + Pydantic** for all passes. Fixes JSON reliability, schema ordering, adds auto-retry. Half-day implementation.\n\n2. **Add `epistemic_status` field to Pass 2 output**. Fixes the \"empirical overuse\" problem identified in the first run. Add five values: `observed | causal_hypothesis | prediction | counterfactual | feasibility`. Half-day.\n\n3. **Add CoT-before-label to all prompts**. Put reasoning field first in schema, restructure prompts to reason before classifying. One day to retune all three pass prompts.\n\n4. **Split Pass 3 into 3a (relationships) + 3b (contested concepts)**. Let 3b see the full graph (claims + types + modalities + relationships) as input. Catches criterion contestedness (the \"fairness\" miss). Half-day refactor.\n\n### Tier 2 — Next sprint (high impact, medium effort)\n\n5. **Self-consistency voting for Pass 2**. Run 3x with varied prompts, take 2/3 consensus. Log `classification_uncertain: true` for disagreements. Adds ~3x API cost for Pass 2 only. One-two days.\n\n6. **NLI-based self-evaluation pass (Pass 4a)**. After extraction, check entailment of each claim against source. Use a lightweight NLI model (e.g., cross-encoder/nli-deberta-v3-small) locally or Gemini for this. Produces per-run quality score. Two days.\n\n7. **Fix Pass 1 to POINT not PARAPHRASE**. Redesign scout output to produce argument spans + verbatim candidate thesis sentences from the source text, not LLM-authored summaries. This fixes the cascade of over-decomposition in Pass 2.\n\n### Tier 3 — Research phase (high impact, high effort)\n\n8. **Criterion contestedness detector**. Novel approach: extract evaluative terms + their normative grounding structure, compare across claims, flag incompatible groundings. Requires careful prompt design and validation.\n\n9. **Implicit premise recovery pass** (after MArgE/multi-agent debate approach). For high-value argument structures, run a two-agent mini-debate to surface unstated premises. Heavy but potentially transformative for deliberation analysis.\n\n10. **Cross-document claim matching** (embedding-based dedup + few-shot matching). Required for the graph to scale beyond single documents. Implement after the single-document pipeline is stable.\n\n---\n\n## Cross-References\n\n- [extraction-pipeline-design.md](extraction-pipeline-design.md) — pipeline architecture decisions\n- [first-pipeline-run-analysis.md](first-pipeline-run-analysis.md) — first run results, six improvement priorities\n- [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md) — four node types, concept tracking\n- [walton-argument-schemes.md](walton-argument-schemes.md) — scheme library review\n- Claimify code and paper (local clone)\n"}