{"path":"extraction-pipeline.md","content":"# The Deliberus Extraction Pipeline\n\nThis is the short, plain-language version of how Deliberus turns a pasted text, URL, or PDF into graph structure today.\n\nTwo limits worth knowing up front. **Audio and video are not accepted as sources** — spoken input is transcribed for the conversational path, but there is no way to submit a recording or a video for analysis. And **a whole source is analyzed at one evidential level**: methodology is assessed per source rather than per claim, which is fine for a single study and wrong for a review that covers many at different levels of certainty. Both are open, with the reasoning in [research/long-form-sources-and-meta-analysis-weighing.md](research/long-form-sources-and-meta-analysis-weighing.md).\n\nIt is the pedagogical layer, not the full design argument. For the deeper reasoning behind the pipeline, see [research/extraction-pipeline-design.md](research/extraction-pipeline-design.md), [research/temporal-extraction-architecture.md](research/temporal-extraction-architecture.md), and [research/scheme-bounded-decomposition-and-evidence-as-subgraph.md](research/scheme-bounded-decomposition-and-evidence-as-subgraph.md).\n\n## Why It Has Multiple Phases\n\nDeliberus does not trust one giant prompt to do everything at once.\n\nEach phase has one job. That makes the process easier to inspect, easier to improve, and less likely to quietly flatten important distinctions like evidence vs values, support vs attack, or shared words with different meanings.\n\n## The 8 Steps\n\n### 1. Scout\n\nDeliberus first reads the whole source for a quick overview.\n\nIt asks: what are the main arguments here, who is saying what, and which stretches of the text seem structurally important?\n\n### 2. Extraction\n\nDeliberus then breaks each argument into smaller claims.\n\nThis is where a bundled sentence like \"we should do X because of Y and Z\" starts becoming separate pieces that can later be challenged, supported, or decomposed further.\n\n### 3. Structure Pass (two parallel lanes)\n\nThis phase splits into two lanes that run side by side.\n\n#### 3a. Relations\n\nDeliberus maps how the claims relate: support, attack, qualify, reframe, or decompose into subclaims.\n\nIt also classifies the argument pattern when possible, using Walton-style schemes.\n\n#### 3b. Contested Words\n\nAt the same time, Deliberus checks whether important words are being used in different ways.\n\nThis is where it notices that two people may both say \"freedom\" or \"safety\" while meaning different things.\n\n### 4. Critical Questions\n\nOnce Deliberus has a candidate argument pattern, it generates the specific critical questions that pattern must survive.\n\nThose questions are not decorative. They are part of the reasoning structure and often become new claims that need answers.\n\n### 5. Self-Check\n\nBefore saving anything, Deliberus audits its own work.\n\nIt asks whether claims were missed, relations look wrong, or classifications feel too loose.\n\n### 6. Store\n\nThe current extraction is saved into the graph and database.\n\nClaims, sources, relations, contested concepts, and other extraction artifacts become inspectable product state rather than temporary prompt output.\n\nThat still does not mean they count as final truth. New extraction output can enter as candidate graph structure and later be clarified, decomposed, strengthened, weakened, or superseded as the graph evolves.\n\n### 7. Embed And Link\n\nEach claim gets an embedding so Deliberus can search by meaning, not just exact words.\n\nThat makes it possible to find nearby claims, build local neighborhoods, and prepare later cross-source linking.\n\n### 8. Auto-Connect\n\nFinally, Deliberus looks for claims from different sources that appear to support, attack, or meaningfully overlap with each other.\n\nThis is how the graph stops being a pile of separate extractions and starts becoming one shared deliberation space.\n\n## What Comes Out\n\nAt the end of the pipeline, Deliberus has not just a summary, but a structured map:\n\n- claims that can be inspected and challenged\n- relationships between those claims\n- contested terms that may explain disagreement\n- critical questions that show what still needs work\n- links to related reasoning elsewhere in the graph\n\n## What This Does Not Mean\n\nThe pipeline is not the whole product.\n\nShort authored questions can now take a lighter \"Think with Deliberus\" path before full extraction. And the graph can later be queried as a read layer without re-running the full pipeline. This document is only about the extraction path itself.\n\n\n## When no scheme fits (added 2026-08-19)\n\nThe relationship pass classifies inferences against a curated set of argument schemes — and that set is open by construction (Walton's own field test found 37% of real arguments initially fit none; see [scheme-set-exhaustiveness](research/scheme-set-exhaustiveness.md)). So the classifier carries an honest escape value, `does_not_fit`, rather than forcing the nearest label. An unclassified inference still opens a descent: it receives four generic critical questions that apply to any inference at all — is the premise acceptable, does the conclusion actually follow from it, is there an exception, is there a stronger case for the opposite (the meta-level set from Yu & Zenker 2020). The rate of `does_not_fit` firings is itself a signal: rising means the taxonomy has met a kind of text it was not built from.\n"}