{"path":"research/extraction-pipeline-design.md","content":"# Extraction Pipeline Design — Session 3 Reasoning\n\n**Date**: March 28, 2026\n**Status**: Direction agreed, key decisions made. Three-pass hybrid pipeline. Gemini 3 Flash. Direct FalkorDB. Ready to build.\n\nFor the short, current, non-research explanation of the live extraction path, see [../extraction-pipeline.md](../extraction-pipeline.md).\n\n---\n\n## The Starting Question\n\nFredrik: \"The claim extraction experiment, how would we go about it? I think an interesting experiment to run would be to study the Claimify code and research paper and consider how we might pragmatically implement our own version of it locally with a consciously chosen tech stack that suits my preferences and allows for a fast feedback loop to iterate fast and see if we could apply it to a set of interconnected articles from Wikipedia, for example, or random Reddit threads on a given topic.\"\n\n## The \"Don't Build Throwaway\" Principle\n\nFredrik: \"I don't want to build something that I will have to tear down the next week or the next day when I want to build something that is closer to the full vision. So I want to start thinking about how the fuller vision likely will need to be implemented.\"\n\n**Resolution**: With 8-15x AI-augmented dev velocity, the cost difference between \"quick script\" and \"first module of the real system\" is days, not months. The throwaway-then-rebuild cycle wastes more time than building slightly more thoughtfully from the start. The experiment should BE the first layer of the real system — modular, extensible, storing results in FalkorDB from day one.\n\n## Tech Stack Reasoning\n\n**Python** for the extraction pipeline:\n- Fast iteration loop (no compilation)\n- Rich LLM ecosystem (anthropic SDK, google-generativeai)\n- uv for dependency management (fast, no Homebrew pip conflicts)\n- FalkorDB client available\n- Only switch to Rust when latency matters more than iteration speed\n\n**LLM provider — DECIDED**: Gemini 3 Flash. Fredrik's reasoning (verbatim): \"I am of the firm opinion that it would be impractically expensive to use Claude through the Anthropic API because if we use Claude outside of Claude Code, then I cannot use my max plan. And so it will be prohibitively expensive and it might be hard to scale up. On the other hand, if Claude performs much better with these sorts of tasks, then I will reconsider. But let's cross that bridge when we come to it after we've experimented with Gemini 3 Flash.\" Gemini 3 Flash at $0.50/1M input tokens — a Wikipedia article costs ~$0.004.\n\n**Storage — DECIDED**: Direct FalkorDB connection (falkordb-py), NOT through Graphiti layer. Simplest path to start. **CRITICAL**: All Deliberus data MUST be scoped to `group_id = \"deliberus_*\"` namespace. Never write to the default graph or any other existing scoped graph (dotfiles, brf-auto, etc.).\n\n**Package manager — DECIDED**: uv (modern, fast, Fredrik's preference for frontier tooling).\n\n**Interface — DECIDED**: Start with whatever minimizes rework. FastAPI with a CLI wrapper — the API is the real interface, CLI calls it. Both work from day one, no duplication.\n\n**Embeddings**: Local model on Darwin GPU or API — for similarity, dedup, and future clustering/worldview features. Decision deferred to implementation time.\n\n## Why Not NLTK for Sentence Splitting\n\nFredrik: \"What are the real advantages to using NLTK compared to using a modern LLM like GPT-5.2 or Gemini 3 Flash for this part as well? Think deeply about it.\"\n\n**The deeper answer: sentence splitting is the wrong abstraction for Deliberus.**\n\nClaimify splits by sentences because it extracts atomic *factual* claims — one sentence, one proposition. But Deliberus extracts *argument structures*: a normative claim + its premises, which often span multiple sentences:\n\n> \"We should ban factory farming [sentence 1] because it causes animal suffering [sentence 2] and accelerates climate change [sentence 3].\"\n\nONE argument unit across THREE sentences. NLTK would split it into three independent items. The LLM, given the full paragraph, understands it as one conclusion with two premises.\n\n**NLTK's actual advantages:**\n- Free (zero cost per call)\n- Fast (microseconds)\n- Deterministic (same input → same output)\n- Works offline\n\n**Why these don't matter for Deliberus:**\n- Cost: Gemini 3 Flash at $0.50/1M tokens makes the cost argument irrelevant (~$0.01 per article even with majority voting)\n- Speed: LLM extraction takes seconds anyway; microsecond sentence splitting doesn't help\n- Determinism: Valuable for reproducibility, but extraction quality matters more. Can achieve pseudo-determinism with temperature=0\n- Offline: Not a constraint when the extraction itself requires LLM API calls\n\n**What LLMs do better:**\n- Context-aware splitting (understands \"St. Louis\" isn't a sentence boundary)\n- Handles malformed text (Reddit, social media, voice transcripts with no punctuation)\n- Can split by SEMANTIC units (argument structures) rather than punctuation\n- Captures relationships between sentences that NLTK-then-process destroys\n\n**Decision (tentative)**: Skip NLTK entirely. Feed text chunks to the LLM and ask it to identify argument structures directly.\n\n## The Claimify Divergence — Quantified\n\nClaimify's Selection stage asks \"does this contain a specific, verifiable proposition?\" and discards everything that fails. When virtually run on the UBI Wikipedia article ([extraction-virtual-run.md](extraction-virtual-run.md)), Claimify would extract **3-4 claims** from text that the Deliberus pipeline extracted **22+ claims** from. Everything normative (N1-N6), definitional (D1-D5), and value-premise (V1-V5) would be DISCARDED. The Deliberus divergence (wider mouth) is massive in practice.\n\nFredrik's reflection: \"Claimify seems quite limited in scope in comparison with what we are imagining in the long run. Deliberus is made to handle the larger scope and seemingly fuzzy cloud of interrelated statements yet still (try hard to) make rational sense of everything.\"\n\n**However, Claimify's granular pipeline has real advantages:**\n\n| Advantage | Why it matters for Deliberus |\n|-----------|------------------------------|\n| **Focused prompts** (one task per stage) | More reliable than multi-task prompts |\n| **Debuggability** | Know WHERE in the pipeline a claim was lost |\n| **Majority voting** (3x, 2/3 agreement) | Catches LLM flakiness on binary decisions |\n| **The \"stranger test\"** | Decontextualized claims work as graph nodes |\n\n**The prompt shapes the ontology**: Claimify's granular pipeline produces DIFFERENT claims than holistic extraction, because the prompts frame the task differently. Sentence-by-sentence \"is this verifiable?\" → crisp atomic facts. Full-text \"extract all argument structures\" → richer but potentially fuzzier output.\n\n## The Three-Pass Hybrid Pipeline — DECIDED\n\nTakes the best of both: holistic for STRUCTURE, Claimify-style for PRECISION, cross-structure for RELATIONSHIPS.\n\n| Pass | Purpose | Method | Output |\n|------|---------|--------|--------|\n| **1. Holistic scout** | Map the argument landscape | LLM reads ENTIRE text (no chunking — see RLM below for large texts). Identifies conclusions + premise chains as argument units. Not extracting atomic claims yet | List of argument structures with approximate boundaries in source text |\n| **2. Focused extraction** (Claimify-inspired) | Decompose each argument structure into atomic claims | Focused prompt per argument unit: extract claims, classify four types (empirical/normative/definitional/value premise), decontextualize (stranger test) | Atomic claims with types, source spans |\n| **3. Cross-structure analysis** | Connect the argument structures | Identify relationships BETWEEN argument units: supports, attacks, qualifies, reframes, shared premises, contested concepts | Relationship edges, concept disambiguation flags |\n\n**Results of each pass are persisted independently** — each can be inspected, reflected upon, and used differently in later stages. Fredrik: \"There would be three passes and the results of each would be persisted and could be inspected and reflected upon and used in different ways in the later stages of the process.\"\n\n## The RLM Pattern for Large Texts — NO CHUNKING\n\nFredrik flagged: \"Chunking into a thousand word sections risks losing important high level context from the source text. Meaning flows naturally through and within the whole text in most cases.\"\n\n**Resolution**: Apply the RLM pattern (Recursive Language Models, MIT paper 2512.24601):\n\n- **Small texts** (Wikipedia article, Reddit thread, single essays): Fit in context entirely. Feed the whole thing to the LLM. No chunking needed.\n- **Large texts** (book-length, entire forum threads, multiple interconnected articles): RLM pattern — store the full text externally, LLM uses tool calls to peek/search/navigate the text while maintaining awareness of the whole structure. The model holds \"steering logic + goal\" and accesses the text through search, never needing it all in context simultaneously but always able to reference any part.\n\n**Key insight from RLM research**: The model treats the source text as an EXTERNAL OBJECT it can reason about, not as something that must fit in the context window. Scout phase identifies relevant sections; focused extraction zooms in; synthesis connects across sections. No arbitrary chunk boundaries sever meaning mid-argument.\n\nThis pattern should be documented carefully as it will be valuable for ingesting large corpora (the civilizational graph vision).\n\nSee: RLM optimization plan and PageIndex vs RLM commands comparison (local reference docs)\n\n### PDF Ingestion — Implemented (Mar 31, 2026)\n\n**Tier 1/2 PDFs** (<30 pages) are now handled by sending the PDF directly to Gemini as `inline_data` with `mime_type: application/pdf`. Gemini reads the PDF natively — sees tables, figures, formatting, headers, footnotes in their visual context. No `pymupdf`/`pdfplumber` dependency.\n\n**Implementation**: base64-encode PDF, send as first `Part` in contents alongside the text prompt via the `google-genai` SDK's `inline_data` mechanism. The `instructor` library doesn't support multimodal input — `call_genai_with_pdf()` in `deliberus/extraction/client.py` bypasses instructor and uses the genai SDK directly with `response_schema` for constrained JSON output + Pydantic validation.\n\n**Two Gemini calls for PDF input**: (1) Scout pass sends PDF directly (best quality — argument structure identification benefits from seeing visual layout). (2) Text extraction call (`extract_text_from_pdf()`) provides source text as a string for subsequent passes that use it in system messages (focused extraction, self-eval). PDF bytes stored on disk (not in Temporal payload — 2MB limit) and read by activities as needed.\n\n**Tier 3 (book-length)** still requires the RLM pattern described above.\n\n## Key Divergences from Claimify (Summary)\n\n- **Wider mouth**: Keep ALL claims (normative, definitional, value premises — Claimify discards these)\n- **Four-type classification**: empirical / normative / definitional / value premise (currently undecomposed)\n- **No copout axioms**: Value premises are depth-relative, always decomposable further\n- **Relationship detection**: support / attack / qualify / defines-term-for / reframes\n- **Concept flagging**: polysemous terms, contested definitions as first-class nodes\n- **Argument-unit granularity**: not sentence-level but argument-structure-level\n- **No chunking**: full text or RLM pattern, never arbitrary segmentation\n- **Three-pass hybrid**: holistic scout → focused extraction → cross-structure analysis\n\n## Project Structure — DECIDED\n\nOption C: Python package at repo root. `pyproject.toml` at root, source in `deliberus/` directory matching the package name. Standard Python convention (what uv expects).\n\n```\ndeliberus/              # repo root\n  pyproject.toml        # project config + deps (uv)\n  deliberus/            # Python package\n    __init__.py\n    extraction/         # three-pass pipeline (scout, focused, cross-structure)\n    graph/              # FalkorDB storage layer (falkordb-py, deliberus_* scope)\n    api.py              # FastAPI endpoints\n    cli.py              # CLI wrapper (calls API)\n  docs/                 # existing (38 research docs)\n  data/                 # existing (gitignored — voice memos, FB archive)\n  scripts/              # existing (FB scraper)\n```\n\nStart with extraction/ + graph/ + cli.py + api.py. Each module is independent and improvable. Nothing needs tearing down.\n\n## Graph Schema — DECIDED\n\nMinimal starting schema, iterable:\n\n```\nNode labels:\n  :Claim {id, text, type, source_url, source_span, confidence, created_at}\n         type: \"empirical\" | \"normative\" | \"definitional\" | \"value_premise\"\n\n  :Concept {id, term, senses: [...]}  // for contested concepts\n\n  :Source {id, url, title, fetched_at}  // the ingested text\n\nEdge types:\n  :SUPPORTS {strength, pass}\n  :ATTACKS {strength, pass}\n  :QUALIFIES {text, pass}      // \"true, but only when...\"\n  :DEFINES {sense, pass}       // links Claim to Concept with specific sense\n  :REFRAMES {text, pass}       // frame shift (e.g. Straubhaar \"UBI = tax reform\")\n  :EXTRACTED_FROM {span}       // links Claim back to Source\n  :DECOMPOSES_INTO {}          // normative claim → its premises\n```\n\nThe `pass` field records which pipeline pass detected the relationship (1, 2, or 3) — enables independent inspection of each pass's contribution. All nodes carry `group_id = \"deliberus_extraction\"`.\n\n## First Test Text — DECIDED\n\nWikipedia \"Universal basic income\" — the Economics/Costs and Philosophy/Morality sections. Known ground truth from the virtual extraction run ([extraction-virtual-run.md](extraction-virtual-run.md)) — 22+ claims, 10 relationships, 4 contested concepts. Pipeline output can be directly compared to hand-extraction.\n\n## The Graphiti MCP Idea\n\nFredrik raised a radically pragmatic shortcut: \"Could we simply have Claude ingest random articles and save the episodes into Graphiti using the MCP available? Just as a quick way to test what that would result in?\"\n\n**What this would test**: Whether Graphiti's built-in LLM processing pipeline (episode text → entity extraction → reflexion → edge extraction → summary generation) produces useful argument-graph-like structures from deliberation content.\n\n**Why it probably won't work directly**: Graphiti's pipeline is designed for conversational memory and entity tracking, not for argumentation analysis. It extracts ENTITIES (people, places, concepts) and FACTS (relationships between entities), but not CLAIMS (propositions that can be true/false/normative), ARGUMENT STRUCTURES (conclusion + premises), or FOUR-TYPE CLASSIFICATIONS. The ontology mismatch means Graphiti would produce a knowledge graph but not a deliberation graph.\n\n**What would be needed**: A Deliberus-specific fork of the Graphiti MCP that replaces the entity extraction / reflexion pipeline with argument-aware extraction. The MCP infrastructure (FalkorDB connection, episode management, embedding generation, temporal metadata) is reusable; the extraction logic needs replacing.\n\n**Documented as idea, deferred**: Creating a Deliberus-Graphiti MCP fork is a viable path but adds a dependency on Graphiti's architecture. The standalone extraction pipeline (Python + direct FalkorDB) is more flexible for the initial experiment. The Graphiti fork could come later if the MCP infrastructure proves valuable.\n\n## Test Data Candidates\n\n| Source | Why | Characteristics |\n|--------|-----|-----------------|\n| **Wikipedia \"Universal basic income\"** | Pro/con sections, empirical evidence, normative arguments, definitional disputes | Well-structured, sourced, all four claim types |\n| **Reddit r/changemyview** | Structured arguments, delta system = ground truth for compelling arguments | Messy but authentic, tests extraction on real discourse |\n| **FB group archive** (406 posts) | About argumentation itself — meta-level | Already ingested, highly relevant |\n| **EA Forum threads** | Target audience's natural discourse | High-quality, long-form, rationalist norms |\n\n## The Feedback Loop\n\n1. Run extraction on ONE Wikipedia section (e.g., \"Arguments for UBI\")\n2. Inspect output: claims, types, relationships\n3. Manually evaluate: what was caught? missed? misclassified?\n4. Tweak prompts, rerun\n5. Try a Reddit thread (messier — where does it break?)\n6. Try interconnecting claims ACROSS two articles — this is where the graph shows its value\n7. Explore in FalkorDB / eventually Lins\n\n## What's Been Decided\n\n- **Three-pass hybrid pipeline** (holistic scout → focused extraction → cross-structure)\n- **Gemini 3 Flash** as LLM provider (cost-effective, proven on structured extraction)\n- **Direct FalkorDB** (not through Graphiti layer), scoped to `deliberus_*` group_id\n- **uv** as package manager\n- **FastAPI + CLI** (API is the real interface, CLI wraps it)\n- **No NLTK** — argument-unit granularity via LLM, not sentence splitting\n- **No chunking** — full text in context, or RLM pattern for large texts\n- **Python** for iteration speed\n\n## What Remains Undecided\n\n- **Exact prompt design**: The prompts for each of the three passes. The most important design artifact — to be developed through iteration\n- **Majority voting**: Claimify's 3x with 2/3 agreement — worth it for Pass 2 (focused extraction)? Or does temperature=0 suffice?\n- **Embedding model**: Local on Darwin GPU vs API\n- **Graph schema**: Exact FalkorDB node/edge labels and properties for claims, relationships, concepts\n- **The Graphiti fork path**: Deferred. Build standalone first, evaluate later\n\n## Tech Stack Summary\n\n| Component | Choice | Reasoning |\n|-----------|--------|-----------|\n| Language | Python | Iteration speed, LLM ecosystem |\n| Package manager | uv | Modern, fast |\n| LLM | Gemini 3 Flash | Cost-effective ($0.50/1M tokens), proven on extraction. Claude reconsidered if quality insufficient |\n| Graph DB | FalkorDB (direct, falkordb-py) | Already running on Darwin. Scoped to `deliberus_*` |\n| Interface | FastAPI + CLI wrapper | Build once, use both ways |\n| Large text handling | RLM pattern (external storage + peek/search) | No arbitrary chunking. Preserves meaning flow |\n| Embeddings | TBD | Local on Darwin GPU or API |\n\n## Cross-References\n\n- [../technical-direction.md](../technical-direction.md) §Claimify vs Deliberus Extraction — the wider-mouth pipeline\n- [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md) — four node types, concept tracking\n- [combinatorial-mvp-reasoning.md](combinatorial-mvp-reasoning.md) — why the combination matters\n- [consensus-path-forward.md](consensus-path-forward.md) — the build order\n- [embeddings-tension-and-ai-slop.md](embeddings-tension-and-ai-slop.md) — embeddings discover, humans decide\n- Claimify paper + code (local clone)\n"}