{"path":"research/temporal-extraction-architecture.md","content":"# Temporal Extraction Architecture\n\n**Date**: March 30, 2026 (Session 7)\n**Status**: Built and deployed to production. Worker connected and polling.\n\n---\n\n## Why Temporal\n\nThe extraction pipeline takes 3-5 minutes (scheme detection adds to this). When `kamal deploy` happens mid-extraction, the container is killed, the SSE stream drops, and the extraction is lost. Users see nothing — the UI resets silently.\n\n**Alternatives considered:**\n- **Increase drain/stop timeouts** (5 min work): kamal-proxy waits longer before killing old container. Solves 90% but adds deploy latency.\n- **FalkorDB as job queue** (hours): store job status in FalkorDB, background thread processes, SSE polls. Simple but no retry/resumption.\n- **Temporal** (half day): full workflow durability, activity-level retry, heartbeat-based dead worker detection, state survives container restarts.\n\n**Decision**: Temporal. Already running on Darwin (server 1.24.2, Docker), used in BRF Auto. The extraction pipeline maps naturally to activities. The 30-second heartbeat timeout means deploys only cause a brief pause, not data loss.\n\n---\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ deliberus.com (Docker container on Darwin)               │\n│                                                          │\n│  FastAPI (lifespan)         Temporal Worker               │\n│  ├── POST /extract/start    ├── ExtractionWorkflow       │\n│  │   → starts workflow      │   ├── fetch_url_activity   │\n│  │   → returns workflow_id  │   ├── cleanup_activity     │\n│  │                          │   ├── scout_activity       │\n│  ├── GET /extract/{id}/     │   ├── focused_extraction   │\n│  │   status/stream          │   ├── relationships (∥)    │\n│  │   → queries workflow     │   ├── concepts (∥)         │\n│  │   → streams SSE          │   ├── cq_generation        │\n│  │                          │   ├── self_eval            │\n│  ├── GET /extract/{id}/     │   ├── store                │\n│  │   status                 │   ├── embed                │\n│  │   → one-shot check       │   └── auto_connect         │\n│  │                          │                             │\n│  └── /extract/stream        │  (inline fallback if       │\n│      (legacy fallback)      │   Temporal unavailable)     │\n└──────────────┬───────────────┴──────────────┬────────────┘\n               │  gRPC :7233                   │\n               ▼                               │\n┌──────────────────────────────┐               │\n│ Temporal Server 1.24.2       │               │\n│ (separate Docker container)  │               │\n│ PostgreSQL 16 backend        │               │\n│ UI at :8233                  │               │\n│ Namespace: deliberus         │               │\n└──────────────────────────────┘               │\n                                               │\n               ┌───────────────────────────────┘\n               ▼\n┌──────────────────────────────┐\n│ FalkorDB :6380               │\n│ Qwen3 Embedder :8080         │\n└──────────────────────────────┘\n```\n\n**Key design**: Worker runs IN the same container as FastAPI. Temporal CEO Maxim Fateev: *\"I would start by running it in the same process. If the process becomes overloaded, you can always move some logic out.\"* The Temporal server (separate container) persists workflow state. When the app container restarts, the worker reconnects and Temporal reschedules any timed-out activities.\n\n---\n\n## Deploy Survival: How It Works\n\n1. User clicks \"Extract\" → `POST /extract/start` → creates Temporal workflow → returns `workflow_id`\n2. Frontend connects SSE to `GET /extract/status/{workflow_id}/stream`\n3. Worker picks up workflow → runs activities sequentially (with parallel 3a+3b)\n4. **Deploy happens** → container killed → worker disconnects\n5. Temporal detects dead worker via **heartbeat timeout (30 seconds)** → marks activity as timed out\n6. **New container starts** → new worker connects → Temporal reschedules the timed-out activity\n7. **Cleanup activity runs first** — removes partial results from old code (Data Freshness Directive)\n8. Extraction restarts from scratch with the new code\n9. Frontend SSE auto-reconnects (2-second retry) → sees progress continue\n10. User experience: brief \"Reconnecting...\" then \"System redeployed; restarting extraction with latest code...\"\n\n**Why restart from scratch, not resume**: The extraction pipeline is stateless — same input produces same output. But if the code CHANGED between attempts (new scheme detection prompt, different CQ templates, updated models), partial results from the old version could be subtly wrong. The cleanup activity ensures every extraction run is internally consistent with the code that produced it.\n\n---\n\n## Activity Design (11 Activities, One Per Pipeline Pass)\n\n| Activity | What It Does | Timeout | Heartbeat |\n|---|---|---|---|\n| `fetch_url_activity` | Fetch URL content, resolve source_id slug | 60s | 30s |\n| `cleanup_activity` | Remove partial results for this source_id from FalkorDB | 60s | 30s |\n| `scout_activity` | Pass 1: holistic argument landscape | 10min | 30s |\n| `focused_extraction_activity` | Pass 2a+2b: decompose + decontextualize | 10min | 30s |\n| `relationships_activity` | Pass 3a: relationships + scheme classification | 10min | 30s |\n| `concepts_activity` | Pass 3b: contested concepts (parallel with 3a) | 10min | 30s |\n| `cq_generation_activity` | CQ parameterization with stranger test | 10min | 30s |\n| `self_eval_activity` | Pass 4: quality self-evaluation | 10min | 30s |\n| `store_activity` | Write claims, relationships, CQs to FalkorDB + JSON | 60s | 30s |\n| `embed_activity` | Pass 5: embeddings + SIMILAR_TO auto-link | 10min | 30s |\n| `auto_connect_activity` | Pass 6: cross-extraction SUPPORTS/ATTACKS discovery | 10min | 30s |\n\n**Design principles applied:**\n- Activities call existing `deliberus/extraction/` modules — never reimplement (BRF Auto gotcha #7)\n- All LLM-calling activities use `RetryPolicy(maximum_attempts=3, backoff_coefficient=2.0)` — Temporal manages retries, not the LLM SDK (gotcha #5)\n- Heartbeats every ~30 seconds — dead worker detected in under a minute (gotcha #4)\n- Passes 3a + 3b run in parallel via `asyncio.gather` — no change from inline pipeline\n\n---\n\n## SSE Progress: Workflow Queries\n\nThe workflow exposes a `get_status()` query handler that returns `ExtractionStatus`:\n\n```python\n@workflow.query\ndef get_status(self) -> ExtractionStatus:\n    return self._status  # Updated after each activity completes\n```\n\nThe FastAPI SSE endpoint polls this query every 500ms:\n\n```python\n@app.get(\"/extract/status/{workflow_id}/stream\")\nasync def extraction_status_stream(workflow_id: str):\n    handle = client.get_workflow_handle(workflow_id)\n    async def generate():\n        while True:\n            status = await handle.query(ExtractionWorkflow.get_status)\n            if status.message != last_message:\n                yield sse_event(\"progress\", status_dict)\n            if status.status in (\"completed\", \"failed\"):\n                break\n            await asyncio.sleep(0.5)\n```\n\n**Upgrade path**: When sub-activity granularity is needed (\"claim 12/74\"), add Redis PubSub within activities (the BRF Auto pattern). Workflow queries handle coarse progress; Redis handles fine-grained.\n\n---\n\n## BRF Auto Lessons Applied\n\n| BRF Auto Mistake | How Deliberus Avoids It |\n|---|---|\n| String keys after JSON deserialization | Python SDK uses Pydantic dataclasses natively |\n| Workflows accessing DB directly | Every external call is an Activity |\n| Scheduled workflows never created | User-triggered one-shot workflows, not scheduled |\n| `execute_all` doesn't exist (Ruby) | Python: `asyncio.gather()` for parallel activities |\n| Continuous workflows terminated on deploy | One-shot extraction workflows, not continuous |\n| State machine skipped intermediate states | Linear pipeline, no state machine |\n| Activity reimplemented shared logic | Activities import from `deliberus/extraction/` |\n| API shape mismatch from docs | Latest Python SDK, verified imports |\n\nPlus 4 new gotchas identified during Deliberus research:\n- Workflows must be deterministic (no LLM calls in workflow code)\n- Always heartbeat long-running activities\n- Disable LLM SDK client retries (Temporal manages retries)\n- Classify errors: non-retryable (auth, bad input) vs retryable (rate limit, timeout)\n\nAll 12 gotchas documented in global CLAUDE.md §Temporal Comprehensive Directive.\n\n---\n\n## Infrastructure\n\n- **Temporal Server**: `temporalio/auto-setup:1.24.2` (Docker on Darwin)\n- **PostgreSQL**: 16-alpine (Temporal backend)\n- **UI**: `temporalio/ui:2.31.2` at `darwin.home:8233`\n- **gRPC**: `darwin.home:7233` (LAN-exposed, was localhost-only before this session)\n- **Namespace**: `deliberus` (created via CLI)\n- **Task Queue**: `deliberus-extraction`\n- **CLI**: v1.6.2 installed on Mac Mini (brew) + Darwin (curl). `tctl` deprecated — use `temporal` exclusively.\n- **Python SDK**: `temporalio` v1.24.0 (in pyproject.toml)\n- **Docker network**: Temporal server has alias `temporal` on the `kamal` network — reachable from app container\n\n---\n\n## When to Split Worker from API\n\nCurrently co-located (same process/container). Split when:\n- API response latency degrades because activities consume too much CPU/memory\n- Need independent scaling (1 API instance, 3 worker instances)\n- Container memory exceeds Darwin's limits\n- Want blast-radius isolation (worker crash shouldn't affect API)\n\nThe split is straightforward: extract `Worker(...)` startup into a separate Kamal role/container. Activity and workflow code stays the same.\n\n---\n\n## Cross-References\n\n- Global CLAUDE.md §Temporal Comprehensive Directive — 12 gotchas\n- [Project CLAUDE.md §Deployment](../../CLAUDE.md) — infrastructure details\n- [.claude/specs/scheme-bounded-decomposition/](../../.claude/specs/scheme-bounded-decomposition/) — the extraction pipeline spec\n- [scheme-bounded-decomposition-and-evidence-as-subgraph.md](scheme-bounded-decomposition-and-evidence-as-subgraph.md) — pipeline design + cost analysis\n- BRF Auto Temporal implementation plan — original gotchas source\n"}