{"path":"research/llm-speed-optimization.md","content":"# LLM API Speed Optimization for the Deliberus Extraction Pipeline\n\n**Date**: March 28, 2026\n**Scope**: 6-pass Gemini 3 Flash extraction pipeline (~25 API calls per 10-argument article, 3-10 min total)\n**Stack**: Python, google-genai SDK, instructor, Pydantic structured outputs\n\n---\n\n## Executive Summary\n\nThree changes can be implemented this week and will likely cut total pipeline time by 70-80%:\n\n1. **Async concurrent Pass 2** — fire all argument extractions in parallel instead of sequentially\n2. **Implicit prefix caching** — the source text is resent 20+ times; with caching it costs 90% less and processes faster\n3. **Multi-argument single-call batching** for Pass 2a (decompose) — send all argument structures at once, get all atoms back\n\nThe remaining optimizations (Batch API, local GPU, Flash-Lite for some passes) are longer-term and have significant trade-offs.\n\n---\n\n## 1. Async Concurrent Execution — **Implement This Week**\n\n### The Core Problem\n\n`extract_all()` in `deliberus/extraction/focused.py` processes arguments sequentially in a `for` loop. For 10 arguments × 2 sub-steps each = 20 sequential API calls. If each takes 4 seconds, that's 80 seconds just for Pass 2. Parallelizing them all down to the longest single call collapses this to ~4-8 seconds.\n\n### How It Works\n\nThe google-genai SDK's `client.aio` namespace exposes async versions of every method. Instructor wraps this cleanly:\n\n```python\nimport asyncio\nimport instructor\nfrom google import genai\n\nbase_client = genai.Client(api_key=GEMINI_API_KEY)\nasync_client = instructor.from_genai(\n    base_client.aio,  # <-- the async namespace\n    mode=instructor.Mode.GENAI_STRUCTURED_OUTPUTS,\n)\n\nasync def extract_claims_async(arg, source_text, semaphore):\n    async with semaphore:  # rate limiting\n        # Step 2a\n        decompose_result = await async_client.chat.completions.create(\n            model=DEFAULT_MODEL,\n            messages=[{\"role\": \"user\", \"content\": build_decompose_prompt(arg, source_text)}],\n            response_model=DecomposeResult,\n        )\n        # Step 2b\n        result = await async_client.chat.completions.create(\n            model=DEFAULT_MODEL,\n            messages=[{\"role\": \"user\", \"content\": build_decontext_prompt(arg, decompose_result, source_text)}],\n            response_model=FocusedResult,\n        )\n        return result\n\nasync def extract_all_async(scout_result, source_text):\n    semaphore = asyncio.Semaphore(10)  # max 10 concurrent (well within Tier 1 300 RPM)\n    tasks = [\n        extract_claims_async(arg, source_text, semaphore)\n        for arg in scout_result.argument_structures\n    ]\n    results = await asyncio.gather(*tasks, return_exceptions=True)\n    return [r for r in results if not isinstance(r, Exception)]\n```\n\n### Rate Limit Headroom\n\nTier 1 paid limits for Gemini 3 Flash: **300 RPM, 1M TPM**. For a 10-argument article with 10 concurrent calls, you're sending 20 requests simultaneously — that's well under the 300 RPM limit. The semaphore can be set to 20 without risk. Even at 50 concurrent requests you'd be fine on RPM; the TPM limit (1M) is the real constraint, and each article call is well under 10K tokens.\n\n### Expected Speedup\n\nThe instructor async blog benchmarks show 7 sequential calls at 6.17s vs 0.85s concurrent — a **7x speedup**. For Pass 2 specifically (the bottleneck at ~20 calls), expect 5-8x improvement. Total pipeline time: from 3-10 minutes down to ~1-2 minutes.\n\n### Compatibility Note\n\nThe current `instructor.from_genai(base_client, mode=...)` must be changed to `instructor.from_genai(base_client.aio, mode=...)` for the async variant. The `mode=instructor.Mode.GENAI_STRUCTURED_OUTPUTS` works identically. The `max_retries=3` parameter also works in async mode.\n\n---\n\n## 2. Implicit Prefix Caching — **Implement This Week**\n\n### The Core Opportunity\n\nEvery Pass 2 call (and Pass 3) sends the full source text in the prompt. For a Wikipedia article, that's ~3,000-8,000 tokens sent 20+ times. With implicit caching, calls that share an identical prompt prefix get a **90% discount on cached token reads** — and cached tokens process faster (no computation needed for the cached prefix).\n\n### How Implicit Caching Works\n\nGoogle automatically caches identical prompt prefixes across calls within a session. When you send call #2 with the same source text prefix as call #1, the model reads the cached state instead of reprocessing. The discount is automatic — no code changes required. Your bill is reduced in the background.\n\n**Requirements:**\n- Prefix must be character-for-character identical across calls\n- Minimum 1,024 tokens must be in the prefix\n- Works with Gemini 3 Flash (confirmed)\n\n### Critical Structural Change Needed\n\nThe current prompts interleave source text with argument-specific content:\n\n```\n# Current (BAD for caching):\n\"[argument-specific header]\\n\\nSOURCE TEXT:\\n{source_text}\\n\\n[argument-specific content]\"\n```\n\nThis puts argument-specific content BEFORE the source text, which means the shared prefix ends at the first argument-specific word — and the cache never hits.\n\n**Restructure to put shared content first:**\n\n```\n# Better (source text as prefix):\n\"SOURCE TEXT:\\n{source_text}\\n\\n---\\nNow extract from argument structure:\\n{argument_specific}\"\n```\n\nBut even better — move the source text entirely to a **system prompt**, which is always the start of the context window and thus naturally cacheable:\n\n```python\nmessages = [\n    {\"role\": \"system\", \"content\": f\"You are a claim extraction specialist.\\n\\nSOURCE TEXT:\\n{source_text}\"},\n    {\"role\": \"user\", \"content\": f\"Extract from this argument:\\n{argument_specific_prompt}\"},\n]\n```\n\nWith this structure, the system prompt (source text) is identical across all 20 calls for the same article, and the cache will hit reliably starting from call #2.\n\n### Known Issue with Gemini 3 Flash Preview\n\nThere is a documented GitHub issue (`vercel/ai#11513`) that implicit caching doesn't work reliably when tools are defined with Gemini 3 Flash Preview. Since the extraction pipeline doesn't use tools (it uses response schemas), this should not apply. But verify by checking `usageMetadata.cachedTokenCount` in responses — if it stays 0 after the first call, caching isn't hitting.\n\n### Cost Impact\n\nFor a 5,000-token article sent 20 times:\n- Without caching: 20 × 5,000 × $0.50/1M = $0.05 per article\n- With caching (hits on 19/20 calls): 1 × 5,000 × $0.50/1M + 19 × 5,000 × $0.05/1M = $0.0025 + $0.0048 ≈ $0.0073 per article\n- **Savings: ~85% on input tokens**, plus faster processing\n\n---\n\n## 3. Multi-Argument Single-Call Batching — **This Week (Pass 2a)**\n\n### The Idea\n\nInstead of calling decompose (Pass 2a) once per argument, send ALL argument structures to a single call and ask for decomposition of all of them at once. The structured output schema wraps this as a `list[DecomposeResult]`.\n\n### What Makes This Feasible\n\nGemini supports returning `list[SomeModel]` as a structured output. The response schema can be defined as an array:\n\n```python\nclass AllDecomposeResults(BaseModel):\n    results: list[DecomposeResult]\n\n# Single call for all arguments\nall_decompositions = client.chat.completions.create(\n    model=DEFAULT_MODEL,\n    messages=[{\n        \"role\": \"user\",\n        \"content\": f\"\"\"Decompose ALL of these argument structures into atomic assertions.\n\nSOURCE TEXT:\n{source_text}\n\nARGUMENT STRUCTURES:\n{json.dumps([arg.model_dump() for arg in scout_result.argument_structures])}\n\nReturn a DecomposeResult for each argument, in the same order.\"\"\"\n    }],\n    response_model=AllDecomposeResults,\n)\n```\n\n### Trade-offs\n\n**Pros:**\n- Cuts Pass 2a from N calls to 1 call (10 calls → 1 call)\n- Reduces total API round-trips significantly\n- Works within 1M context window for typical articles\n\n**Cons:**\n- Larger prompt may be harder for the model to track — it may conflate arguments or miss some\n- Harder to debug (when one argument fails, you lose all of them)\n- Token limit: 10 arguments × ~500 tokens each = ~5,000 tokens of argument context + source text. Well within limits for typical articles.\n- If quality degrades vs. per-argument calls, this isn't worth it\n\n**Recommendation:** Try this for Pass 2a (decompose), keep Pass 2b (decontextualize) as per-argument concurrent calls. Decompose is simpler and less likely to be confused by batching. Run a quality comparison on 5-10 test articles before committing.\n\n---\n\n## 4. Model Selection by Pass — **Medium Term**\n\n### Current: All Passes Use Gemini 3 Flash\n\nNot all passes require the same intelligence level. Here's a per-pass analysis:\n\n| Pass | Task Complexity | Model Recommendation |\n|------|----------------|---------------------|\n| 1. Scout | High — holistic argument landscape mapping | **Flash (keep)** |\n| 2a. Decompose | Medium — rule-following, mechanistic splitting | **Flash-Lite candidate** |\n| 2b. Decontextualize + Classify | High — nuanced epistemic judgment | **Flash (keep)** |\n| 3a. Cross-structure | Medium — relationship pattern matching | **Flash-Lite candidate** |\n| 3b. Concept disambiguation | High — semantic precision required | **Flash (keep)** |\n| 4. Self-eval | Low — scoring/rating existing output | **Flash-Lite strong candidate** |\n| 5. Embeddings | N/A — separate API | **text-embedding-004** |\n\n### Gemini 3.1 Flash-Lite Performance Numbers\n\n- Output speed: **381.9 tokens/s** (vs 363 t/s for Flash)\n- TTFT: **~6.74s** per Artificial Analysis measurements (worse than Flash's ~1.74s median)\n- Cost: **$0.25/$1.50 per 1M in/out** (half the cost of Flash at $0.50/$3.00)\n\nThe paradox: Flash-Lite has higher throughput but worse TTFT. For a pipeline where you're waiting for sequential results, TTFT dominates — Flash-Lite is actually **slower** for your use case despite being \"faster\" in throughput benchmarks. TTFT of 6.74s vs 1.74s means each Flash-Lite call takes 4x longer to START.\n\n**Recommendation:** Don't switch to Flash-Lite for latency-sensitive passes. Consider it only for Pass 4 (self-eval), where quality requirements are lower and you could run it fully asynchronously without blocking the main pipeline. Do not use it as a drop-in replacement for Flash in Passes 2b or 3.\n\n---\n\n## 5. Context Caching (Explicit) — **Medium Term, High ROI at Scale**\n\nBeyond implicit caching, Google's explicit context caching API lets you pre-cache content with a specific TTL and reference it by name across all calls. This is explicit (you create the cache object) vs. implicit (automatic prefix matching).\n\n### How It Works\n\n```python\n# Create cache once per article\ncache = client.caches.create(\n    model=\"gemini-3-flash-preview\",\n    config={\n        \"system_instruction\": \"You are a claim extraction specialist.\",\n        \"contents\": [{\"parts\": [{\"text\": source_text}], \"role\": \"user\"}],\n        \"ttl\": \"3600s\",  # 1 hour\n    }\n)\n\n# Each subsequent call references the cache\nresult = client.models.generate_content(\n    model=\"gemini-3-flash-preview\",\n    contents=extraction_prompt,\n    config={\"cached_content\": cache.name},\n)\n```\n\n### Pricing Mechanics\n\n- Cache creation: billed at standard input rate\n- Cache storage: $1.00 per 1M tokens per hour (for Flash)\n- Cache read: 90% discount on cached token reads\n\nBreak-even: approximately **4 cache reads per hour per 1M cached tokens**. For a single article extraction (20+ calls within seconds), the break-even happens immediately — you're making 20 reads in under a minute against content cached once.\n\n### Limitation: Compatibility with Instructor/Structured Outputs\n\nThis is the critical unknown. The explicit caching API uses a different call path than the standard `generate_content` with response schemas. As of March 2026, there is no documented confirmation that explicit caching works with instructor's `GENAI_STRUCTURED_OUTPUTS` mode. It likely requires switching to raw `generate_content` + manual Pydantic validation for cached calls. Worth investigating before committing — start with implicit caching (which works transparently) and measure actual cache hit rates first.\n\n---\n\n## 6. Google Batch API — **Not for This Pipeline**\n\nThe Batch API offers 50% cost reduction with a **24-hour turnaround**. It supports structured outputs via `response_schema`. However, it is explicitly designed for offline/asynchronous workloads — \"not suitable for real-time pipeline processing.\"\n\nFor the Deliberus extraction pipeline (which needs results within minutes for interactive use), the Batch API is not applicable to the real-time flow. It would be valuable for:\n- Bulk re-extraction of an entire article corpus offline\n- Generating training data\n- Running quality evaluations across many articles\n\nFile away for when there's a corpus of 1,000+ articles to process in bulk at half cost.\n\n---\n\n## 7. Local GPU Inference (Darwin GTX 1650) — **Limited Applicability**\n\nDarwin has a GTX 1650 (4GB VRAM) running Ollama. The question is whether any pipeline passes could run locally.\n\n### GTX 1650 VRAM Constraint\n\nAt 4GB VRAM, the largest model that fits fully is roughly **7B parameters at Q4 quantization** (needs ~4.1GB). In practice, you're limited to 4B models for reliable operation, or 7B with severe quantization (Q2/Q3) that degrades output quality significantly.\n\n**Models that fit in 4GB:**\n- `gemma3:4b` (~2.5GB) — structured output capable, ~60-80 tokens/s on GTX 1650\n- `llama3.2:3b` (~2.0GB) — reliable JSON mode, ~80-100 tokens/s\n- `phi4:3.8b` (~2.3GB) — strong instruction following\n\n**Models at 7B that may struggle:**\n- `gemma3:12b` — does NOT fit (needs 7-8GB VRAM)\n- `llama3.1:8b` — does not fit at full precision; Q4 needs ~5GB (too large)\n\n### Quality Reality Check\n\nFor the Deliberus extraction pipeline, the core task is nuanced epistemic classification (empirical vs. normative vs. definitional claims, epistemic status, decontextualization). This requires strong instruction following and fine-grained semantic understanding. Benchmarks consistently show that **sub-7B models degrade significantly on complex structured extraction tasks** — they produce malformed JSON, miss required fields, and lose semantic precision.\n\n**Realistic assessment:** A `gemma3:4b` on the GTX 1650 would handle Pass 4 (self-eval/scoring) acceptably, and potentially Pass 2a (decompose, which is more mechanical). It would likely fail at Pass 2b (decontextualize + classify), where subtle epistemic distinctions matter.\n\n**Speed on GTX 1650:** ~60-80 tokens/s for a 4B model. A typical Pass 2a response is ~200-400 tokens, so ~3-6 seconds per call. Comparable to Gemini Flash for simple tasks, but with worse quality and no concurrent request advantage.\n\n**Conclusion:** Don't route complex passes to Darwin's GPU. Consider routing Pass 4 (self-eval) as an experiment, since it scores already-extracted claims rather than requiring high-fidelity extraction. This frees up Gemini API budget for the hard passes.\n\n---\n\n## 8. Prompt Length and Structured Output Mode\n\n### Does Prompt Length Affect Latency?\n\nYes, but less than you might think for Flash models. The model's TTFT (time to first token) increases with prompt length because it must process all input tokens before generating output. For a 5,000-token source text sent in every prompt, this adds processing time proportional to token count. Implicit caching eliminates this by skipping reprocessing of cached prefixes.\n\nStructured output mode (`GENAI_STRUCTURED_OUTPUTS`) vs. free text + parse: structured mode is slightly slower at generation (the model constrains outputs to the schema) but eliminates post-processing failures and retry loops. Given that the `max_retries=3` in the current code means a failed parse can triple call time, structured mode is net faster in practice.\n\n### Compressing Source Text\n\nOne option for passes that only need argument context (not full text verbatim): summarize or truncate the source to just the relevant paragraphs. Pass 2a (decompose) only needs the text surrounding each argument's `source_span`. Passing only the relevant 300-word excerpt instead of the full 2,000-word article could cut input tokens for that call by 80%.\n\n---\n\n## Implementation Priority\n\n### This Week (high confidence, low risk)\n\n1. **Async concurrent Pass 2** — change `extract_all()` to use `asyncio.gather()` with `client.aio`. Largest single speedup, well-understood pattern, no quality impact. Expected: 5-8x speedup on Pass 2.\n\n2. **Source text as system prompt** — move `{source_text}` to the system prompt in all Pass 2 and Pass 3 calls. Enables implicit caching. Check `usageMetadata.cachedTokenCount` in response to verify hits. Expected: 85% reduction in input costs, modest latency improvement from call #2 onward per article.\n\n3. **Multi-argument Pass 2a batching** — test sending all argument structures to decompose in one call. Run quality comparison against per-argument approach on 5 test articles. Implement if quality holds.\n\n### Next Week (requires testing)\n\n4. **Pass 4 on Flash-Lite or local model** — route self-eval scoring to either `gemini-3.1-flash-lite-preview` or `gemma3:4b` on Darwin. Measure quality vs. latency trade-off.\n\n5. **Explicit context caching** — investigate compatibility with instructor structured outputs. If compatible, implement with a TTL of 600s (covering one article extraction session).\n\n### Longer Term\n\n6. **Batch API for corpus processing** — when there's a corpus of 100+ articles to process offline, submit as batch jobs at 50% cost with JSONL input files.\n\n7. **Streaming architecture** — pipeline Pass 2 results into Pass 3 as each argument completes rather than waiting for all Pass 2 results. Reduces total wall-clock time for the full pipeline. More complex to implement.\n\n---\n\n## Quick Reference: API Limits\n\n| Model | Tier 1 RPM | Tier 1 TPM | Input cost | Output cost |\n|-------|-----------|-----------|-----------|------------|\n| gemini-3-flash-preview | 300 | 1M | $0.50/1M | $3.00/1M |\n| gemini-3.1-flash-lite | 300 | 1M | $0.25/1M | $1.50/1M |\n| Batch API (any Flash) | async (24h) | — | 50% of standard | 50% of standard |\n\nAt 300 RPM with 10 concurrent requests per article, you could theoretically process 30 articles simultaneously without hitting limits. The TPM limit (1M) with 5,000 tokens per call = 200 calls before throttling — your pipeline of ~25 calls per article leaves ample headroom.\n\n---\n\n*Sources consulted: Gemini API docs (caching, batch, rate limits), instructor async docs, google-genai SDK documentation, Artificial Analysis benchmarks, Gemini 3.1 Flash-Lite launch blog.*\n"}