{"path":"research/frontier-extraction-experiment.md","content":"# The Frontier Extraction Experiment (Dogfood Run 3F)\n\n*July 9, 2026. The Gemini API credits ran dry mid-run-3, so the founder proposed an experiment: have the frontier model driving the development session (Claude Fable 5) execute the entire extraction pipeline by hand, at full care, on the same texts the pipeline would have processed. What follows is the protocol, the findings (H-series, continuing run 2's G-series), and the comparison design for when the pipeline can run again.*\n\n## What this measures\n\nThe pipeline's output quality confounds two things: the **ontology** (are atomic claims, schemes, critical questions, contested concepts, and typed residues the right representation?) and the **model** (is gemini-3-flash good enough to produce that representation?). A frontier model executing the same passes manually, schema-faithfully, at unlimited care, gives a **quality ceiling**: everything the manual extraction catches that the pipeline misses is model headroom; everything the manual extraction also struggles with is ontology work. The artifacts are the benchmark.\n\n## Protocol\n\n- Topic prediction was pre-registered before any text was fetched: the founder asked for two topics predicted from his own priors; the predictions (AI existential risk; psychedelic-assisted therapy) and their reasoning were stated in conversation first.\n- Texts, batch 1 (AI existential risk, a worldview clash): Yudkowsky, \"Pausing AI Developments Isn't Enough. We Need to Shut it All Down\" (TIME, 2023) and Andreessen, \"Why AI Will Save the World\" (a16z, 2023). Batch 2 (psychedelic-assisted therapy, an evidentiary clash): Doblin, \"The future of psychedelic-assisted psychotherapy\" (TED2019) and van Elk, \"Two reasons I'm sceptical about psychedelic science\" (The Conversation, 2024). The two batches were chosen orthogonal on purpose: batch 1 disagrees about values and predictions, batch 2 disagrees about whether an effect is real, which stresses different machinery (methodology tiers, testimony, checkable cruxes).\n- Passes executed manually: scout (argument structures with verbatim thesis quotes), focused extraction (decompose, decontextualize, classify: stranger test, four types, epistemic modality), relationships with Walton-scheme detection, contested concepts, critical questions, self-evaluation, weighing detection, and the cross-source connect pass. Field names mirror the production schema; per-claim reasoning fields were omitted for artifact economy (declared deviation).\n- Artifacts: two extraction files plus one cross-source file under `docs/research/frontier-extraction-experiment/`.\n\n## Headline numbers\n\n**Batch 1, AI existential risk:**\n\n| | Yudkowsky | Andreessen | Cross-source |\n|---|---|---|---|\n| Extracted claims | 42 | 54 | |\n| Implicit premises surfaced | 3 | 3 | |\n| Relationships | 26 | 32 | 22 edges |\n| Contested concepts | 4 | 5 | |\n| Edge profile | | | attack-dominant (19/1/1) |\n\n**Batch 2, psychedelic-assisted therapy:**\n\n| | Doblin | van Elk | Cross-source |\n|---|---|---|---|\n| Extracted claims | 29 | 16 | |\n| Implicit premises surfaced | 3 | 2 | |\n| Relationships | 20 | 16 | 17 edges |\n| Contested concepts | 4 | 3 | |\n| Edge profile | | | attack-dominant-with-agreement (14/2/1) |\n\n## Findings\n\n**H1: The debate lives at the implicit nodes.** The single most consequential discovery. Yudkowsky never states the orthogonality thesis or instrumental convergence; Andreessen never states that goal-directedness requires being alive or that present-tense human control extends to future capability levels. Yet the deepest cross-source edge in the whole debate is between two *implicit* premises (ai1 attacks yi1): claims neither author wrote down. An extraction that only captures stated claims would connect these essays at their surfaces (bans vs builds) and miss that the real crux is unstated on both sides. The pipeline has an implicit-premise pass; whether it surfaces THESE premises is the sharpest single test the comparison run should score.\n\n**H2: Affect-carrying testimony has no comfortable slot.** Yudkowsky's most powerful passage is his partner's email about their daughter losing a tooth. Propositionally it is testimony about insider grief (extracted as such, y34); rhetorically it is the essay's emotional core. Any extraction preserves the proposition and loses the force. This is not a bug to fix by force-typing emotion as evidence; it is a boundary of the representation worth stating honestly: the graph captures reasoning, and some of what moves humans in argument is not reasoning. (The analysis-attunement dialectic predicts exactly this boundary.)\n\n**H3: Polemic is handled by schemes, not by laundering or moralizing.** Andreessen's cult section is argument-from-bias and genetic-fallacy territory. The extraction neither launders it into neutral empirical claims nor editorializes against it: motive-attributions are typed as causal hypotheses, tagged with bias schemes, and the critical questions carry the audit (\"does payment to study a risk predict overstating it more than industry payment predicts understating it?\", \"applied symmetrically, does the author's venture position discount his optimism equally?\"). Run 2's G8 finding (polemic in, attributed sociology out) generalizes: the scheme-plus-CQ machinery is the right treatment for rhetorical heat.\n\n**H4: Tense-scope bundling is a systematic polemic pattern.** Andreessen's \"it is owned by people and controlled by people\" is true in the present tense and predictive in the future tense, and the argument needs both while asserting only the checkable one. The same fusion appears in \"AI doesn't want\" (true of 2023 chatbots, contested for optimizing agents). A decomposition cue worth adding to the pipeline: when a present-tense fact is deployed against a future-tense worry, split the tenses.\n\n**H5: Two new weighing dialects for the lexicon.** Yudkowsky's \"priority above\" (rank-ordering: AI extinction above nuclear exchange) and Andreessen's \"single greatest risk\" (superlative ranking: China dominance above extinction) are both explicit weighings the v1/v2 lexical detector would miss. Rank-ordering and superlative-ranking join \"outweigh\" (moral philosophy) and \"net benefits\" (policy analysis) as register-specific dialects; the count of mapped registers is now five, strengthening run 2's case (G10) for the semantic detection tier.\n\n**H6: A bridging signal hides inside a maximally hostile debate.** One cross-source SUPPORTS edge exists: Andreessen's \"AI is highly likely to be the control layer for everything in the world\" supports Yudkowsky's claim that what happens past the capability threshold is the only thing that matters. The two essays agree the stakes are near-maximal and disagree on sign. That shared premise is invisible in the culture war and precisely the kind of structure the bridging signal exists to surface.\n\n**H7: The residue prediction, registered.** Walking the central clash down before the platform can: the aliveness-vs-optimization layer is classified *empirical and dissolving* (agentic systems increasingly exhibit the disputed behavior class); the primary surviving residue is predicted **structural** (which evidential standard applies to one-shot irreversible risks: falsifiability-first or precaution-first, the a24-vs-y27 seam); the secondary residue is **risk-posture, axiom-choice-adjacent** (how much certain foregone benefit a small probability of total loss outweighs, the corner where expected-value reasoning is itself the contested axiom). Not fittingness. When credits return and the descent runs on the platform, this prediction gets scored.\n\n*Later reading, Aug 2026: H8 is this corpus's clearest datapoint on the structure-versus-scale question — a frontier model doing the arsenal's job at the ceiling. Run 6 then measured the reverse case, where a constrained enum beats the frontier model outright, and [structure-versus-scale.md](structure-versus-scale.md) works out what the pair implies for the architecture bet.*\n\n**H8: Cost and the arsenal boundary.** This extraction cost zero API dollars and roughly two hours of frontier-model session attention; the pipeline does it for about thirty cents in two minutes, unattended, and stores the result in a living graph with computed strength, embeddings, and challengeable verdicts. The experiment is not an argument for manual extraction; it is the measurement of what the cheap path should be aspiring to, and a live demonstration of the arsenal-and-harness boundary: the harness (a frontier model) can do the arsenal's job at the ceiling, and the arsenal's job is to make 95 percent of that ceiling cost pennies.\n\n## Batch 2 findings, and what the two batches jointly show\n\n**H1 confirmed across a second, orthogonal debate: the crux is stated on one side, implicit on the other.** This is now the experiment's strongest cross-batch result. In batch 1 the decisive clash was Andreessen's implicit \"goal-directedness requires aliveness\" against Yudkowsky's implicit orthogonality thesis: neither written down. In batch 2 the decisive clash is van Elk's clearly stated \"the between-arm difference is placebo-driven\" (v6) against Doblin's never-stated \"the between-arm difference is the drug\" (di1): the skeptic states the crux, the advocate leaves it implicit because to him it is obvious. Same structure, opposite direction. The lesson generalizes past both topics: the load-bearing node of a real disagreement is routinely the one that at least one side never says out loud, and an extraction that only captures stated claims connects the two texts at their surfaces (heal-the-world vs be-cautious) and misses where they actually collide. Implicit-premise recall is therefore the single most important axis of the whole comparison, and it is where a cheap pipeline is most likely to fall short.\n\n**H9: register drives residue type, exactly as the wager predicts.** Batch 1 (worldview clash) bottomed out in a structural residue (which evidential standard governs one-shot irreversible risks) plus a risk-posture residue, with the aliveness question dissolving empirically. Batch 2 (evidentiary clash) is mostly empirical and substantially dissolving: the parties agree the phenomenon is real (van Elk concedes psychedelics \"appear capable of enhancing wellbeing\") and dispute only attribution, a checkable question the 2024 FDA-advisory and ICER reviews already moved on. Its one surviving residue is again structural (burden of proof under asymmetric error costs), not fittingness. Predicted residue fraction: lower for the evidentiary debate than the worldview debate. This is the convergence wager's core claim showing up as a measurable difference between debate genres: evidentiary disputes decompose further than worldview disputes, and the residue that survives is more often structural than value-bedrock. Neither batch produced a fittingness residue, which is itself a finding: fittingness (the retribution case from run 1) may be rarer than the typed-residue taxonomy's four equal-looking boxes suggest.\n\n**H10: the shared contested concept is the bridge, and it can be merged without flattening the fight.** Both batch-2 texts contest \"efficacy\" with the identical two senses (package-improves-patients vs drug-causes-the-effect). Typing them identically in both files makes the cross-source concept-merge detectable while the opposing claims about that concept stay separate nodes. This is disagreement-preservation operating at the concept layer rather than the claim layer: the term merges, the dispute does not. A concrete design cue for the pipeline's contested-concept pass: concept identity across sources is a bridge to draw, not a reason to collapse the claims that use it.\n\n**H11: testimony and statistics coexist in one advocacy text and must not be leveled.** Doblin's talk reports RCT-class numbers (107 participants, 23 vs 56 percent, two-thirds at one year: methodology and sample-size fields populated) and leans on two vivid single-case narratives (Marcela, Tony). The manual extraction keeps them at different evidence tiers with the anecdotes flagged for selection, rather than letting the emotional force of the cases borrow the authority of the trial data. Whether the pipeline holds that line, or quietly upgrades a moving story toward the strength of a statistic, is a batch-2-specific test the comparison should score.\n\n## Comparison protocol (when the pipeline can run)\n\nExtract all four texts through production; then score, per text: claim recall against the manual set (which manual claims have no pipeline counterpart), atomization grain (bundling rate, tense-scope splits), implicit-premise recall (H1 is the headline test across both batches: does the pipeline surface orthogonality and the aliveness premise in batch 1, and the drug-attribution and burden-of-proof premises in batch 2?), evidence-tier discipline (batch 2 test: does the pipeline keep Doblin's anecdotes below his statistics?), scheme agreement on shared relationships, contested-concept overlap including coinage sensitivity (\"doomer\" and \"entropy\" are the two batches' coinage tests, after run 2's G9) and the H10 cross-source concept-merge, weighing detection against the now seven-plus known dialects, cross-source edge recall against the manual edge sets (22 for batch 1, 17 for batch 2) including the bridging SUPPORTS in each, and disagreement-preservation scores on all four extractions. Finally, score the two registered residue-descent predictions (batch 1: structural plus risk-posture; batch 2: mostly empirical, one structural residue, lower residue fraction) against the platform's own descents. Every delta is either model headroom (pipeline missed, ceiling caught) or shared difficulty (both struggled: ontology work).\n\n## Relation to the evaluation program\n\nThe funding applications propose measuring whether groups reason better on this substrate. This experiment measures something upstream: how much of the reasoning that is actually in a text the substrate can hold, at ceiling versus at production cost. The two questions compound: the group evaluations inherit whatever the extraction layer loses.\n"}