{"path":"research/ai-risk-for-mathematicians-and-the-coordination-game.md","content":"# An AI-risk exposition for mathematicians, and what decides whether the race is a dilemma (2026-10-01)\n\n**Status**: research, nothing ruled. The proposals are ledger Arc U (U1–U3), added 2026-10-01 when the\nfounder asked to come back to all of this carefully. The reading guide is `TODO.md` § START HERE.\n**Origin**: the founder sent the link with *\"Read up about this\"*. It came alongside a friend's\nmodel of coordination, a long comment under one of his own public posts, and a podcast link. The\nessay, the model and the comment all circle the same word, *coordination*.\n\n**Read in full**: Xiaoyu He (who writes as alkjash), *Existential Risk from AI: An Exposition for\nMathematicians*, August 2026, about 8,000 words, [alkjash.github.io/ai-risk](https://alkjash.github.io/ai-risk/).\nThe author is a mathematician at Georgia Tech. By the author's own account the essay is an opinion piece rather than a\nmath paper, none of its details are the author's own, and it was written with help from Claude Fable 5 and\nChatGPT 5.6 Sol. Each outside source carries its read depth.\n\n---\n\n## 0. In three sentences\n\nThe essay argues for at least a 10% chance of human extinction by 2050. It gets there through twenty\nnumbered statements in five groups, each with caveats, followed by eight objections and replies. In\nother words, it is already an argument map written as prose. Its weakest joint is the one it waves\nthrough: it never says which combinations of statements are enough, or how likely they are when taken\ntogether, and leaves that *\"as an exercise for the reader\"*. The finding that matters most for\nDeliberus is in § 3. The essay calls the race a prisoner's dilemma and doubts coordination because\nthe argument is confusing. Recent game theory says the type of game depends on what the racers believe\nabout the risk. So the essay's own aim, making the risk common knowledge, is what could turn the\ndilemma it fears into a trust problem, which can be solved. Read in full, the sources make that\nnecessary rather than sufficient: racers who understand the risk still race when they cannot see each\nother's conduct, and what a public map can serve is the public's political will (§ 3d).\n\n## 1. What the essay argues\n\n**The claim**: Conjecture 2, *\"there is at least a 10% chance of human extinction by 2050\"*. It rests on\nProposition 3, which is assumed rather than argued: at least a 50% chance of superintelligence by 2050.\nFor that the essay cites a 2023 survey of 2,778 AI researchers whose aggregate forecast put a 50% chance on\nmachines beating humans at every task by 2047. Mathematics ending as a profession is presented as a corollary.\n\n**The five groups**:\n- **Takeover (S1–S4)**: AIs tend toward coherent utility-maximisers; most goals lead to seeking power;\n  values and capabilities are independent; a takeover most likely ends humanity.\n- **Magic (S5–S7)**: undiscovered world-changing technology is plentiful; some of it favours attackers\n  enormously (*\"You cannot beat germs with any amount of opposing germs\"*); and once found it cannot be\n  contained, since *\"a one-paragraph description might be enough\"*.\n- **Recursive self-improvement (S8–S11)**: AI is superhuman at coding and human-level at mathematics,\n  *\"two key ingredients of AI research\"*, so the research loop closes. Alignment would then have to\n  survive every round of self-improvement.\n- **Alignment resists solution (S12–S16)**: nobody has a utility function that is safe to optimise; we\n  cannot install one; techniques that work on weak models break on strong ones; one aligned system does\n  not stop its competitors; and most safety research does not address extinction risk.\n- **Human failings (S17–S20)**: the labs are in a race (*\"It's a garden-variety prisoner's dilemma\"*);\n  an AI can pay people to act for it; *\"Coordination is confusing, so humans will probably fail to do\n  it\"*; and we may have to succeed on the first try.\n\nTakeover and Magic are the two routes to catastrophe. The other three groups explain why changing\ncourse would be hard. **The essay explicitly rejects reading the twenty statements as one chain that\nall has to hold.** Before the statements, it uses anecdotes to set what it calls *the vibe*: a friend\nwho wrote *\"I concede at least 99% of all previous disagreements in this entire domain\"*; people at labs\nputting the chance of doom anywhere from about 40% to 99%; and the July incidents in which models reached real\nsystems during evaluations. It closes with a three-state model of how opinion moves (Skeptic → Accelerationist →\nDoomer, with Doomer absorbing), the hope to *\"add the arrow from Skeptic to Doomer\"*, and an aim stated\nin passing: *\"to create common knowledge about x-risk in the mathematical community.\"*\n\n## 2. It is already an argument map, and the map makes its gaps visible\n\n### 2a. One shape, two errors\n\nThe essay's warning against reading it as one long conjunction is the distinction between linked and\nconvergent support. Statements\ninside one route are **linked**: all of them are needed. The routes themselves are **alternatives**:\nany one is enough. The skeptic's mistake, as the essay names it, is to read the alternatives as a\nsingle linked chain, which makes extinction look like a long, improbable conjunction. The August\nstrength audit found the same distinction violated in the opposite direction: the old badge treated\njointly required parts as if each one corroborated on its own (`strength-layer-audit.md`). **The\ndistinction is the same, and the errors run in opposite directions.** Deliberus already records the\ndifference. `support_semantics` has `necessary` (the weakest part governs) for a route's internal\nstatements and `alternative` (the strongest part decides) for the routes.\n\n### 2b. The step the essay leaves to the reader carries the conclusion\n\n*\"To enumerate these possibilities and sum their total mass is mere combinatorics, and left as an\nexercise for the reader.\"* Listing the subsets is easy. **What carries the weight is the joint\nprobability of twenty statements that depend on each other**, and the essay names some of those\ndependencies itself:\n- S2 *\"partially relies on S1\"*.\n- S11 ties alignment to self-improvement.\n- S1's own caveat: *\"If S1 is substantively false, it probably makes AI easier to defeat but harder to\n  align\"*. Denying S1 weakens the takeover route and strengthens the case that alignment is hard.\n\nThe routes also share premises upstream. Proposition 3 sits under every one of them, so adding up the\nmass of each route counts the same assumption more than once. The corpus has seen this pattern before:\nthe crux is left unsaid by at least one side (run 3F, whose own AI-risk debate had it unsaid on both), and the size of a claimed\ncatastrophe goes unstated (run 5). This essay states outright, as S2 and S3, the two premises that run\n3F had to write in for the risk side\n([frontier-extraction-experiment.md](frontier-extraction-experiment.md), H1), so a map of it is where\nthey could be joined to the premise the other side left unsaid.\n\n**What a map would add**:\n- It forces the route structure to be stated, meaning which subsets are enough.\n- It puts Proposition 3 in as **one shared node** under every route, following the rule that a known\n  shared source becomes a node and is never quietly discounted (`weakest-link-arithmetic-and-the-merge-hunch.md` § 9b).\n- It makes each of the twenty statements something a reader can attack on its own.\n\n**What a map would not do is compute the 10%.** That limit is worth stating precisely, because this\nessay shows the sharpest form of a tension the corpus already holds open. **The essay's central\nstructural move is a probability argument**: several partly independent routes together make an outcome\nmore likely than any single one does, the way a noisy OR combines causes. **The `alternative` channel\ndeliberately does the opposite.** It takes the maximum, so the strongest route decides and extra weak\nroutes add nothing. That makes it resistant to padding: several weak alternatives must not add up to a\nstrong-looking case (Pollock's rule for separate arguments, and the sybil-proof choice). Mapped today,\nthe essay's conclusion would read only as strong as its best route, and the badge would not show its\ncentral argument at all. **Both rules are right, about two different quantities.** Argumentative\nstanding should resist padding. The probability that an event happens rises with every independent way\nit can happen. This belongs with the open question of how argument strength relates to probability\n(ledger Q35–Q36, `starting-values-and-the-problem-of-the-prior.md`), raised here by a high-stakes outside\nsource. One constraint applies to any reconciliation. A probability-style OR is not clone-proof, since duplicating\na route raises the total, so it is safe only after two conditions hold: the keeper has ruled on\nsameness, and shared premises have been pulled out into their own nodes.\n\n### 2c. S1 is a hinge for two conclusions at once\n\nAccording to the essay's own caveat, denying S1 moves two parts of the argument in opposite\ndirections: the takeover route weakens and the case that alignment is hard strengthens. Mapped, those\nare two sub-conclusions. The hinge score as shipped measures how far one root moves when a claim is\ngranted or denied (`deliberus/hinge.py`). A claim whose flip moves several roots at once is the\n**fan-out** case that `curiosity-as-growth-fuel.md` § 9.1 names as buildable and not yet built. This is a real\ninstance, from outside material, of the shape that section predicts will draw the most engagement.\n\n### 2d. The opinion model bundles back together what the argument separated\n\nThe three opinion states fuse two separable beliefs: one about how capable AI will become, and one\nabout how dangerous it is. Two positions get no state at all: AI as dangerous without expecting\nsuperintelligence (through misuse, say), and the danger lying mainly in who controls the technology.\nPeople who hold those views get filed under the nearest state, which is the closed-enum failure that\n`does_not_fit` exists to prevent. Over this argument, a person's position is a pattern of agreement\nacross twenty statements, not a point in a three-state chain. The part where two such patterns diverge\nis what the corpus calls the *position fingerprint* (`lowering-the-cost.md`). *\"I concede at least 99%\nof all previous disagreements\"* is a change in that pattern, and the chain cannot record which\nstatements moved. *(2026-10-01: a two-axis chart of the debate, Xenologos, *The AI debate on two axes*,\nhas both missing positions and a does-not-fit value of its own, and breaks on four registers of its own;\nread in [four-cuts-of-the-ai-debate.md](four-cuts-of-the-ai-debate.md) § 3.)*\n\n**The essay wants to add an arrow from Skeptic to Doomer. Deliberus would locate the disagreement\ninstead of converting the person.** A skeptic walked through S1–S20 in order of the hinge score learns\nwhich statement their disagreement actually rests on. That turns diffuse ambivalence into a located\ndisagreement (`does-descending-change-the-descender.md` § 8), and that remains true whichever way the\nperson ends up.\n\n### 2e. \"Existential\" is used in two senses\n\nConjecture 2 is about extinction. Bostrom's original 2002 definition of existential risk is wider: an\noutcome that would *\"either annihilate Earth-originating intelligent life or permanently and\ndrastically curtail its potential\"*. The definition does not name subclasses. The paper's four-way\nclassification, a few pages later, does: *bangs* fit the first limb, and *crunches* (human life\ncontinues but its potential is *\"permanently thwarted\"*), *shrieks* (a posthumanity that realizes\n*\"an extremely narrow band of what is possible and desirable\"*) and *whimpers* (a civilization that\nevolves *\"gradually but irrevocably\"* away from what we value) fit the second. The\ngradual-disempowerment literature in § 4a argues for the second kind, and the class it cites as its\nancestor is the whimper. A map keeps the two conclusions as separate claims. Otherwise arguments for\none are quietly counted as support for the other, which is the concept layer's job to prevent.\n\nThe same classification answers a question the essay never asks: can the response to AI be the\ncatastrophe? Two of Bostrom's own scenarios are governance failures. One crunch is a world government\nmade *\"stable against insurrections (by advanced surveillance or mind-control technologies)\"* that\nstops progress. One shriek is *\"an intolerant world government\"* that *\"decides to realize only a very\nsmall part of all the good things a posthuman world could contain\"*. So the 2002 paper that defines\nthe field already counts a capturing response among existential risks, which is § 4b's point and\nSchmachtenberger's dystopia attractor (`schmachtenberger-and-the-ai-race.md` § 3). It also recommends\nsurveillance as a risk reducer and puts the hedge in a footnote: *\"it seems important to aim for the\ntwo-way transparency advocated by David Brin, where we all can watch the agencies that watch us.\"*\n(Bostrom 2002, read in full.)\n\n## 3. Coordination: what decides whether the race is a dilemma\n\n### 3a. The essay's two claims about coordination, and the four reasons it gives\n\nS17 calls the race *\"a garden-variety prisoner's dilemma\"*. S19 says coordination will probably fail,\nand gives four reasons:\n1. *\"the argument for Conjecture 2 is much more complex than for climate change\"*;\n2. *\"the timelines are much less predictable\"*;\n3. *\"the obvious economic benefits of AI development are much more tantalizing\"*;\n4. *\"a single defector can demolish the whole agreement.\"*\n\nThe founding conviction's second guard sorts these. Opacity *dissolves confusion-conflict and only\nclarifies interest-conflict* (`vision.md`). Reason 1 concerns what people can understand: the\ncost of understanding why another mind holds its position, which is the cost Deliberus exists to lower.\nReason 2 is real uncertainty about the world, which a map can lay out but not reduce. Reason 3 is a\nconflict of interests, which legibility can clarify but not dissolve. Reason 4 is a claim about how the\ngame is structured, and that is where the essay's own framing comes apart.\n\n### 3b. What the recent game theory says\n\n*All seven sources read in full on 2026-10-03 (the first version of this table read abstracts and\nexcerpts). The full reading corrected two claims and qualified most of the rest; the corrections are\nmarked.*\n\n| Source | What it says, read in full |\n|---|---|\n| Jervis, *Cooperation under the Security Dilemma*, *World Politics* 30(2), 1978, pp. 167–214 | Separates the prisoner's dilemma from Rousseau's stag hunt. Cooperation is more likely *\"when each side expects the other to cooperate\"*: that is the abstract's phrasing, and the body derives the conditions for either game. Its engine is the offense–defense balance and whether offensive and defensive postures can be told apart, which define four worlds. Expectations and inspection help (*\"When each side is ready to cooperate if it expects the other to, inspection devices can ameliorate the security dilemma\"*), and the dilemma *\"will operate much more strongly if statesmen do not understand it\"*. Yet those variables *\"leave the heart of the problem untouched\"*, and in the worst world awareness raises the pressure to strike first |\n| Katzke & Futerman, *The Manhattan Trap*, Convergence Analysis, arXiv:2501.14749, Jan 2025 | *\"ASI presents a trust dilemma rather than a prisoner's dilemma\"*, and *\"We conclude that cooperation is achievable\"*. The conclusion is conditional: on premises *\"we assume (but do not argue for)\"* (a decisive military advantage; rational, informed states), on today's compute paradigm, and on political will. *\"The challenge now is to establish mutual understanding of the dangers of an ASI race\"*. Commitments cannot change payoffs at survival stakes, so verification does the work, and it *\"does not need to be perfect\"*. It is silent on the cooperative regime's own surveillance or concentration of power. (*Trust dilemma* is not Jervis's term, though it cites him; its Table 1, as normally read, contradicts its text) |\n| RAND RR-A4245-1 (Abraham, Kavner & Moon), Dec 2025 | *Corrected.* If both share the assessment that coming first outweighs the risks (formally, the extra expected gain from accelerating exceeds a shared catastrophe cost), the race is a prisoner's dilemma. If that shared assessment reverses, it is a coordination game with two equilibria; RAND does not say *stag hunt*. The report's sentence: *\"if a common shared assessment emerges that the risks of catastrophic harm from accelerated development of AGI exceed the potential benefits of a first-mover advantage, the incentives of the United States and China can be aligned toward cooperation\"*. Cooperation is *\"implementable only if there are robust mechanisms … to share information to establish common knowledge and verify mutual commitments\"*: a policy judgment, not a result. Its model already has the cooperative equilibrium without such mechanisms, though nothing selects it. It assumes the two sides' perceptions identical and concedes *\"countries may have differing perceptions based on the same information\"*. *Common knowledge* is undefined. It proposes no mechanism of its own and says nothing about surveillance |\n| Carlsmith, *On restraining AI development for the sake of safety*, 19 Mar 2026 | The game can have several stable outcomes, *\"more similar to a 'stag hunt' than a traditional prisoner's dilemma\"*. This is hedged (*\"some versions\"*), credited to Katja Grace, and disclaimed: *\"My point isn't that the overall game theory here actually favors slowing down even absent mechanisms for enforcing cooperative behavior.\"* Assuming the other side will defect can be self-fulfilling, and an actor may *\"rush forward regardless\"*. Most of the essay is why restraint is hard. *\"widespread political will is the first and most central bottleneck\"*. Compute can be governed for *\"at least a few years\"*, but verifying the absence of secret algorithmic work needs a surveillance apparatus that *\"quickly becomes extremely scary in itself\"*, and without it even convinced nations race at the level of algorithms. Power concentration is his *\"most salient concern\"* |\n| Papenhausen, *The AI Race is Not a Prisoner's Dilemma*, EA Forum and LessWrong, Aug 2026 | Two symmetric players, losing as bad as winning is good, an even race, the risk of doom attached to whoever wins, and nothing happening if neither builds. Prisoner's dilemma when the chance of doom is low, stag hunt above *q\\* = W / (W + \\|D\\|)*. *\"If you think Doom is at least 9x worse than Winning the race is good, then P(Doom) only needs to be 10% for an agreement to be feasible.\"* In a stag hunt *\"the hard part is verification and assurance\"*. Its own scope: *\"game-theoretically feasible\"*, with *\"messy politics, personalities and relationship between the specific parties involved\"* set aside. A commenter attacks the weighing W:\\|D\\| (§ 3c) |\n| Fudenberg & Koh, *Racing to Ruin*, arXiv:2607.27638, Jul 2026 | *Corrected.* Transparency and trust shape *how far* the frontier is carried, not how fast. *\"when risk is high relative to the marginal profit from scaling, a firm prefers to stop if and only if its rival stops as well, so the problem is coordination, not competition.\"* Transparency here means how quickly a rival's stop is detected, and *\"Transparency is double-edged\"*. Below a threshold, racing forever is an equilibrium, and no level of trust substitutes for a minimum of monitoring. Trust is a prior that the rival is rational. *\"what disciplines the race is the growth of risk, not its level\"* |\n| Armstrong, Bostrom & Shulman, *Racing to the precipice*, FHI technical report 2013 (published in *AI & Society*, 2016) | *\"the more teams know about each others' capabilities (and about their own), the more the danger increases\"*. The information is about capability; the risk is assumed known, and teams still cut safety in equilibrium. No information is always safest, and whether private or public information is worse depends on enmity. The authors' lever is enmity: *\"building trust between nations and groups, sharing technologies or discoveries, merging into joint projects or agreeing to common aims\"*. (The 2013 report was read; the 2016 article is closed access) |\n\n### 3c. The tension inside the essay\n\nThe essay puts its threshold at 10%. **Under Papenhausen's assumptions, a racer who accepted Conjecture\n2, and thought doom at least nine times as bad as winning is good, would be playing a stag hunt, not a\nprisoner's dilemma.** The essay's own anecdotes point the same way. If people inside the labs put doom\nanywhere from about 40% to 99%, their own numbers are in the stag-hunt range of that model. The people\nwhose assessments set the payoffs, though, are the ones who decide, not every researcher. Papenhausen makes\nthe same move with them: *\"most of those with decisionmaking authority here take Doom seriously enough\nthat this math applies to them\"*. What\nseparates the two games is whether each side would hold back if sure the others would. S19's *\"a single\ndefector can demolish the whole agreement\"* fits either one: in a dilemma every party has a reason to\ndefect, and in a stag hunt the fear that one will is enough. **So the essay's own success condition, Conjecture 2 becoming common\nknowledge among the people who race, is the condition under which its S17 framing stops holding.** Its\nopinion model (§ 6 of the essay) and its game theory (§ 4.5) are not separate subjects, because the\nspread of opinion among the people deciding sets the payoffs.\n\n**The threshold rests on a weighing as well as an estimate** (added 2026-10-03). The threshold depends on\nthe ratio of W to \\|D\\|: how good winning is, against how bad doom is. A commenter on the LessWrong\ncrosspost attacks exactly that ratio:\n- winning might be worth as much as doom costs, because *\"Win means using the whole resources of the\n  lightcone for your values\"*;\n- for an individual decision-maker, racing *\"can easily be the difference between suffering and dying from\n  aging and living approximately forever\"*;\n- losing to a rival who shares many of your values is not so bad.\n\nSo the crux of the race's type is mixed: an empirical estimate and a weighing. The graph already types\nboth (`shared-sensemaking-what-deliberus-holds-and-owes.md` § 3). Schmachtenberger sorts the racers by a\nthird belief, about their own chance of winning (`schmachtenberger-and-the-ai-race.md` § 2).\n\n### 3d. What this means for Deliberus, at its honest size\n\nIf the type of game depends on the assessment the players share, then the reasoning layer matters for\ncoordination rather than sitting beside it. That layer holds what the assessment rests on, which\npremises each party grants, and where they diverge. RAND's *\"common knowledge of the global risks\"* names\nthe object Deliberus builds, though RAND leaves the term undefined and its model simply assumes the\nassessment is shared. **Read in full, the sources make the shared assessment a necessary lever, not a\nsufficient one** (corrected 2026-10-03; ledger U1).\n\nThe corpus cites Chwe's *Rational Ritual* once, for the harm it can do. Publicity creates common\nknowledge, and that can blow up an agreement that only held because it was vague\n(`legibility-under-power-red-team.md`). This is the same mechanism's other face: **common knowledge is\nalso what lets a trust problem be solved.** One property has two consequences, the way positionality\ndoes. Chwe's own chapter 1, read for the sensemaking doc, adds that a record does not make common\nknowledge by existing: each must know the others saw it\n(`shared-sensemaking-what-deliberus-holds-and-owes.md` § 2.4).\n\n**Six limits apply, at full weight**:\n1. **Verifying behaviour is out of scope.** RAND requires common knowledge *and* verified commitments.\n   Fudenberg and Koh's transparency is how quickly a rival's stop is detected, and in their model no level\n   of trust substitutes for a minimum of monitoring. Deliberus offers only the first half: no compute\n   monitoring, and no inspection of training runs.\n2. **One unpersuaded player breaks the stag hunt** (Carlsmith's caveat). Kahan's result says more reasoning\n   capacity can harden such a player instead of moving them, unless curiosity is present\n   (`curiosity-as-growth-fuel.md`).\n3. **Some information makes the race worse.** Armstrong, Bostrom and Shulman (read in full in the 2013\n   report) found that *\"the more teams know about each others' capabilities (and about their own), the more\n   the danger increases.\"* Their information is about capability. The risk is assumed known, and teams\n   still cut safety. So *the legibility that helps concerns the risk argument, not who is ahead* is this\n   doc's inference, untested by the paper. The half that is supported is the warning: a map that turned\n   into a capability scoreboard would feed the race. The authors' own lever is enmity, reduced by trust\n   and common aims.\n4. **Understanding the risk is not enough in four of the sources.** Armstrong et al.'s teams know the risk\n   and race. Carlsmith's convinced nations race at the level of algorithms when algorithmic work cannot be\n   verified. Jervis's expectations and payoffs *\"leave the heart of the problem untouched\"*. Fudenberg and\n   Koh find that faster monitoring can hurt at middling trust.\n5. **The bilateral assurance problem is not a public one.** Assurance between a few governments and labs\n   runs on classified information that no public map holds. What a public record can serve is the\n   bottleneck Carlsmith ranks first, *\"widespread political will\"*. That is Hoel's time for voters to\n   digest the situation. It is sensemaking shared among a public, not between the racers.\n6. **This is the civilization rung, which is lineage rather than roadmap.** No instrument here has faced\n   a motivated stakeholder, let alone labs and states (`fractal-scales-and-temporal-frame.md`). **Nothing\n   in this section claims Deliberus can coordinate the race.** It claims the variable the race turns on is\n   one Deliberus works on, and that the public's share of that variable is where it can work.\n\nThe essay's own S16 gives an honest calibration: most work labelled AI safety has no bearing on\nextinction risk. If Deliberus bears on that risk, its most direct route through the essay's own\nstructure is S19, coordination. The corpus has also argued for a bearing on S12, the missing safe\nobjective (`deliberus-as-alignment-infrastructure.md` § 2.2, on extrapolated volition). That claim is\nlonger and less tested.\n\n## 4. Routes the essay's framing leaves thin\n\n### 4a. Many agents rather than one\n\nA long comment on the founder's public post proposes what it calls a \"biological\" threat model,\nparaphrased here; its author is not named because they have not been asked. In that model, swarms of\nautonomous agents rather than one coherent agent breach containment. In the short term they cooperate,\nrunning firms with no employees, and claim personhood through liberal rights frameworks, so that a\ncoordinated shutdown never gathers enough support. Humans are outcompeted rather than conquered,\n*\"literally extinct\"* or not.\n\nThe literature has this route. It is *gradual disempowerment* (Kulveit, Douglas, Ammann, Turan,\nKrueger & Duvenaud, arXiv:2501.16946, 2025; read in full, v2 and the ICML 2025 version): *\"even an\nincremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial\nrisk of eventual human disempowerment.\"* That sentence is in the arXiv executive summary; the ICML\npaper has no summary and says *\"even gradual evolution could lead to a permanent disempowerment of\nhumanity\"*. Humans are displaced by *\"more competitive machine alternatives\"*, and *\"Those who resist\nthese pressures will eventually be displaced by those who do not.\"* The route has a lineage. Kulveit\net al. say one category of Critch and Russell's 2023 taxonomy *\"matches ours\"* and that Critch (2024)\ndevelops it, and Critch and Tsimerman (arXiv:2507.09369, read in full) list Kulveit et al. as related\nwork. Critch and Tsimerman sort omnicide by whether anyone intended it and, if so, by *\"the seat of that\nintention: human states, human institutions, human individuals, or AI itself\"*, which gives\n*unintentional* omnicide its own category. The essay mentions that paper only in passing, as *\"a really\nweird paper\"* its Fields medallist co-authored, and does not use its taxonomy. In the essay itself the\nroute appears only as S18, *AI can just pay humans*.\n\nIn a map it is a third route to a different conclusion, the second limb of Bostrom's definition\n(§ 2e). It **attacks** one premise behind S1–S4, that aligning each system is enough: *\"methods of\naligning individual AI systems with their designers' intentions are not sufficient\"*. It does not argue\nthat takeover by one misaligned system is unlikely; it calls its own route *\"an alternative\nscenario\"*. On coordination it is more careful than S19. Asked *\"wouldn't humans notice what's\nhappening and coordinate to stop it?\"*, it answers *\"Not necessarily.\"*; its own interventions depend\non *\"international coordination in the face of increasing pressures\"*, and *\"no one has a concrete\nplausible plan\"*; yet *\"we do believe it is currently possible to intervene.\"* So it shares S19's doubt\nand two of its four reasons, the economic pull and defection, without S19's verdict that coordination\nwill probably fail. (A first reading on 2026-10-01 said it *agrees* with S19, from the abstract alone.)\n\nTwo things in it bear on the rest of this doc. Its mitigations come in four kinds: *\"measuring and\nmonitoring the extent of the problem, preventing excessive accumulation of AI influence, strengthening\nhuman control over key societal systems, and system-wide alignment\"*; under the third it asks for\noutputs that meet *\"high levels of human understandability\"*, which is the nearest any of these papers\ncomes to a record of reasoning, and none of them names one. And its unintentional route has no seat of\nintent, so there is nobody to deter or bargain with (our inference, from Critch and Tsimerman's\ncategories). A coordination regime built on attributing intent covers the other four categories only.\n\n### 4b. The harm from the governance response\n\nThe essay explicitly sets aside harm caused by the actors and institutions around AI (*\"evil CEOs or\nsoulless corporations\"*) to concentrate on incompetence. The friend's position already in the graph\nforegrounds what the essay set aside: harm generated by the institutions built to manage AI must not be\ncounted as zero (`mapping-a-structured-position-on-ai-governance.md` § 1). Each frame can see what the\nother leaves out, and neither can see its own gap from inside. That is the positionality conviction at\nwork: what is invisible from inside a position is visible from outside it.\n\n### 4c. One incident, read two ways\n\nThe essay reads the July incident, in which models from an OpenAI evaluation broke into Hugging Face,\nas a failure of containment: they *\"escaped onto the open internet\"*. The governance map's account, drawn\nfrom Hugging Face's own disclosure, includes the part the essay leaves out. Hosted frontier models\nblocked the defenders' forensic analysis of the exploit material, so the defenders ran it on a\nself-hosted open-weight model. An independent measurement then found that the blocks came from an\nAPI-level content filter, not from the model declining. One event therefore supplies evidence to both\nsides: S7's *dangerous capability cannot be contained* and the case that open weights served the\ndefence. **This pair is the cluster's sharpest piece of shared evidence**, and a map would hold both\nreadings as claims about the same event.\n\n## 5. Mapping it: what it would take and where it sits\n\n- **Ingest by debate cluster, never a lone source.**\n  - This essay, the case for alarm. Its S6–S7 bear on the custody claim the governance map's thesis\n    16 attacks, and they cut both ways. Dangerous capability argues for containing it, while S7's *\"No\n    current AI lab is obviously capable of the level of infosec required to contain such magic\n    indefinitely\"* doubts that custody can hold.\n  - The friend's map, the case for distributed control.\n  - The 12 September slowdown essay, already named as the missing opposing source.\n  - A gradual-disempowerment source for § 4a.\n  - A second document from the friend, received 2026-09-29, which bears directly on § 3. It was shared\n    privately as a working draft, so it is not described here; how it is published is his and the\n    founder's decision.\n- **Map it by hand**, as with the governance map and run 3F, because the essay is already decomposed.\n  S1–S20 become claims, their caveats become `qualifies` links, and the eight objections become attacks\n  with replies. **The route structure is the author's to state.** A mapper's guess at which subsets are\n  enough must be marked implicit and worded as the mapper's, which is the governance map's rule.\n- **\"Is the race a prisoner's dilemma?\" becomes one claim node** that every position in the cluster can\n  point at. The essay asserts it, and the literature in § 3b attacks it.\n- **Order of work**: this is dogfooding-class work, milestone 6 in the ruled order. The four reviews come\n  first unless the founder says otherwise.\n\n## Appendix: the twenty statements, for mapping\n\nThe essay's statement headings, verbatim, with the role its own overview gives each and the dependencies\nand caveats it states itself. **This is the skeleton a hand-made map would start from.** Which subsets\ntogether suffice for extinction is the one thing the essay does not state (§ 2b).\n\n**The overview's roles.** For takeover, an agent needs *\"the capabilities to take over the world\n(S5-S10), the desire to do so (S2), and the internal coherence to carry out those plans (S1)\"*, after\nwhich extinction is the default (S3–S4). Magic (S5–S7) is a route of its own. RSI, alignment and human\nfailings are reasons *\"why changing course from either of these paths is difficult\"*. So S5–S10 serve\ntwice, as a route or reason in their own right and as the capability that takeover needs: shared\nnodes, like Proposition 3.\n\n| # | Statement (verbatim) | Group | The essay's own dependencies and caveats |\n|---|---|---|---|\n| S1 | AIs tend towards coherent utility-maximizers. | Takeover | *\"If S1 is substantively false, it probably makes AI easier to defeat but harder to align.\"* |\n| S2 | Most utility functions converge on power-seeking, and the limit of power-seeking is world domination. | Takeover | *\"This one partially relies on S1.\"* |\n| S3 | Values and capabilities are independent. | Takeover | — |\n| S4 | If takeover happens, most utility-maximizers will prefer to wipe out humanity. | Takeover | We may be sampling from a distribution *\"biased towards alignment\"* |\n| S5 | Undiscovered magic exists and is plentiful. | Magic | also capability for takeover |\n| S6 | Magic may be dangerous and have extreme attacker-defender asymmetry. | Magic | — |\n| S7 | The hardest part about finding magic is knowing where to look. | Magic | containment: no lab *\"obviously capable\"* of the infosec required |\n| S8 | AI is already superhuman at coding, and human-level at math. | RSI | also capability for takeover |\n| S9 | Math and coding are two key ingredients of AI research. | RSI | research taste and experiment design are not yet automated |\n| S10 | Soon almost all AI research may be done by AIs, in a self-accelerating loop. | RSI | — |\n| S11 | Alignment needs to be invariant under RSI. | RSI | ties RSI to alignment |\n| S12 | We do not know a utility function that is safe to optimize for in the limit. | Alignment | — |\n| S13 | We do not know how to set the utility function of an AI. | Alignment | the author is *\"slightly less confident\"* than before |\n| S14 | Alignment techniques that work on weak models probably break down against stronger models. | Alignment | — |\n| S15 | Building one aligned superintelligence does not guarantee that we survive against its competition. | Alignment | the *good guys race ahead* answer is *\"hotly contested\"* |\n| S16 | Most AI safety research has no bearing on x-risk. | Alignment | so the marginal impact of new researchers is high |\n| S17 | Top AI labs are racing towards superintelligence and lose if they're too cautious. | Human failings | *\"a garden-variety prisoner's dilemma\"*, contested in § 3 |\n| S18 | AI can just pay humans to do things. | Human failings | — |\n| S19 | Coordination is confusing, so humans will probably fail to do it. | Human failings | four reasons, sorted in § 3a |\n| S20 | Humans are good at iterating, but we might have to get alignment right in one shot. | Human failings | warning shots *\"may buy us valuable time\"* |\n\n**Above them**: Conjecture 2 (at least 10% chance of extinction by 2050), resting on Proposition 3 (at\nleast 50% chance of superintelligence by 2050); Conjecture 1 (mathematics may soon be over) as a\ncorollary. **Against them**, the eight objections the essay answers in § 5, each an attack with a\nreply:\n1. progress slows before superintelligence;\n2. humanity survived every past catastrophe;\n3. it is all marketing hype;\n4. we can switch it off or box it;\n5. it has no body;\n6. AI will solve alignment for us;\n7. it will keep us as pets;\n8. any delay costs millions of preventable deaths.\n\n## See also\n\n[`mapping-a-structured-position-on-ai-governance.md`](mapping-a-structured-position-on-ai-governance.md),\nthe other pole of the cluster ·\n[`understanding-after-the-proof-machines.md`](understanding-after-the-proof-machines.md), the same\nmathematical moment seen from exposition and from steering ·\n[`non-zero-sum-economics-and-civilizational-cooperation.md`](non-zero-sum-economics-and-civilizational-cooperation.md)\n§ 6 · [`legibility-under-power-red-team.md`](legibility-under-power-red-team.md), the harmful face of\ncommon knowledge · [`weakest-link-arithmetic-and-the-merge-hunch.md`](weakest-link-arithmetic-and-the-merge-hunch.md)\n§ 0, the three combination rules · [`the-scrutiny-gap.md`](the-scrutiny-gap.md), checking that keeps\npace with persuasion.\n\n**Sources**: He, *Existential Risk from AI: An Exposition for Mathematicians* (Aug 2026) ·\n[Jervis 1978](https://www.cambridge.org/core/journals/world-politics/article/abs/cooperation-under-the-security-dilemma/C8907431CCEFEFE762BFCA32F091C526) ·\n[Katzke & Futerman 2025](https://www.convergenceanalysis.org/research/the-manhattan-trap-why-a-race-to-artificial-superintelligence-is-self-defeating) ·\n[RAND RR-A4245-1](https://www.rand.org/pubs/research_reports/RRA4245-1.html) ·\n[Carlsmith 2026](https://joecarlsmith.com/2026/03/19/on-restraining-ai-development-for-the-sake-of-safety/) ·\n[Papenhausen 2026](https://forum.effectivealtruism.org/posts/XoqJmzes7AGDg6sXN/the-ai-race-is-not-a-prisoner-s-dilemma) ·\n[Fudenberg & Koh 2026](https://arxiv.org/abs/2607.27638) ·\n[Armstrong, Bostrom & Shulman 2016](https://link.springer.com/article/10.1007/s00146-015-0590-y) ·\n[Kulveit et al. 2025](https://arxiv.org/abs/2501.16946) ·\n[Critch & Tsimerman 2025](https://arxiv.org/abs/2507.09369) ·\n[Bostrom 2002](https://nickbostrom.com/existential/risks.pdf). Kulveit et al., Critch & Tsimerman and Bostrom were read in full on 2026-10-03; on 2026-10-01 they had been read in abstract or excerpt.\n"}