{"path":"research/testable-bets/prediction-markets-resolution-practice.md","content":"# How real platforms turn a vague belief into a resolvable question, and what goes wrong\n\nResearch note for Deliberus. Compiled 2026-10-19/20. Complements, and does not repeat, `docs/research/prediction-markets-argumentation.md` (market mechanisms, LMSR, futarchy, Brier scores, Metaculus conditional pairs and question groups, argument markets).\n\nScope: the practice of question and rule writing at Polymarket, Kalshi, Metaculus, Manifold, Good Judgment and the Forecasting Research Institute (FRI), the documented disputes, and what is known about AI-written questions and AI resolution. The founder's ruling that this note is read against: an author's stance hedges (\"may\", \"probably\", \"I'd guess\") move out of a claim's wording into a stance field; scope words (\"tend to\", \"most\", \"some\"), stated probabilities (\"a 70% chance\") and possibility-as-content (\"this policy could cut emissions\") stay in the wording.\n\nSource tiers used below. Primary: platform documentation and rule text (Polymarket's public market API, Kalshi contract PDFs and CFTC filings, Metaculus FAQ and guidelines, UMA's blog and docs), court filings, FRI and academic papers. Reputable press: Wired, The Verge, MarketWatch, NPR, Bloomberg Law, CoinDesk, Cointelegraph, The Block, Decrypt. Weaker, flagged where used: community guides, advocacy archives, vendor blogs, single-author analyses. A list of what could not be verified is in the last section.\n\n---\n\n## Summary (plain)\n\n**What transfers to turning extracted claims into testable bets**\n\n1. **A resolvable question is a bundle, and a sentence rarely carries the whole bundle.** Every platform converges on the same parts: a proposition in observable terms, a window (dates, time zone, \"before\" versus \"by\"), a named resolution source, a statement of which kind of event counts (occurred, announced, disclosed, signed, enacted), and a fallback for when the source fails or reality is unclear. Nearly every famous dispute below is one of these parts missing or contradicted: Strategy (occurrence versus disclosure), Ukraine minerals (announcement versus signing), Zelensky's suit (undefined noun), Khamenei (death carve-out), World Cup attendance (source error), Iran (permanent versus interim).\n2. **The hedge/credence split you ruled on is exactly what the forecasting world does.** Platforms do not put \"probably\" in the question text (I found no counter-example); the forecaster's credence goes in a probability field. Tetlock's group calls verbal hedges the thing that blocks accountability, and the translation of hedge words into numbers is lossy (the same \"may/could/might\" ranges from about 0.08 to 0.59 across readers). So a stance field is right, but the mapping from words to numbers should be stored as a band, with the original word kept and the mapping marked as inferred.\n3. **Operationalization is a separate, interpretive act, and the best practice is to keep it separate from the claim.** FRI keeps a \"question prompt\" (the expert's idea) apart from the \"operationalized question\" (the resolvable version), writes several candidate operationalizations, lists false-positive and false-negative scenarios for each, and sends the result back to the person whose idea it was.\n4. **Refusal outcomes are standard.** Metaculus splits them into Ambiguous (reality unclear) and Annulled (reality clear, question flawed); UMA has \"unknown, pays 50/50\" and \"too early\"; Manifold has N/A; Kalshi sets a \"fair\" payout; Good Judgment voids. Reported rates of annulled or ambiguous questions run from about 4% (LLM-generated, small sample) through about 8% (Metaculus, secondhand) to 18% (a small long-horizon cyber tournament); UMA reports about 2% of assertions disputed. A strength computation needs a \"no evidence\" state.\n5. **Soft words survive in titles but are pinned down in rules** by thresholds, named indices, enumerated categories with examples and counter-examples, label triggers (\"WHO calls it a pandemic\"), and filters on kinds of statement (\"official announcements only; no plans contingent on future conditions\").\n6. **Automated generation works only with templates plus verification gates.** FutureSearch reports about 4% annulment for LLM-generated questions; Kalshi lists from 500+ reviewed templates with a two-person check; ForecastBench generates dataset questions from fixed templates. Text-only checks do not predict which rules will be disputed.\n\n**What does not transfer**\n\n- **Stakes and adversaries.** The worst 2025-2026 failures involve money, token-weighted voters and post-hoc rule edits. Deliberus has no payouts, so manipulation matters less, but the extractor then plays the part of the question writer and the same wording failures apply.\n- **Forecasts about the future.** Platforms resolve events by a date. Many extracted claims are about the present, the past, mechanisms or values. Those are verification tasks or not bets at all.\n- **Whose meaning counts.** Platform question writers choose their own wording. Deliberus must keep a stranger's meaning (FRI's closest analogue is the expert interview, and it keeps a feedback loop to the interviewee).\n- **Binary framing.** Almost all platform machinery is Yes/No; much of what authors assert is graded or comparative.\n\n---\n\n## 1. Polymarket\n\n### 1.1 Anatomy of a market\n\nEach market has a **title**, **rules text**, a **resolution source** named inside the rules, an **end date**, and edge-case language. Polymarket's own documentation says every market has pre-defined rules covering resolution source, end date and edge cases, and warns: \"The market title describes the question, but the rules define how it resolves.\" ([Polymarket docs, Resolution](https://docs.polymarket.com/concepts/resolution))\n\n- **Rules text is retrievable as data.** The rendered market page shows the rules only after JavaScript runs, but the public API returns the same text: `https://gamma-api.polymarket.com/events?slug=<slug>` (used throughout this note, fetched 2026-10-19). Whether the API field includes later on-chain clarifications is not confirmed (see section 11).\n- **Clarifications.** After trading starts Polymarket may publish an \"Additional context\" note. Per its docs, clarifications cannot change the question's fundamental intent, are published on-chain via a bulletin-board contract, and \"should be considered\" by UMA voters. Requests go to a Discord channel. ([Polymarket docs, Resolution](https://docs.polymarket.com/concepts/resolution))\n- **Outcome encoding.** UMA's yes/no query uses p1 (No), p2 (Yes), p3 (unknown, pays 0.5) and, in practice, p4 \"too early\". ([UMA docs, YES_OR_NO_QUERY](https://docs.uma.xyz/verification-guide/yes_or_no); [UMA blog, May 2025](https://blog.uma.xyz/articles/recent-improvements-to-umas-optimistic-oracle))\n- **AI commentary is fenced off from the rules.** Market pages now carry an \"Experimental AI-generated summary\" that, per the disclaimer printed on the page, \"plays no role in how this market resolves\". ([example market page](https://polymarket.com/event/ai-lab-announces-another-millennium-prize-solution-by))\n\n### 1.2 The UMA optimistic oracle\n\n| Step | What happens | Source |\n|---|---|---|\n| Proposal | Anyone proposes an outcome and posts a bond (typically $750). A wrong or early proposal forfeits it. | [Polymarket docs](https://docs.polymarket.com/concepts/resolution) |\n| Challenge | 2-hour window to dispute with a matching bond. A first dispute opens a second proposal round. | same |\n| Escalation | If the second proposal is disputed too, a 24-48 hour debate (Discord) and then a token-weighted vote of UMA holders, about 48 hours. Disputed markets take 4-6 days. | same |\n| Outcomes | Proposer wins, disputer wins, \"too early\", or unknown/50-50 (rare). | [Polymarket help, disputes](https://help.polymarket.com/en/articles/13364551-how-are-markets-disputed) |\n| Finality | Polymarket says it is non-custodial and cannot alter a finalized UMA resolution. | same |\n\nBase rates reported by UMA: about 98% of assertions are accepted without dispute; over 80% of the roughly 2% disputed concern timing (\"proposal too early\") rather than accuracy; about 0.4% of all assertions face a substantive accuracy challenge. In April 2025 UMA raised the vote approval threshold from a simple majority to 65%, put Polymarket clarifications inside the voter app, and set cut-offs for issuing clarifications. UMA's own explanation, in the post's words as I read them, is that contentious disputes often come down to how people interpret the rules. ([UMA blog, 8 May 2025](https://blog.uma.xyz/articles/recent-improvements-to-umas-optimistic-oracle))\n\nScale of trouble in 2026, from press: more than 1,150 disputed markets so far in 2026, already above 2025's total ([Decrypt, 7 Jul 2026](https://decrypt.co/372899/traders-sue-polymarket-over-no-ruling-on-strategy-bitcoin-sale)); a May 2026 Wall Street Journal investigation, as summarized by Galaxy Research, found that in most disputed markets over half the UMA votes came from the ten largest wallets, at least 60% of active voters could be linked to live Polymarket accounts, and in about one dispute in five a voter held a stake in the contract ([Galaxy Research, 4 Jun 2026](https://www.galaxy.com/insights/research/strategy-bitcoin-sale-polymarket-resolution-dispute-may-2026); WSJ original is paywalled: [WSJ](https://www.wsj.com/finance/polymarket-bet-disputes-fb1b8c6a)).\n\n### 1.3 Documented controversies, with the exact rule wording\n\nRule texts below were pulled from Polymarket's API; press and community sources supply the outcomes.\n\n**(a) Ukraine minerals deal, March 2025.** Rules: Yes if the US and Ukraine \"agree to any deal\" between 2 February and 31 March 2025 that explicitly involves Ukrainian rare earths; then \"An announcement of a deal will qualify regardless of if/when the deal is enacted.\"; the resolution source is official information from the two governments. ([rules via API](https://gamma-api.polymarket.com/events?slug=ukraine-agrees-to-give-trump-rare-earth-metals-before-april)) About $7.08M traded ([market page](https://polymarket.com/event/ukraine-agrees-to-give-trump-rare-earth-metals-before-april)). The market resolved Yes around 25 March 2025 although no deal had been signed; the agreement was signed on 30 April ([Cointelegraph, 4 Jun 2026](https://cointelegraph.com/news/polymarket-dispute-strategys-bitcoin-sale-resolves-no)). In a Discord post Polymarket said the market \"resolved against the expectations of our users and our clarification\", but, since it \"wasn't a market failure\", gave no refunds ([Decrypt, 26 Mar 2025](https://decrypt.co/311634/polymarket-allegations-oracle-manipulation); [crypto.news](https://crypto.news/polymarket-suffers-governance-attack-due-to-uma-whale-intervention/)). Reported cause (allegation, from Wu Blockchain): one UMA holder cast about 5 million tokens across three wallets, roughly 25% of votes ([crypto.news](https://crypto.news/polymarket-suffers-governance-attack-due-to-uma-whale-intervention/)); an analyst separately alleged that two addresses control over half of UMA's votes ([Decrypt](https://decrypt.co/311634/polymarket-allegations-oracle-manipulation)). A detailed single-author trader analysis argues the real hinge was the speech-act level: whether a US statement that a deal would be \"signed soon\" counted as \"an announcement of a deal\", and what \"enacted\" meant ([mr.ozi, Substack](https://mrozi.substack.com/p/05m-scam-on-the-mineral-deal-market), weaker source). A sibling market worded \"signed\" rather than \"agree\" existed ([PolymarketGuide, community](https://polymarketguide.gitbook.io/polymarketguide-archive/precedents/polymarket/bad-market-descriptions)).\n*Wording lesson:* \"agree\", \"announce\", \"sign\" and \"enact\" are four different events; the rule picked \"announcement\", and the title said \"agrees\".\n\n**(b) \"Will Zelenskyy wear a suit before July?\", April-July 2025.** Rules: Yes if he is \"photographed or videotaped wearing a suit\" between 22 May and 30 June 2025; images must be authentic; \"The resolution source will be a consensus of credible reporting.\" ([rules via API](https://gamma-api.polymarket.com/events?slug=will-zelenskyy-wear-a-suit-before-july)) \"Suit\" was never defined ([PolymarketGuide](https://polymarketguide.gitbook.io/polymarketguide-archive/precedents/polymarket/bad-market-descriptions); [Brierly](https://www.brierlyresearch.com/dispute/zelenskyy-suit)). After the June 2025 NATO summit, headlines said \"suit\" while article bodies said \"suit jacket\"; two Yes proposals (25 and 30 June) were disputed and the market resolved No on 8 July ([PolymarketGuide, community case study, NATO dispute](https://polymarketguide.gitbook.io/polymarketguide-archive/case-studies/the-suitgate-saga/the-nato-dispute); [same guide, resolution](https://polymarketguide.gitbook.io/polymarketguide/case-studies/the-suitgate-saga/market-resolution-and-response)). Volume is reported as about $237M by Decrypt, about $242M by the community guide, and $160-242M by Brierly (they disagree) ([Decrypt](https://decrypt.co/372899/traders-sue-polymarket-over-no-ruling-on-strategy-bitcoin-sale); [PolymarketGuide](https://polymarketguide.gitbook.io/polymarketguide/case-studies/the-suitgate-saga/market-resolution-and-response)).\n*Wording lesson:* an undefined object noun plus a soft source (\"credible\") moves the whole decision to the arbiter; headline-versus-body disagreement inside one outlet became the deciding evidence.\n\n**(c) Venezuela \"invasion\", January 2026.** Rules for the family of markets: Yes if the US \"commences a military offensive intended to establish control over any portion of Venezuela\", with territory defined by de facto control at a fixed date and \"a consensus of credible sources\" ([rules via API](https://gamma-api.polymarket.com/public-search?q=US%20invade%20Venezuela)). After the 3 January 2026 operation that captured Maduro, Polymarket declined to count it as an invasion: a \"snatch-and-extract\" mission did not meet the intent-to-control predicate, and Trump's remark that the US would \"run\" Venezuela did not on its own change that; more than $10.5M had been wagered ([Yahoo Finance, 7 Jan 2026](https://finance.yahoo.com/news/polymarket-withholds-payouts-venezuela-invasion-130701655.html); [SAN](https://san.com/cc/whats-an-invasion-gambling-sites-definition-costs-gamblers-millions/)). Press accounts differ on whether the contract was declared No or left unsettled at that point.\n*Wording lesson:* the title's everyday verb (\"invade\") and the rule's predicate (intent to establish control) came apart, and the rule won.\n\n**(d) Strategy bitcoin sale, May-July 2026.** The monthly series asks whether the firm \"sells any of its Bitcoin\" by a date, resolution source \"information from MSTR and on-chain data\", with credible reporting as backup (the 2025 sibling's wording: [rules via API](https://gamma-api.polymarket.com/public-search?q=MicroStrategy%20sells%20any%20Bitcoin); the 2026 contract's wording as reported: [CryptoPotato](https://cryptopotato.com/trader-claims-polymarket-scammed-him-for-500k-on-microstrategys-bitcoin-sale-market/)). Strategy sold 32 BTC between 26 and 31 May and disclosed it in an 8-K on 1 June. On 1 June Polymarket added context: \"Confirmation achieved outside of the market's time frame does not qualify.\" ([The Block](https://www.theblock.co/post/403600/polymarket-upholds-no-outcome-strategy-bitcoin-sale-market)) After two dispute rounds, 98.6% of the vote (607 participants, per Cointelegraph) chose No; the June contract in the series resolved Yes ([Cointelegraph, 4 Jun 2026](https://cointelegraph.com/news/polymarket-dispute-strategys-bitcoin-sale-resolves-no); [CoinDesk, 4 Jun 2026](https://www.coindesk.com/markets/2026/06/04/polymarket-says-no-for-may-yes-for-june-after-strategy-s-recent-bitcoin-sale)). Galaxy Research said the original rules read event-based and the later bulletin read confirmation-based, and that traders treated the gap as a retroactive rule change, \"which, in our view ... it was\"; it recommended \"lock criteria at listing\". Galaxy disclosed it held Yes shares, so it is an interested party ([Galaxy](https://www.galaxy.com/insights/research/strategy-bitcoin-sale-polymarket-resolution-dispute-may-2026); [Cointelegraph](https://cointelegraph.com/news/polymarket-dispute-strategys-bitcoin-sale-resolves-no)). Two traders sued in New York on 3 July 2026 alleging, in the complaint's words as reported, that the platform \"does not seek truth; it controls payout\" ([Decrypt](https://decrypt.co/372899/traders-sue-polymarket-over-no-ruling-on-strategy-bitcoin-sale)). Reported volume: over $60M ([Sepia, secondary](https://sepia.software/blog/prediction-market-resolution)) to over $80M ([Cointelegraph](https://cointelegraph.com/news/polymarket-dispute-strategys-bitcoin-sale-resolves-no)). Polymarket now lists similar markets phrased around whether Strategy will \"announce\" a sale ([Gaming Today](https://www.gamingtoday.com/news/lawsuit-against-polymarket-alleges-market-rules-were-changed-after-bets-were-placed/), secondhand).\n*Wording lesson:* \"did the thing happen\" versus \"did we learn that it happened\" is a distinct slot. Fixing it after money was in was worse than either reading.\n\n**(e) US-Iran \"permanent peace deal\", June 2026.** Rules: Yes if the parties agree \"a permanent peace deal\", defined as an agreement that explicitly indicates hostilities \"have ended or will permanently cease\" or \"uses equivalent language clearly signaling a lasting end\"; explicitly temporary agreements and \"statements of progress\" do not count; an agreement is established if both sign or formally adopt a written agreement, or both governments give clear public confirmation that a qualifying agreement has been definitively established ([Polymarket market page](https://polymarket.com/event/us-x-iran-permanent-peace-deal-by/us-x-iran-permanent-peace-deal-by-june-15-2026-734-856-129)). A memorandum of understanding that included a 60-day ceasefire was announced around 14-15 June, with formal signing and further negotiation to follow. The Yes proposal was disputed; over 99% of UMA voting power backed Yes while critics called the deal interim ([The Block, 17 Jun 2026](https://www.theblock.co/post/405117/trump-comments-fuel-dispute-over-polymarkets-120-million-permanent-iran-peace-deal-market); [TNW, 15 Jun 2026](https://thenextweb.com/news/polymarket-345-million-iran-peace-deal-dispute-uma-whale-voting)). Final outcome shown on the page: Yes. Volume figures differ between sources ($120M for the 15 June contract, $345M across the series, $177M \"CLOB volume\" on one tracker).\n*Wording lesson:* a rule that defines the event by what the documents say (\"explicitly indicates\", \"equivalent language\") lets a speech act satisfy a substantive word (\"permanent\"). This is the opposite failure to (c): letter satisfied, spirit not.\n\n**(f) \"Mention\" markets.** Polymarket's rules for what a speaker will say are lexical: any usage regardless of context counts; plurals and possessives count but other forms do not; compound words count regardless of context; Q&A counts; only the named event counts ([Saylor keynote rules via API](https://gamma-api.polymarket.com/public-search?q=What%20will%20Saylor%20say%20during%20his%20Strategy%20World%202025%20keynote)). UMA's own AI proposer is weakest on this category: 72% on mention markets against about 95% overall once \"too early\" outcomes are excluded (see section 9).\n\n**(g) Hidden-truth markets get a default.** \"Fact Check: Maduro capture staged?\" resolves Yes only if \"widely and credibly confirmed\", counts US-government statements that it was staged as qualifying, and says that without broad consensus by the deadline it resolves No ([market page](https://polymarket.com/event/fact-check-maduro-capture-staged)).\n\n### 1.4 Reforms and what they leave open\n\nUMA's 65% threshold, clarification cut-offs and voter-app context are above. A managed oracle variant (UMIP-189, \"MOOV2\") restricts who can propose; sources disagree on whether this took effect in August or November 2025 and on the whitelist's size and role ([Cryptonews](https://cryptonews.net/news/altcoins/32922362/); [Sepia](https://sepia.software/blog/prediction-market-resolution)). In August 2026 Polymarket replaced single-price snapshots in short crypto markets with Chainlink time-weighted averages after researchers found likely settlement manipulation ([CoinDesk, 7 Aug 2026](https://www.coindesk.com/business/2026/08/07/how-a-five-second-trick-let-traders-drain-millions-from-polymarket)): a numeric rule widened from an instant to an interval. Nothing found changes the underlying arrangement that a token-weighted vote settles contested language.\n\n---\n\n## 2. Kalshi\n\n### 2.1 How a regulated contract specifies its terms\n\nKalshi is a CFTC-regulated exchange, and each product has a public contract-terms PDF. The fixed fields are Underlying, Source Agency, Type, Issuance, Payout Criterion, Minimum Tick, Position Accountability, dates, Settlement Value, Expiration Value and Contingencies. ([WLEADEROUT terms](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/WLEADEROUT.pdf)) The rulebook defines the \"Payout Criterion\" as the set of Expiration Values that makes a contract pay, and the \"Market Outcome\" as Yes if the Expiration Value falls inside it, otherwise No ([KalshiEX rulebook](https://www.cftc.gov/filings/orgrules/rules07012525155.pdf)).\n\nFeatures worth copying:\n\n- **Hierarchical source agencies.** A court contract lists \"official records and electronic filing systems\" first and press outlets after, \"in hierarchical order\" ([CASEWIN terms](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/CASEWIN.pdf)). The same contract defines \"superior outcome\" by an exclusive three-step hierarchy (net money, then substantive victory, then case termination), with settlement and mistrial rules ([same](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/CASEWIN.pdf)).\n- **Worked examples in the rule text.** A runoff contract states what \"qualifies\" and \"announced to qualify\" mean, then lists cases that \"would\" and \"would NOT\" resolve Yes ([ADVANCE terms](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/ADVANCE.pdf)).\n- **Statements about the future are classified explicitly.** The leader-exit contract counts an announcement that the person \"will leave the office within the next year\" but not an acknowledgement of leaving solely because a legally mandated term ends ([WLEADEROUT terms](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/WLEADEROUT.pdf)).\n- **A residual standard.** A \"role before date\" contract tells the exchange, if qualification is disputed, to use the standard of \"a reasonable, informed observer\" and may start the outcome review ([ROLEBEFOREDATE terms](https://assets.kalshi.com/regulatory/product-certifications/ROLEBEFOREDATE.pdf)).\n- **Missing data.** Rule 6.3 (amended November 2024) says that when it cannot be determined whether the Payout Criterion covers the Expiration Value the exchange determines a fair payout allocation; Rule 7.2 allows moving expiration when the Underlying cannot be measured ([CFTC filing, Rulebook v1.14](https://www.cftc.gov/filings/orgrules/rules1114248723.pdf)). Rule 7.1's review process is run by a three-member Outcome Review Committee with at least two public directors, with a 24-hour decision window as quoted by one explainer ([rulebook text](https://www.cftc.gov/filings/orgrules/rules1114248723.pdf); [OddsShopper](https://www.oddsshopper.com/articles/prediction-markets/cftc-complaint-process-prediction-markets)).\n- **Templates and review.** Kalshi says it has more than 500 pre-worked market templates; listing normally needs two people, a one-to-two hour delay before going live, and a paid bounty for flaws found ([Bloomberg via predictionmarkets.org, 15 Jun 2026](https://www.predictionmarkets.org/kalshi-builds-ai-agent-to-stress-test-prediction-market-bets/)).\n\n### 2.2 Disputes and one rule change\n\n**Khamenei, February-March 2026.** The market \"Ali Khamenei out as Supreme Leader?\" had contract text saying that if the person leaves solely because they have died, the exchange sets payouts from the last traded price before the death ([MarketWatch, 2 Mar 2026](https://www.marketwatch.com/story/prediction-market-traders-betting-on-irans-khamenei-got-a-messy-reminder-to-always-read-the-fine-print-72a39382)). When Khamenei was killed in strikes on 28 February, Kalshi paused and settled at pre-death prices instead of paying Yes; the CEO said Kalshi does not list markets \"directly tied to death\" ([The Verge](https://www.theverge.com/tech/887210/kalshi-void-bets-khamenei-death)). A spokesperson said: \"Our rules were clear from the beginning.\" Wired reports that the carve-out notice had only been added to the market page after the attack began, so some traders never saw it ([Wired, 2 Mar 2026](https://www.wired.com/story/iranian-ayatollah-khameneis-death-sparks-revolt-among-kalshi-customers/)). Polymarket's equivalent market resolved Yes ([MarketWatch](https://www.marketwatch.com/story/prediction-market-traders-betting-on-irans-khamenei-got-a-messy-reminder-to-always-read-the-fine-print-72a39382)). Kalshi reimbursed fees and net losses, reportedly $2.2M on roughly $55M wagered (figures vary: a Washington Post headline says $54M) ([Business Insider](https://www.businessinsider.com/kalshi-iran-prediction-market-death-rule-fine-print-2026-3); [Washington Post](https://www.washingtonpost.com/technology/2026/03/04/kalshi-khamenei-bet-controversy/)). A class action in New York (Golan v. KalshiEX) says a plaintiff's \"before April 1\" contracts settled at $0.29 each, while MarketWatch cites 39.5% for the same market family ([complaint](https://www.pacermonitor.com/public/filings/DQN623BA/Golan_v_KalshiEX_LLC_et_al__nysdce-26-02394__0001.0.pdf); [MarketWatch](https://www.marketwatch.com/story/prediction-market-traders-betting-on-irans-khamenei-got-a-messy-reminder-to-always-read-the-fine-print-72a39382)). Another suit was filed in California federal court ([Bloomberg Law, 6 Mar 2026](https://news.bloomberglaw.com/securities-law/kalshi-sued-over-death-carveout-in-iran-leader-prediction-market)). Kalshi then filed a rule that standardizes the treatment for all markets that implicitly depend on a person surviving, effective 17 March 2026 ([Business Insider](https://www.businessinsider.com/kalshi-iran-prediction-market-death-rule-fine-print-2026-3)).\n*Wording lesson:* \"leaves office\" and \"dies\" overlap in the world but not in the contract; the title was a superset of the payout criterion. The remedy was a standard clause applied to the whole product class, which is the template approach.\n\n**World Cup final attendance, 2026.** A Fox sportscaster mistook Matt Damon for Brad Pitt; several outlets that Kalshi lists as source agencies reported Pitt as attending; the market settled Yes strictly under its contract rules even though Pitt was not there, and holders of No lost about $288k ([NPR, 13 Aug 2026](https://www.npr.org/2026/08/13/nx-s1-5930689/cftc-probe-mention-markets-prediction-markets-kalshi)). The same article reports that the CFTC has opened a review of \"mention markets\" and Kalshi pulled sports mention markets.\n*Wording lesson:* naming the source makes the market a bet on what the source says, not on the world; a wrong source then wins.\n\n**\"Warner Bros.\" versus \"Warner Brothers\", January 2026.** Kalshi resolved a market on whether a Netflix executive would say \"Warner Bros.\" as No because the person said \"Warner Brothers\" ([Bloomberg via predictionmarkets.org](https://www.predictionmarkets.org/kalshi-builds-ai-agent-to-stress-test-prediction-market-bets/)). A community compilation of Kalshi mention-market disputes (plurals, homographs such as \"Quanta\" for \"quantum\", foreign-language words) shows the same pattern and a Kalshi staff remark that precedent does not bind ([Bernard Bulletin, community, Dec 2025](https://bernardbulletin.substack.com/p/kalshi-mention-market-dispute-bible)).\n\n---\n\n## 3. Metaculus\n\n### 3.1 Question anatomy and writing rules\n\nA Metaculus question has separate fields for background, the question, resolution criteria and fine print, introduced to \"standardize and improve question formats\" ([platform update thread, undated entry](https://www.metaculus.com/notebooks/3430/platform-update-thread/)). Its guidelines tell writers to aim for tight criteria that limit the scope for ex-post quarrels about what happened and who was right, define terms, be concrete, use authoritative sources with fallbacks, consider edge cases and \"unknown unknowns\", and avoid questions needing more than about 15 minutes of admin effort. Only admins resolve; authors may not hold resolution authority. Two further rules matter here: \"The title and wording of the question should match the resolution conditions\", and pass Tetlock's clairvoyance test (if a clairvoyant could see the future, could they answer without asking what you meant?). ([Metaculus question-writing guidelines](https://www.metaculus.com/question-writing/)) The FAQ adds that good questions concern \"tangible, objective facts about the world\" rather than opinion, and that Metaculus questions are not polls ([Metaculus FAQ](https://www.metaculus.com/faq/)). The FAQ also defines \"credible source\" and \"before [date]\" (UTC, first moment of that date) in one place so questions can reuse them ([same](https://www.metaculus.com/faq/)).\n\n### 3.2 Ambiguous versus Annulled, and how often\n\n- **Ambiguous:** the truth of the matter is too uncertain. **Annulled:** reality is clear but a key assumption of the question was overturned or the question failed to specify resolution. Both stop forecasting and are unscored. Annulment became a separate option in April 2023. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Named reasons:** no clear consensus; imbalanced outcomes (no clear mechanism for one of Yes or No, which creates bad incentives and forces annulment even if a qualifying event occurs); subverted assumptions; underspecified questions. Examples: \"ignition\" for a fusion reactor was undefined; a question on \"general mobilization\" used definitions that did not fit Russia's partial mobilization; a question tied to a US report that was never going to be published was annulled; a question asking whether any FTX depositor would withdraw assets had no feasible route to a No. ([Metaculus FAQ](https://www.metaculus.com/faq/); [FAQ mirror with the imbalance section](https://www.metaculus.com/help/faq/))\n- **Fairness principle:** resolve by \"the actual written text\" of the criteria whenever possible, because criteria work like a contract and two diligent readers should not diverge; if the outcome does not clearly fit the text, ambiguity or annulment preserves fair scoring. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Clarifying mid-flight:** admins weigh fairness to forecasters; clarifications are most appropriate for new questions, internally inconsistent criteria, or where they add specificity without changing meaning. Claims of unclear criteria raised only after closing or resolving face \"a higher standard of evidence\". Questions can be re-resolved if later found in error. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Source tracks truth or the source?** The FAQ says some questions track what an organization or person says, while others treat the source as an approximation of truth, and intent is often unclear. Default if a source is defunct or in error: replace it with a functional equivalent or annul, unless the question says otherwise (for instance by stating that it tracks the source's publications whatever errors they contain). If a source changes methodology, minor changes are accepted, substantial ones revert to the old methodology or an equivalent source, else annul. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Rates (secondhand):** FutureSearch states Metaculus annuls about 8% of questions ([FutureSearch](https://futuresearch.ai/automating-forecasting-questions/)); Lawfare's report on its White Hat cyber tournament says about 8% of Metaculus's other questions resolved ambiguously against 18% (6 of 43) in the tournament, where several data series the questions relied on were discontinued ([Lawfare, 2023](https://www.lawfaremedia.org/article/findings-from-the-white-hat-cyber-forecasting-tournament)). Metaculus's own figure was not found.\n- **A letter-versus-intent case:** a COVID question asked for the figure on a spreadsheet tab \"on April 26th at 16:00 ET\"; the intent was the April 26 row, the literal reading was the latest row (April 25). Metaculus's standard is the literal reading, and the question resolved Ambiguous after the community had read the criteria two ways ([Metaculus notebook, 2020](https://www.metaculus.com/notebooks/22691/a-preliminary-look-at-metaculus-and-expert-forecasts/)).\n\n### 3.3 Long horizons, groups, conditionals\n\n- **Conditional pairs** ask P(child | parent Yes) and P(child | parent No); the branch whose parent did not occur is Annulled and not scored. **Question groups** bundle independent sub-questions. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Resolution Councils** are named individuals or groups who resolve important questions at their own authority, reserved for questions with potential to be in the top 1% by impact. ([Metaculus FAQ](https://www.metaculus.com/faq/))\n- **Tournaments and Question Series** organize questions around a theme; Series have no prize pool and award no medals. ([Metaculus FAQ](https://www.metaculus.com/faq/)) I did not find Metaculus guidance on proxy \"indicator\" questions for long horizons; the nearest material is FRI's short-run indicators (section 5.2).\n- **Long-term sets:** a Metaculus tournament proposal separates a Calibration set (resolves within the tournament) from a Long-Term set that is \"out-of-sample\" for prizes yet still scored on the platform, plus \"fortified essays\", defined there as \"an opinion with testable predictions fortifying its claims\", judged by experts ([Metaculus notebook, undated](https://www.metaculus.com/notebooks/22485/aligning-incentives-for-forecast-accuracy-relevance-and-efficacy-a-new-paradigm-for-metaculus-tournaments/)).\n\n---\n\n## 4. Manifold Markets\n\n- **Creator resolution is the default.** The FAQ says the creator is \"free to use their judgment\", encourages clear criteria in advance, and says \"In exceptional circumstances, our team of moderators will overturn resolutions.\" Disputes are reported to moderators; resolutions that follow the rules stand. ([Manifold FAQ](https://docs.manifold.markets/faq))\n- **N/A is not the creator's tool.** Resolving N/A cancels the market and returns mana; it is limited to moderators or staff, and creators should not count on it. Partial resolution to a probability is allowed but discouraged without justification. ([Manifold FAQ](https://docs.manifold.markets/faq))\n- **Guidance for writing criteria:** define what counts as Yes, No and N/A before betting; name the source; cover cancelled events and unavailable sources; use \"before [date]\" rather than \"by\"; if criteria must change, edit the description and post a comment. ([Manifold creator guide](https://manifold.markets/community-guidelines/creator-guide))\n- **Trust problem in the platform's own words.** A staff poll lists misresolution (rare), authors who interpret a question very differently from the consensus, abandoned markets, and inconsistent approaches as downsides, and proposes a dispute-and-override layer ([Manifold poll](https://manifold.markets/jack/poll-should-it-be-possible-to-dispu)). An older FAQ version said individual resolution \"is not enforced by the site\" ([archived FAQ](https://git.nunosempere.com/open.source/manifold/src/commit/c59de1be2e321ac5506015d5b47bbd84fe93c2f6/docs/docs/faq.md)).\n- **Data on disputes is thin.** A moderator's comment under a Manifold market reports that about 3,600 markets have been resolved by someone other than the creator and about 1,800 unlisted by moderators; that is a user-posted figure, not an official statistic ([Manifold market comment](https://manifold.markets/CodeandSolder/will-manifold-moderation-get-signif)). A LessWrong critique argues that vague wording makes traders bet on the resolver's psychology and gives anecdotes but no rates ([LessWrong](https://www.lesswrong.com/posts/DpDnXHcPejd9tn8R5/ambiguity-in-prediction-market-resolution-is-harmful)).\n- **Self-resolving markets** (resolve to the market's own final probability) are described as failing \"spectacularly\" so often that the staff author recommends resolving to a poll, or resolving normally with some probability and N/A otherwise, which makes manipulation unprofitable at the cost of weaker incentives ([Manifold post, Sep 2023](https://manifold.markets/post/selfresolving-markets-why-they-dont)).\n\n*Reading:* Manifold chose flexibility over precision; subjective questions are allowed (\"Will I enjoy ...\") but the price of vagueness is paid in trust. In my reading that is the nearest platform analogue to a graph where a reader, not an oracle, ends up deciding.\n\n---\n\n## 5. Good Judgment and the Forecasting Research Institute\n\n### 5.1 Good Judgment Project and Good Judgment Open\n\n- **The clairvoyance test.** Questions must be so well framed that a genuine clairvoyant could answer without asking for clarification (\"What did you mean by ...?\"); \"Will I fall in love?\" fails, \"Will I marry Jane Smith by this time next year?\" passes. ([Tetlock, Edge Master Class 2015](http://edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-i); [Tetlock 2020 essay](https://www.stat.berkeley.edu/~aldous/Blog/tetlock2020.pdf); [Expert Political Judgment excerpt](https://emilkirkegaard.dk/en/wp-content/uploads/Philip_E._Tetlock_Expert_Political_Judgment_HowBookos.org_.pdf))\n- **The rigor-relevance tradeoff.** Tetlock states there is \"a tradeoff between rigor and relevance\": rigorous questions are small and resolvable, relevant ones (stability of the Eurozone, Putin's intentions) are big and cannot enter tournaments. His fix is **question clusters**: sets of resolvable indicators, each with a small bearing on a large question, so the answers together tip the plausibility of the big claim. ([Edge Master Class, Class II](https://edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii))\n- **Vague verbiage defeats accountability.** In the IARPA tournament the questions had to be \"resolvable\" in a specified window; the authors argue \"Vagueness thus precludes accountability\", citing that readers translate \"may\", \"could\" and \"might\" into probabilities from 0.08 to 0.59. ([Tetlock, Mellers & Scoblic 2017](https://pdf.stafforini.com/tetlock-2017-bringing-probability-judgments.pdf)) Tetlock also notes that only probabilities of 0 and 1 are technically falsifiable by a single outcome; accuracy of other probabilities can only be assessed across many events (calibration and discrimination). ([Expert Political Judgment excerpt](https://emilkirkegaard.dk/en/wp-content/uploads/Philip_E._Tetlock_Expert_Political_Judgment_HowBookos.org_.pdf))\n- **A void in practice.** In a 2016 Good Judgment Open challenge about electric and autonomous vehicles, a question on counting \"partners\" in a Department of Energy programme turned on whether subsidiaries counted; Good Judgment issued a clarification, learned the DoE counted some subsidiaries and not others, and decided to \"void this question by closing it\", with no scores issued. The same report says the design aim was to balance relevance with the rigor needed to resolve unambiguously, and that \"taking orders\" and \"public testing\" caused confusion. ([Wharton/Good Judgment summary report](https://pvmi.wharton.upenn.edu/wp-content/uploads/2018/11/GJ-Challenge-2016-Summary-Report.pdf))\n- **Current Good Judgment Open question-writing rules** were not found in public form; the sources above are older reports and essays.\n\n### 5.2 Forecasting Research Institute\n\n- **Conditional trees** (Aug 2024): experts name indicator events that would move their view of a far-future outcome; these \"question prompts\" are turned into \"fully operationalized forecasting questions\" by a separate writing team, then ranked by Value of Information using superforecasters' conditional forecasts. From 24 interviews they produced 75 questions; questions resolving by 2030 were about nine times more informative than the highest-engagement AI questions on two platforms (p = .025), but the method is labor-intensive and the authors suggest LLMs or crowdsourcing might lower the cost. ([FRI report](https://forecastingresearch.org/research/ai-conditional-trees))\n- **How they operationalize.** The first stated goal is to \"capture as much of a question prompt's original intent as possible\" while keeping the question highly resolvable. Writers draft several distinct operationalizations and compare them on relevance, efficiency, \"salient hypothetical cases of false positive resolution and false negative resolution\", feasibility and forecaster load; the detailed conditions go back to the interviewee for feedback. The report's definition: \"major progress in interpretability by 2030\" is a prompt; the operationalized question must settle in advance whether progress counts as \"major\". ([FRI report](https://forecastingresearch.org/research/ai-conditional-trees))\n- **Wording sensitivity.** In other FRI projects they ran five rounds of revising questions after seeing how experts interpreted them, because forecasts are \"highly sensitive to the precise wording of a question\". ([FRI, LLM-enabled biorisk](https://forecastingresearch.org/research/llm-enabled-biorisk))\n- **Pre-specified criteria as policy.** The longitudinal expert AI panel (LEAP) gives respondents resolution criteria at forecast time, to avoid disagreement about what is predicted and how it will be judged. ([FRI, LEAP](https://forecastingresearch.org/research/longitudinal-expert-ai-panel-leap-working-paper))\n- **Unanswerable questions.** *Reciprocal scoring* asks forecasters to predict other forecasters' forecasts; FRI reports it matched accuracy of forecasters on objectively resolvable outcomes in one study and was used for policy effects on COVID mortality ([FRI](https://forecastingresearch.org/research/reciprocal-scoring-forecasting-unanswerable-questions)). A critique says it resembles a Keynesian beauty contest because forecasters predict the mainstream rather than reality ([Sempere](https://nunosempere.com/blog/2022/04/16/optimal-scoring/)). In the Existential Risk Persuasion Tournament, of 32 resolved questions 47% were definitively resolved from authoritative data and 53% only \"provisionally\", partly through expert panels ([FRI near-term XPT report](https://forecastingresearch.org/research/near-term-xpt-accuracy)).\n\n---\n\n## 6. Hedged and vague language in questions\n\n### 6.1 What hedge words do, according to the research\n\n- The same verbal expression gets very different numbers from different readers; Kent proposed a fixed lexicon in 1964 (for example \"probable\" about 75%, \"chances about even\" 50%, \"probably not\" about 30%) ([Words of estimative probability](https://en.wikipedia.org/wiki/Words_of_estimative_probability)). The US intelligence community's own guidance treats \"might\" as a case where the likelihood cannot be assessed, which is a third meaning beside credence and possibility ([CIA/CSI history](https://www.cia.gov/resources/csi/static/ICs-Struggle-to-Express.pdf)).\n- In a 924-person experiment, putting the numeric range in brackets right after the word raised agreement with the intended meaning to 66%, against 32% with no guidance; tooltips and linked tables did not help ([Wintle et al., PLOS ONE 2019](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0213522)).\n- Semantics research is split between \"threshold\" readings (possible means probability above zero) and prototype readings; empirical studies find gradient, focal ranges (for example \"possible\" near 38.5% with a wide spread) ([PMC review](https://pmc.ncbi.nlm.nih.gov/articles/PMC9987346/)).\n\n### 6.2 What platforms do instead\n\nI found no platform guidance that allows epistemic hedges (\"may\", \"likely\") inside the resolution trigger of a question; in the rule texts I read, \"may\" appears only in clauses about the platform's own discretion (\"Kalshi may ...\", \"the market may remain open\"). That is a sample, not a census. Soft evaluative words do appear, usually inside a named arbiter or a defined standard. The moves, with primary examples:\n\n| Vague word or structure | Platform move | Example |\n|---|---|---|\n| \"major / significant\" | Keep the word, enumerate target classes, add examples and counter-examples | A cyberattack market requires \"significant disruption, damage, or unauthorized access\" affecting \"core national systems\", lists qualifying and non-qualifying incidents ([rules](https://gamma-api.polymarket.com/public-search?q=major%20cyberattack)) |\n| \"large\" | Numeric threshold or named index | Troop withdrawal market: \"1,000 or more troops\" ([rules](https://gamma-api.polymarket.com/public-search?q=Trump%20announces%20large%20troop%20withdrawal%20from%20South%20Korea)); eruptions: VEI of 4 or higher from a named volcano programme ([rules](https://gamma-api.polymarket.com/public-search?q=How%20many%20large%20volcano%20eruptions)) |\n| \"normal\" | Named index, threshold, window, revision and data-failure policy | \"Traffic returns to normal\" means IMF PortWatch's 7-day moving average of ship arrivals at or above 60 ([rules](https://gamma-api.polymarket.com/public-search?q=Strait%20of%20Hormuz%20traffic%20returns%20to%20normal)) |\n| \"best\" | Named leaderboard, check time, tie-break ladder | Arena rank with style control off, then unrounded score, then alphabetical ([rules](https://gamma-api.polymarket.com/public-search?q=Which%20company%20has%20best%20AI%20model%20end%20of)) |\n| Category label (\"pandemic\") | Trigger on a named authority's use of the label | Yes only if WHO explicitly calls it a \"pandemic\" ([rules](https://gamma-api.polymarket.com/public-search?q=Hantavirus%20pandemic%20in%202026)) |\n| \"agree / announce\" | Say which speech act and whose confirmation counts | Iran: both sign or both governments confirm; \"statements of progress\" excluded ([page](https://polymarket.com/event/us-x-iran-permanent-peace-deal-by/us-x-iran-permanent-peace-deal-by-june-15-2026-734-856-129)) |\n| Intent and plans | Exclude contingent statements | \"Only official policy announcements will qualify\": informal statements and \"plans contingent on future conditions\" do not ([rules](https://gamma-api.polymarket.com/public-search?q=Trump%20announces%20large%20troop%20withdrawal%20from%20South%20Korea)) |\n| \"say / mention\" | Lexical rules: plurals, possessives, compounds, context-free, which event | Saylor keynote rules above |\n| \"beat expectations\" | Fix the strike at market creation from a consensus estimate | Earnings markets use a consensus non-GAAP EPS figure as strike ([rules](https://gamma-api.polymarket.com/public-search?q=Humana%20beat%20quarterly%20earnings)) |\n| \"reasonable\" | Standard of a reasonable, informed observer plus a review process | Kalshi role contracts ([terms](https://assets.kalshi.com/regulatory/product-certifications/ROLEBEFOREDATE.pdf)) |\n| \"credible\" | Define it once for the platform | Metaculus FAQ definition ([FAQ](https://www.metaculus.com/faq/)); Polymarket uses \"consensus of credible reporting\" without a definition in the rules and relies on precedent |\n\nAn independent audit site (methodology not verifiable, see section 11) claims that a taxonomy of title-versus-rule gaps (narrower scope, stricter threshold, hidden source, hidden timing, different entity) is common: 64 of 500 high-volume Polymarket and Kalshi markets flagged on 19 October 2026 ([Asterisk](https://asteriskmarkets.com/)). Treat the numbers as indicative only; the taxonomy itself maps cleanly onto the cases above.\n\n---\n\n## 7. Claims that cannot be resolved as stated\n\n| Kind of claim | What platforms do | Source |\n|---|---|---|\n| **Value or opinion** | Metaculus refuses (\"not a matter of opinion\"); Manifold allows creator-judged markets and carries the trust cost; Polymarket turns \"best\" into a named leaderboard | [Metaculus FAQ](https://www.metaculus.com/faq/); [Manifold FAQ](https://docs.manifold.markets/faq); leaderboard rules in section 6 |\n| **Hidden or never-settled truth** | Define a consensus requirement and a default: \"no broad consensus by deadline resolves No\"; or Metaculus \"Ambiguous\" | [Maduro fact-check](https://polymarket.com/event/fact-check-maduro-capture-staged); [Metaculus FAQ](https://www.metaculus.com/faq/) |\n| **Counterfactual and conditional** | Metaculus conditional pairs; the branch whose condition fails is Annulled | [Metaculus FAQ](https://www.metaculus.com/faq/) |\n| **Causal (\"X causes Y\")** | Not asked directly; FRI uses reciprocal scoring and conditional forecasts instead; Tetlock's route is indicator clusters | [FRI](https://forecastingresearch.org/research/reciprocal-scoring-forecasting-unanswerable-questions); [FRI conditional trees](https://forecastingresearch.org/research/ai-conditional-trees); [Edge](https://edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii) |\n| **Very long horizon** | Short-run proxy indicators; \"Long-Term\" sets scored but outside prize money; contracts allowed to expire into a fixed maximum (Kalshi's runoff contract: two years); expert panels for provisional resolution | [FRI XPT](https://forecastingresearch.org/pdf/existential-risk-persuasion-tournament.pdf); [Metaculus notebook](https://www.metaculus.com/notebooks/22485/aligning-incentives-for-forecast-accuracy-relevance-and-efficacy-a-new-paradigm-for-metaculus-tournaments/); [ADVANCE terms](https://kalshi-public-docs.s3.amazonaws.com/contract_terms/ADVANCE.pdf); [FRI near-term](https://forecastingresearch.org/research/near-term-xpt-accuracy) |\n| **Subjective numeric quantity** | Resolve to a named council's estimate; or probabilistic N/A | [Resolution-council spec](https://github.com/NunoSempere/clarivoyance/blob/master/list/resolution-council.md) (proposal, not a platform) |\n| **Existence (\"any depositor withdrew\")** | Needs a bounded search domain or source; otherwise No cannot be established and the question is annulled | Metaculus FTX example in [FAQ](https://www.metaculus.com/help/faq/) |\n| **Anything the record cannot settle** | Metaculus Ambiguous/Annulled; UMA p3 50/50; Manifold N/A with refunds; Kalshi exchange-set payout, often last price; Good Judgment void | sections 1-5 |\n| **Self-reference (\"the market itself\")** | Disfavoured; use a poll or random-chance resolution | [Manifold post](https://manifold.markets/post/selfresolving-markets-why-they-dont) |\n\nThe Metaculus \"imbalanced outcomes\" rule deserves emphasis for claims of the form \"some X does Y\": if No can only be shown by exhaustive research, the question has no clean No path, and savvy forecasters learn to shade toward Yes, so Metaculus annuls even when a qualifying event occurs. ([Metaculus FAQ](https://www.metaculus.com/help/faq/))\n\n---\n\n## 8. Precision versus relevance: when the letter beat the spirit\n\nTetlock puts the underlying tradeoff as rigor against relevance (section 5.1). A Rethink Priorities survey of specification problems separates **ambiguous** (spec too loose), **misaligned** (letter diverges from spirit, including Goodhart effects when people change the world to hit the criterion) and **uncompelling** specifications, and lists mitigations: tight definitions and fallbacks, a trusted arbiter who applies the spirit, several well-specified proxies, reserving the right to modify the spec, and handling flawed questions after launch by voiding, keeping the old one beside a new one, or closing early and scoring the original ([Rethink Priorities, Gil 2021](https://rethinkpriorities.org/research-area/types-of-specification-problems-in-forecasting/)).\n\n### 8.1 Cases where a precise rule resolved against the evident spirit\n\n| Case | Letter | Spirit people expected |\n|---|---|---|\n| Strategy sale (2026) | Event-based rule, then a note requiring confirmation in-window | The sale happened in May ([Galaxy](https://www.galaxy.com/insights/research/strategy-bitcoin-sale-polymarket-resolution-dispute-may-2026)) |\n| Ukraine minerals (2025) | \"An announcement ... qualifies\" | A deal being concluded ([crypto.news](https://crypto.news/polymarket-suffers-governance-attack-due-to-uma-whale-intervention/)) |\n| World Cup attendance (2026) | Source agencies' reports | Whether Pitt attended ([NPR](https://www.npr.org/2026/08/13/nx-s1-5930689/cftc-probe-mention-markets-prediction-markets-kalshi)) |\n| \"Warner Bros.\" (2026) | Exact string | Mention of the company ([Bloomberg via predictionmarkets.org](https://www.predictionmarkets.org/kalshi-builds-ai-agent-to-stress-test-prediction-market-bets/)) |\n| Khamenei (Kalshi, 2026) | Death carve-out | Leaves power ([The Verge](https://www.theverge.com/tech/887210/kalshi-void-bets-khamenei-death)) |\n| Venezuela (2026) | Intent to establish control | An invasion in the everyday sense ([Yahoo Finance](https://finance.yahoo.com/news/polymarket-withholds-payouts-venezuela-invasion-130701655.html)) |\n| Iran permanent deal (2026) | \"Explicitly indicates\", \"equivalent language\" | A durable settlement (opposite direction: letter met by words, spirit not) ([The Block](https://www.theblock.co/post/405117/trump-comments-fuel-dispute-over-polymarkets-120-million-permanent-iran-peace-deal-market)) |\n| COVID spreadsheet (Metaculus, 2020) | \"at 16:00 ET\" | The row for the stated date ([Metaculus notebook](https://www.metaculus.com/notebooks/22691/a-preliminary-look-at-metaculus-and-expert-forecasts/)) |\n| North Korea launch (earlier market) | Required Department of Defense confirmation, which never came | A launch occurred ([Rethink Priorities](https://rethinkpriorities.org/research-area/types-of-specification-problems-in-forecasting/)) |\n\n### 8.2 Where each platform puts the balance\n\n| Platform | Default | Escape valve |\n|---|---|---|\n| Metaculus | Written text; \"akin to a legal contract\" | Ambiguous/Annulled; admin clarification early; higher evidence bar later ([FAQ](https://www.metaculus.com/faq/)) |\n| Kalshi | Contract text and named source agencies | Outcome Review Committee, exchange-set payout, standardizing amendments ([rulebook](https://www.cftc.gov/filings/orgrules/rules1114248723.pdf)) |\n| Polymarket | Rules text, applied by a token vote | Clarifications that cannot change intent; no reversal after finalization ([docs](https://docs.polymarket.com/concepts/resolution)) |\n| Manifold | Creator's judgment | Moderator override ([FAQ](https://docs.manifold.markets/faq)) |\n| Good Judgment | Clarify, else void | Void without scores ([Wharton report](https://pvmi.wharton.upenn.edu/wp-content/uploads/2018/11/GJ-Challenge-2016-Summary-Report.pdf)) |\n\nTwo patterns stand out. First, the damaging cases combine a letter/title gap with an arbiter who cannot or will not apply the spirit, or with an edit after money is in. Second, platforms that never edit open questions in a way that shifts meaning (Metaculus) pay in annulled questions instead; the price of fidelity to the written text is a steady stream of \"no result\". The only study found of whether rule text predicts disputes concluded that text clarity was not a sufficient signal: \"disputes tend to emerge from external real-world uncertainty\" ([arXiv 2604.15674, Apr 2026](https://arxiv.org/html/2604.15674v1)).\n\n---\n\n## 9. AI-written questions and AI resolution (2024-2026)\n\n- **FutureSearch, Jan-Mar 2026.** An LLM pipeline proposed, refined and verified 1,499 forecasting questions (verifier agents rated quality, ambiguity and auto-resolvability; a question had to be rated \"great\" on quality and ambiguity and \"very certainly yes\" on resolvability to pass), then resolved them months later with three Gemini 3 Pro agents and a Claude Opus 4.5 tiebreaker. Against 100 questions resolved by an expert, there were 4 errors, including 2 missed annulments; the estimated annulment rate for generated questions was about 3.9% (95% interval 1.1-8.4%), against Metaculus's historical roughly 8% (their figure). Gemini alone never annulled; all five annulments came from the Opus tiebreaker, which the authors attribute to Gemini accepting search snippets without checking source admissibility. Manual resolution took 13 minutes per question even with AI help. ([arXiv 2601.22444](https://arxiv.org/html/2601.22444v2)) The paper's printed expert-rating counts (122 accept, 25 soft reject, 12 hard reject out of 149) do not add up (they sum to 159), so use the 8.1% hard-reject share only as approximate.\n- **ForecastBench (FRI, ICLR 2025).** A nightly automated system builds a question bank from prediction-market questions (chosen by liquidity) and from datasets via fixed templates, for example a template asking whether the number of protests in Niger will rise by at least 10% over the current month's value by the resolution date, asked at several horizons. ([ICLR paper](https://proceedings.iclr.cc/paper_files/paper/2025/file/ea74e45a229dac70b5b63b28d8934db6-Paper-Conference.pdf)) Current market sources include Kalshi, Metaculus and Polymarket ([ForecastBench](https://www.forecastbench.org/about/)).\n- **Kalshi's internal AI agent (June 2026).** Built on Claude, used to stress-test market wording and certification, suggest templates, and cross-check the two human decisions in a resolution; resolution is human-first, with the AI verifying that answers match. ([Bloomberg via predictionmarkets.org](https://www.predictionmarkets.org/kalshi-builds-ai-agent-to-stress-test-prediction-market-bets/))\n- **UMA's Optimistic Truth Bot (June 2025).** A router, solvers and an overseer propose answers; running in parallel to human proposers, not on-chain. About 78% accurate overall, about 95% when \"too early\" and 50/50 outcomes are excluded, 99.3% on sports and asset-price markets, and 72% on mention markets excluding \"too early\". UMA says it is deliberately conservative to avoid false positives. ([UMA blog, 3 Jun 2025](https://blog.uma.xyz/articles/experiment-can-ai-agents-enhance-uma-oracle))\n- **LLMs as dispute arbiters (Apr 2026).** Web-enabled models reproduced UMA's post-dispute outcomes about 89-90% of the time (best models), one OpenAI variant 71%; LLMs could not predict from rule text which markets would be disputed. The paper states disputed volume on Polymarket has reached $972M. ([arXiv 2604.15674](https://arxiv.org/html/2604.15674v1))\n- **Multi-agent resolvers (May 2026 thesis).** On 1,189 resolved Kalshi questions, independent confidence-weighted aggregation reached 83.43%, a deliberative variant about 76% (worse than any single model, attributed to persuasive error), and auto-resolving only unanimous high-confidence cases gave 97.87% accuracy on 47% of the questions; the rest went to a human. ([arXiv 2605.30802](https://arxiv.org/abs/2605.30802))\n- **Vendor and protocol proposals (unverified claims).** LLM committees for UMA votes, \"UNRESOLVABLE\" verdicts as a first-class output, hashed semantic definitions committed before the market opens ([Gensyn](https://www.gensyn.ai/research/credibly-neutral-ai-oracles); [Theseus docs](https://theseus.network/docs/build/adjudicate); [Cournot, Medium](https://medium.com/@CournotProtocol/polymarket-case-study-resolving-the-u-s-government-shutdown-market-with-cournot-protocol-d4220cab0150)). These are design proposals and marketing, not evidence.\n- **Observed pattern across all of the above:** AI does well on questions with a named data source or an API and worst on speech, geopolitics and wording-sensitive cases, which are also where humans dispute. It tends to resolve instead of abstaining, so an explicit abstain verdict has to be designed in.\n\n---\n\n## 10. What this suggests for Deliberus\n\nNumbered, each tied to the ruling where it applies. Items marked (inference) are my reading of the evidence, not something a platform documents.\n\nAt a glance, the four wording classes in the ruling against what the platforms do:\n\n| Class in the ruling | Platform analogue | Suggested treatment (items 2-5) |\n|---|---|---|\n| Stance hedge (\"probably\", \"I'd guess\") | Lives in the forecaster's probability field, never in the question text | Stance field: keep the original word, store a band, mark the mapping as inferred |\n| Scope word (\"most\", \"tend to\", \"some\") | A threshold or population slot, pinned down in the rules by number, index or examples | Stays in the claim; the bet is a separate, attributed sharpening |\n| Stated probability (\"70% chance\") | A forecast, judged by calibration over many events, not by one outcome | Stays in the claim; bet on the underlying event with the number as the author's credence |\n| Possibility-as-content (\"could cut emissions\") | Existence-by-deadline questions, or conditional questions with an Annulled branch | Stays in the claim; derive a demonstration bet and, where a decision is involved, a conditional bet |\n\n1. **Treat a \"testable bet\" as a derived object with named slots, not as a property of the claim.** Slots: (a) proposition in observable terms; (b) which kind of event triggers (occurred, announced, disclosed, signed, enacted); (c) resolution source, with precedence order and a rule for source error or disappearance, and an explicit choice between \"tracks the source's statements\" and \"tracks the world\" (Metaculus's FAQ says the intent is often unclear and sets a truth-tracking default; Kalshi's World Cup settlement followed the source; this contrast is my inference); (d) window with time zone and a \"before/by\" convention; (e) fallback mode (Ambiguous, Annulled, 50/50, default No at deadline). Missing slots should be visible as flags. Almost every failure in sections 1-3 is one slot absent or contradicted.\n2. **Hedge to stance, with a lossy-mapping guard.** The ruling matches platform practice: credence belongs in a probability field, never in the question. Because verbal hedges map to numbers poorly (0.08-0.59 for \"may/could/might\"), store the original token plus a band from a published lexicon (Kent's 1964 table, or the US Intelligence Community Directive 203 bands), mark the numeric mapping as inferred, and show the number next to the word as Wintle et al. found necessary. Keep a distinction between epistemic hedges (stance), content modals (\"could\" as possibility, \"may\" as permission) and \"might\" as unassessable likelihood. In the platform analogy, the author's stance is their forecast on the bet, and the graph's strength plays the part of the aggregate. (inference for the mapping to Deliberus's fields)\n3. **Scope words stay in the wording; the bet is a separate sharpening.** \"Most\", \"tend to\", \"some\" cannot resolve until a threshold and a population are chosen. Platforms keep the word in the title and pin it down in the rules by threshold, named index, or examples and counter-examples (section 6.2). Store each sharpening with its threshold, exemplars, counter-exemplars and who chose it, allow several per claim (FRI writes several operationalizations and lists false-positive and false-negative scenarios), and let a reader see which sharpening a verdict was about. Default quantifier conventions (\"most\" = more than half) should be explicit and overridable. (inference)\n4. **Stated probabilities: bet on the event, judge the number by calibration.** A claim \"70% chance of X\" cannot be resolved by one outcome (Tetlock: only 0 and 1 are falsifiable singly). Platform-consistent handling: derive a bet on X, record 0.7 as the author's credence, and assess the number over the author's or the pool's set of such statements. The claim node itself should not flip to \"false\" when X fails. (inference)\n5. **Possibility-as-content needs two kinds of bet, and a search-domain rule.** \"This policy could cut emissions\" has (i) a demonstration form (Yes can be shown early; No only at a deadline; the source or search domain must be named, else Metaculus's FTX-style \"no path to No\" trap applies) and (ii) a conditional form, \"if the policy is adopted, emissions fall by at least N per source S\", which is Annulled if the antecedent never occurs (Metaculus conditional pairs). Note that a conditional is not a causal test.\n6. **Causal and counterfactual claims should not get a direct bet.** Use indicator clusters (Tetlock), conditional pairs, or an intersubjective or panel resolution (FRI reciprocal scoring, Metaculus resolution councils). Weight these as a weaker evidence class in the strength computation, since reciprocal scoring has been criticized as a beauty contest.\n7. **Value claims get no bet; the empirical premise inside them might.** Metaculus excludes opinion; Manifold shows the trust cost of allowing it. Extract the factual premise when one exists and say plainly when none does.\n8. **Give \"no result\" its own states and make them count as no evidence.** Ambiguous (reality unclear) and Annulled (reality clear, question flawed) mean different things; keep both, plus \"not yet\" and \"too early\". They must not move strength. Track the unresolvable share by claim type as a quality metric; published figures run about 4% (generated, small sample) to 8% (Metaculus, secondhand) to 18% (long-horizon cyber tournament, 6 of 43).\n9. **Freeze and version.** Once a bet is published its text is immutable. Clarifications are dated addenda that cannot change intent, with an effective date, and resolution runs against the frozen text. Borrow Metaculus's asymmetry (later claims that the criteria were unclear need stronger evidence). The Strategy suit is the cautionary case: a note that reads as new criteria after money is in.\n10. **Pipeline acceptance tests, all cheap enough to run on every bet:** the clairvoyance test; a \"path to Yes and path to No\" check; a title/claim-versus-rule match check (Metaculus rule 5); a source-error check (what if the named source is wrong, as with Pitt); false-positive and false-negative scenario listing (FRI); a check that the bet text contains no hedge token. Text-only checks will miss some failures (the arXiv result), so keep a human audit of a sample and log later disputes against bet features.\n11. **Use a small template library for common claim types** instead of free-form generation: threshold-by-date, event-by-date, announcement/disclosure, count or mention, ranking-by-index, label-trigger. Kalshi's 500+ templates, ForecastBench's dataset templates and the repeating Polymarket structures show the template is where the precision lives. Start with a handful; route the rest to \"needs human operationalization\". (inference on the size to start with)\n12. **Show users the gap between claim and bet.** The title-versus-rule gap types (narrower scope, stricter threshold, hidden source, hidden timing, different entity) make a usable label on each derived bet: \"this bet tests a narrower version of the claim\". Use the taxonomy; do not rely on the audit site's numbers.\n13. **Separate forecast bets from verification bets.** Many extracted claims concern past or present facts, which platforms do not host as markets (Metaculus even has a FAQ item on criteria fulfilled before opening). These resolve by lookup against a named source and are cheaper, but they are not forecasts and should not be scored like them. (inference)\n14. **Feed the author back in.** FRI's closest analogue to Deliberus is expert prompt to question, and it sends the operationalized conditions back to the person whose idea it was. A lightweight \"does this bet capture what you meant?\" step, or at least a recorded reading, would protect against the extractor's operationalization drifting from the source's intent.\n15. **If AI drafts or resolves, build in refusal and keep commentary out of the rules.** Use two or more models plus a tiebreak, require an \"unresolvable\" verdict, auto-resolve only unanimous high-confidence cases and send the rest to a human (97.87% on 47% in one study), and keep AI-written context visibly separate from the bet text (as Polymarket's disclaimer does).\n\n---\n\n## 11. Could not verify, and caveats\n\n- **Metaculus annulment rate.** The \"about 8%\" figure comes from FutureSearch and Lawfare, not from Metaculus. Metaculus's own count was not found.\n- **Polymarket rules via API.** Rules text was read from the public API on 2026-10-19. I did not confirm whether the API field includes every later on-chain \"additional context\" note; the Strategy note and the Ukraine clarifications are known from press and community sources, not from the API output. The 2026 Strategy contract's exact wording was taken from press; the retrieved text is the 2025 sibling.\n- **Polymarket help article on clarifications** returned 404; the clarification description is from the docs page. The Zelensky clarification statement and the \"no updates or clarifications\" clause in another market come from a community guide ([PolymarketGuide](https://polymarketguide.gitbook.io/polymarketguide-archive/precedents/polymarket/market-clarifications)), not from Polymarket.\n- **Paywalled or inaccessible:** the WSJ and Bloomberg investigations of UMA voting concentration are known from secondary summaries. Metaculus pages block direct fetching; text was read from search-engine copies of the FAQ and guidelines and may differ from the live site.\n- **Figures that disagree between sources:** volumes for the Zelensky (about $237M versus $242M, range $160-242M), Strategy ($60M versus $80M+) and Iran markets; Kalshi's Khamenei settlement price (29 cents in a complaint, 39.5% in MarketWatch) and amount wagered ($54M versus $55M); the date and nature of the UMA managed-oracle whitelist.\n- **Venezuela outcome:** early January 2026 coverage disagrees on whether Polymarket declared No or left the market unsettled; I did not find a later definitive account.\n- **Community and advocacy sources used with caution:** PolymarketGuide (community-written), Brierly dispute database (polymarketscam.com, an advocacy archive, was read but not cited), the mr.ozi Substack, the Bernard Bulletin compilation of Kalshi mention disputes, the Manifold market comment with moderator counts.\n- **Asterisk audit site:** the page does not state who runs it, who or what scores titles, or how \"error\" is defined; its sample counts range from 14 to 391 markets. Treat as a source of taxonomy ideas, not evidence.\n- **FutureSearch paper:** the expert-rating counts are internally inconsistent (see section 9). The comparison to Metaculus's 8% rests on a secondhand figure, and the 3.9% rests on a small human-checked sample (3 of 100).\n- **Good Judgment Open's current public question-writing guidance** was not found; Good Judgment material cited is older reports.\n- **Kalshi's Khamenei contract text** was quoted from MarketWatch and the current contract-terms template (WLEADEROUT), not from the February 2026 PDF; a rule-text change is known to have been filed afterward.\n- **Vendor material** (Previsions, Vinfotech, Gensyn, Theseus, Cournot, and similar) was read but not relied on for facts.\n- **Nothing here was tested against Deliberus's actual extraction output.** The mapping in section 10 is a reading of the platform evidence against the ruling, not an evaluation of the pipeline.\n"}