{"path":"research/how-independent-reasons-add/literature-pooling-and-dependence.md","content":"# Literature report: pooling, extremizing, dependence, naive Bayes\n\n*A research helper's report for `../how-independent-reasons-add.md`, returned 2026-10-22 and kept verbatim\nbelow (the brief asked about opinion pooling, forecast aggregation, dependent experts and naive Bayes\ncalibration, the four angles the corpus had not covered). Its arithmetic marked \"mine\" is the helper's own,\nfrom a paper's formula. Read by Claude before use; nothing below is a ruling.*\n\n---\n\n# How several supporting reasons should combine: pooling, extremizing, dependence, naive Bayes\n\nI checked every formula and source below. Where I cite a number, the tags in section 5 say whether I read it in the paper or only in an abstract, snippet or secondary source. Arithmetic marked \"mine\" is my own calculation from a paper's formula, not a printed result. I saved two text extractions of fetched papers in the session scratchpad. I wrote no project files.\n\n## 0. Bottom line\n\n- **The founder's \"45% each\" is the equal-weight linear pool.** Two 0.9 sources give 0.5·0.9 + 0.5·0.9 = 0.9, and adding a second agreeing source changes nothing. Dietrich and List say linear pooling can be defended \"procedurally\" but not \"epistemically\". Ranjan and Gneiting prove that a linear pool of calibrated sources is necessarily underconfident.\n- **Whether the answer is 0.9 or 0.988 depends on whether the sources share evidence.**\n  - Identical evidence gives 0.9.\n  - Fully independent evidence gives odds multiplication: 81:1, or 0.988 (with a 0.5 prior).\n  - Partial overlap gives something between, and the literature supplies a dial for it (next point).\n- **The dial is an \"equivalent number of independent sources\".** Clemen and Winkler (1985): n sources with pairwise error correlation ρ are worth n/(1+(n−1)ρ) independent ones, with a ceiling of 1/ρ.\n  - If sources agree and their noise is equicorrelated and Gaussian, the combined log-odds are n_eff times one source's log-odds (my derivation, not printed in the paper).\n  - For two 0.9 sources, ρ = 0 gives 0.988, ρ = 0.25 gives 0.971, ρ = 0.5 gives 0.949, ρ = 0.8 gives 0.920, ρ = 1 gives 0.900.\n- **Forecasting practice confirms that raw averaging is too timid, and that crowds are worth only a few independent voices.** In first-year Good Judgment Project (GJP) data, the optimal extremizing exponent on the mean was 2.43 (experts) and 3.08 (non-experts). On the median it was 1.78 and 2.34. The Satopää–Pemantle–Ungar model ties that exponent to crowd size and information overlap. For forecasters who each use only a small share of the total information, the exponent approaches N/((N−1)λ+1) (λ = pairwise information overlap, N = number of forecasters), which is capped at 1/λ. This is the same expression as Clemen–Winkler, with λ in place of ρ.\n- **Naive Bayes is the documented failure of ignoring dependence.** It counts correlated evidence repeatedly, which makes probabilities too extreme while ranking stays good. Bennett (2000) shows its log-odds grow with every added word.\n- **Scope caveat (my inference).** All four literatures treat each source as stating a probability for the claim itself. A QEM \"reason strength\" is a different object. A single 0.9 source returns 0.9 under every rule above, whereas QEM maps one 0.9 reason to 0.57. Any pooling-based rule therefore needs reason strength first converted into \"how strongly does this reason bear on the claim\" (a likelihood ratio).\n\n## 1. Linear versus logarithmic opinion pooling\n\n**Finding.** Dietrich and List review three pool families and say which situation each suits.\n\n- **Linear pool: weighted average of probabilities.**\n  - It is characterized by \"eventwise independence\" plus unanimity preservation (Aczél–Wagner 1980; McConway 1981).\n  - It is easy to run and cannot be manipulated by changing which other questions are on the agenda.\n  - It has no epistemic justification: it does not \"incorporate the information the opinions are based on\".\n- **Geometric (logarithmic) pool: normalized weighted geometric mean of the probabilities of each possible world.**\n  - It is the only pool with \"external Bayesianity\" (updating before or after pooling gives the same result; Madansky 1964, Genest 1984).\n  - That property is compelling only when everyone holds the same information and differs only in interpretation.\n  - It preserves unanimity, so two 0.9 sources return 0.9.\n- **Multiplicative pool (Dietrich 2010).**\n  - Group probability ∝ P0 · P1 · … · Pn, with P0 ∝ 1/p^(n−1) when everyone shares a prior p.\n  - It is the only rule satisfying \"individualwise Bayesianity\": a single member learning a private fact changes the pooled result as if the group learned it.\n  - It suits sources whose evidence is private and non-overlapping.\n  - It deliberately violates unanimity preservation. The authors' own example is a panel of experts who all say 0.9 because of independent private evidence, where the collective should be \"amplified, above 0.9\".\n\n**Sources that share information (D&L, section 10).**\n\n- Shared information across all sources favours the geometric pool.\n- Fully private information favours the multiplicative pool.\n- Information shared by a proper subset of the sources needs a more complicated recursive pool (Dietrich 2010, Theorem 4).\n- If the private pieces are correlated, individualwise Bayesianity loses its force, because \"the combined evidential strength of two correlated pieces is not their product\". D&L give no formula for the correlated case.\n\n**Dietrich (2010).**\n\n- Group belief π ∝ π1···πn / p^(n−1) under a common prior p.\n- Without a common prior, π ∝ (π1/p1)···(πn/pn) · F(p1…pn), with F a pool of the priors.\n- Subgroup-shared information breaks the independence assumption; Theorem 4 handles it.\n\n**Numbers for two sources, each 0.9, prior 0.5 (mine, from the formulas above).**\n\n| Rule | Result |\n|---|---|\n| Linear, equal weights | 0.900 |\n| Geometric, equal weights | 0.900 |\n| Multiplicative, uniform P0 (binary, prior 0.5) | 0.81/(0.81+0.01) = 0.9878 |\n\n**Calibration argument against linear pooling (Ranjan and Gneiting 2010, Theorem 1).** Any weighted average of two or more distinct, individually calibrated probability forecasts is uncalibrated, and lacks sharpness: on average it sits closer to the base rate than its recalibrated version. They propose a beta-transformed linear pool to repair this. The Satopää et al. (2014) introduction cites this, and also Allard et al. (2012), whose simulations find multiplicative-structure pools beat linear pools.\n\n**Genest and Zidek (1986).** I could read only the abstract: a taxonomy of pooling solutions plus an annotated bibliography. I did not read the paper. Attributions to Genest's own theorems come through D&L.\n\n## 2. Forecast aggregation and extremizing\n\n**Finding.** In first-year GJP data, simple averaging was too timid. The fix is to push the aggregate further from 0.5, usually by raising the pooled odds to a power a > 1. Two separate causes are identified.\n\n**Baron, Mellers, Tetlock, Stone and Ungar (2014).**\n\n- Transformation: t(p) = p^a / (p^a + (1−p)^a), which is the same as raising the odds to the power a (their Figure 1 uses a = 2.5).\n- Cause 1, scale compression: random error near 0 or 1 is asymmetric on the probability scale, so the mean is pulled toward 0.5. This does not affect the median or the mean of log-odds.\n- Cause 2, regression from ignorance: each forecaster shrinks toward 0.5 because they know they lack information the group collectively has. In simulation, a = 1/c, where c is the shrink constant, for the median or for zero noise.\n- Data: 1,973 forecasters, 86 questions (80 two-option questions used for fitting).\n\nOptimal a, from the paper's Table 1 and the prose that repeats it (leave-one-question-out standard error in brackets):\n\n| Group | Mean of probabilities | Median |\n|---|---|---|\n| Self-rated high expertise | 2.43 (0.05) | 1.78 (0.03) |\n| Non-experts | 3.08 (0.09) | 2.34 (0.06) |\n\nThe paper states that experts need less transformation, and so does the median.\n\n**Satopää et al. (2014).**\n\n- The aggregator is based on the geometric mean of expert odds, with one tuning parameter that shifts each probability toward its nearest extreme. It was tested on over 1,300 experts forecasting 69 events and beat arithmetic mean, median and plain logarithmic pool.\n- Their model puts a constant between 0 and 1 on the \"true log-odds\", so the extremizing exponent is roughly its inverse (my reading of the stated model).\n- The only source I found for the optimal a ranging from 1.161 to 3.921 is Jaime Sevilla's EA Forum post (2021). I did not see the paper's results section and could not verify that range.\n\n**Satopää, Pemantle and Ungar (2016), the formula you asked for.**\n\n- Setup: N forecasters, each seeing a share δ of a Gaussian information pool; pairwise overlap λ·δ; the target event has prior probability 0.5; forecasts are calibrated.\n- Revealed aggregator under symmetric overlap (their eq. 6): p″ = Φ( [1/((N−1)λ+1)] · Σ Φ⁻¹(pᵢ)√(1−δ) / √(1 − Nδ/((N−1)λ+1)) ).\n- Probit extremization ratio relative to the probit average (their Proposition 5.1): α = γ√(1−δ) / √(1−δγ), with γ = N / ((N−1)λ+1).\n- Limits: complete overlap gives α = 1 (no extremizing); zero overlap gives the maximum.\n- As δ → 0, α → γ. For large N, γ → 1/λ. So λ = 0.1, 0.25, 0.5 caps the exponent at 10, 4, 2 (mine).\n- They state extremizing rises with N and with diversity, and falls with collaboration, resource sharing and problem difficulty. Their explanation for why GJP superforecaster teams needed little or no extremizing is high individual information and high overlap.\n- Their own Example 3.1: a basket holds a fair coin and a two-headed coin, a coin is picked at random, and forecasters judge whether it is two-headed after seeing heads. If both forecasters saw the same flip, the optimal answer is 2/3. If they saw different flips, it is 4/5. This is exactly \"same evidence versus independent evidence\" in miniature.\n- Numbers for N = 2, both saying 0.9, small δ (about 0.05–0.1), from my arithmetic on their eq. 6:\n\n| Overlap λ | Combined |\n|---|---|\n| 0 | ≈ 0.996 |\n| 0.25 | ≈ 0.98 |\n| 0.5 | ≈ 0.96 |\n| 0.75 | ≈ 0.93 |\n| 1 | 0.90 |\n\nThe λ = 0 value is higher than logit addition (0.988) because the model works on a probit scale. The tail values depend on that scale choice.\n\n**How many independent sources is a crowd?**\n\n- Satopää and Ungar (2015, real-valued forecasts): 5 synthetic forecasters with no information overlap needed extremizing α = 5.01, essentially N. With high overlap α = 1.30.\n- That supports reading the extremizing factor as roughly \"effective number of independent sources\" in the no-regression case. This reading is mine; the papers do not put it that way. Baron et al. estimate a to also absorb individual shrinkage, so a ≈ 2.4–3.1 is not a clean count.\n- Neyman and Roughgarden (2022): in a worst-case-robust linear-extremizing setting around the prior, the optimal factor d(k) = k(√(3k²−3k+1) − 2) / (k²−k−1) rises with the number of experts k and approaches √3 ≈ 1.73. It is about 1.29 at k = 2 and 1.63 at k = 10 (mine, from the formula). They say this is consonant with empirical findings in the literature.\n- Lichtendahl, Grushka-Cockayne, Jose and Winkler (2022): extremizing should be defined as moving away from the base rate, not from 0.5. They show cases where the right move is the opposite (\"antiextremizing\"), between 18% and 73% of cases on public datasets. The question here has prior 0.5, so base-rate shifts do not arise.\n- Tetlock, Mellers, Rohrbaugh and Chen (2014): the log-odds extremizing algorithm contributed about a 35% boost over unweighted averaging in the GJP.\n\n## 3. Dependent experts: Clemen and Winkler (1985)\n\n**Finding.** Correlated experts add much less than independent ones, with a hard ceiling. The paper's model has k experts each giving a point estimate of an unknown quantity, with normally distributed errors and covariance matrix Σ.\n\n**Definition.** The equivalent number of independent experts is n(σ²,Σ) = σ² · eᵀΣ⁻¹e, where e is a vector of ones. This equates the posterior variance from Σ with that from n independent experts of variance σ².\n\n**Closed form (eq. 7), common variance σ² and common pairwise correlation ρ > 0:**\n\nn(σ²,Σ) = k · [1 + (k−1)ρ]⁻¹\n\nThe paper states:\n- n < k for k > 1, and the value falls as ρ rises.\n- n is concave in k with limit 1/ρ as k → ∞.\n- ρ = 0.80, which the authors say is consistent with their football and economic forecasting data, gives n = 1.25: after the first expert, all others combined are worth a quarter of one independent expert.\n- ρ = 0.25 caps the group at four independent experts.\n\n**Marginal value of the k-th expert** (the OCR of the paper is garbled; this is my algebra, consistent with the legible fragments): Δn(k) = (1−ρ) / ([1+(k−1)ρ] · [1+(k−2)ρ]), a decreasing, convex function of k.\n\n**Equivalent-independent-sources table** (mine, from eq. 7):\n\n| ρ | k=2 | k=3 | k=5 | k=10 | k=100 | limit 1/ρ |\n|---|---|---|---|---|---|---|\n| 0.10 | 1.82 | 2.50 | 3.57 | 5.26 | 9.17 | 10 |\n| 0.25 | 1.60 | 2.00 | 2.50 | 3.08 | 3.88 | 4 |\n| 0.50 | 1.33 | 1.50 | 1.67 | 1.82 | 1.98 | 2 |\n| 0.80 | 1.11 | 1.15 | 1.19 | 1.22 | 1.25 | 1.25 |\n\n**Binary-claim transposition (my derivation, not in the paper).** Suppose each source's signal is Gaussian around ±m for claim true or false, with equicorrelated noise, and each source's stated 0.9 is the calibrated posterior from its own signal alone. If all sources agree, the combined log-odds equal n_eff times one source's log-odds. Two 0.9 sources with ρ = 0.5 therefore give odds 9^1.333 ≈ 18.7, or 0.949.\n\n**Caveats.** The model is about point-estimate precision under normal errors, not directly about binary-claim probabilities. Morrison and Schmittlein (1991) give geometric intuition for the result and show the formula breaks down as the covariance matrix approaches singularity (abstract only). Winkler (1981) is the model on which Clemen and Winkler build (abstract only). Winkler, Grushka-Cockayne, Lichtendahl and Jose (2019) restate the limit as k → ∞ being equal to 1/ρ independent forecasts, and note that with ρ = 0.5 any number of forecasts is worth fewer than two independent ones. They also suggest 5–10 forecasts is a good size in practice.\n\n## 4. Naive Bayes calibration\n\n**Finding.** Treating dependent evidence as independent multiplies the dependent items' likelihood ratios again and again. That distorts probabilities strongly toward 0 and 1 while leaving ranking mostly intact.\n\n- **Mechanism (Domingos and Pazzani 1997, section 4).** Take A and C independent and B identical to A. The Bayesian classifier then counts A twice. The correct rule is \"assign + when q > 1−p\". The naive rule is \"assign + when q > (1−p)²/(p²+(1−p)²)\". The two rules disagree only in two narrow regions, so decisions are mostly unaffected even though the probabilities are wrong.\n- **Illustrative extreme (D&P, not a measurement).** True P(+|E) = 0.51 and P(−|E) = 0.49; naive Bayes can output 0.99 and 0.01 and still classify correctly. Their theorem says quadratic-loss optimality of the independence assumption is a second-order infinitesimal fraction of the zero-one optimality region.\n- **Zadrozny and Elkan (2001, section 2.5; repeated in 2002).** Because attributes tend to be positively correlated, naive Bayes scores are typically too extreme: near 0 they are below the true probability, near 1 above it. They still rank examples well. A calibration step (binning, sigmoid, isotonic/PAV) fixes much of it. On the KDD-98 donor data the profit measured with naive Bayes scores was $9,531 on test, against $14,120 with sigmoid correction and $14,447 with PAV. Mean squared error moved only from 0.1011 to 0.0953, so squared error understates the decision damage.\n- **Niculescu-Mizil and Caruana (2005).** Naive Bayes pushes predicted values toward 0 and 1, the opposite of max-margin methods such as SVMs and boosted trees, and its reliability diagram is an inverted sigmoid. A sigmoid is not the right correction for it; isotonic regression is better.\n- **Bennett (2000, CMU-CS-00-155), the mechanism in one line.** Adding a word multiplies the posterior odds by that word's likelihood ratio, R_d = [P(wₙ|c)/P(wₙ|¬c)] · R_d′, so odds grow exponentially in document length. His Figure 1 shows posteriors almost always 0 or 1 on the Reuters classes Earn and Corn. His Figure 2 log-odds axes span about −1200…+400 (Earn) and −500…+400 (Corn). Before correction, mean squared error was 0.0303 (Earn) and 0.0359 (Corn). His diagnosis: adding a word does not add as much new evidence as naive Bayes believes, because words are not independent.\n- **A 0.9 reason counted m times (mine, arithmetic only):** m = 1 gives 0.90, m = 2 gives 0.988, m = 3 gives 0.9986, m = 5 gives 0.99998.\n\n## 5. What this means for Deliberus (my inference, not from the papers)\n\n- **Pure averaging is not neutral.** The \"45% each\" rule hard-wires \"agreeing sources add no confidence\". That is right only if the reasons are the same evidence restated. For reasons with different evidence it is the timid option, and Ranjan–Gneiting say it is miscalibrated even under ideal conditions.\n- **Pure multiplication is not neutral either.** It is the independent-evidence assumption, and naive Bayes shows what it does when that assumption is false: it overclaims.\n- **The workable middle** is to convert each reason to a likelihood ratio, then sum log-odds with an overlap discount: the combined weight of k reasons is k/(1+(k−1)ρ) times one reason's weight, capped at 1/ρ. The remaining design question is where ρ comes from. Candidates are an explicit \"shares evidence with\" link between reasons, grouping by shared source, or a default of about 0.25–0.5 until a grouping is declared. None of this is validated by the papers.\n- **Do not reuse extremizing exponents (1.8–3.1) for two reasons.** They are crowd-level corrections that fold in individual underconfidence.\n\n## 6. Reading status for each source\n\n- **Read in full:** Dietrich and List review (author preprint of the final version); Clemen and Winkler 1985 (author-hosted OCR copy; some symbols garbled); Baron et al. 2014 (the table's decimals are garbled in the PDF extraction, but the same values are stated in the prose); Satopää, Pemantle and Ungar 2016 (author preprint); Domingos and Pazzani 1997 (UCI copy); Bennett 2000.\n- **Read in part (passages from the page or PDF, not the whole paper):** Dietrich 2010 (Theorem 1 and conclusion); Ranjan and Gneiting 2010 (Theorem 1 and discussion); Neyman and Roughgarden 2022 (introduction and Section 4); Satopää and Ungar 2015 (tables and Section 3); Zadrozny and Elkan 2001 and 2002; Niculescu-Mizil and Caruana 2005; Satopää et al. 2014 (introduction, theory stub, results stub, conclusions only); Winkler et al. 2019 (the working-paper passage on how many forecasts to combine); Tetlock et al. 2014 (search-extracted passage).\n- **Abstract only:** Genest and Zidek 1986; Winkler 1981; Morrison and Schmittlein 1991; Lichtendahl et al. 2022.\n- **Secondary only:** the range of optimal a in Satopää et al. 2014 (EA Forum post by Sevilla); the attribution of external Bayesianity to Madansky 1964 and Genest 1984 (via D&L and Sevilla).\n- **Not found:** any paper that states \"the optimal extremizing factor equals the effective number of independent sources\". The nearest is the Satopää–Pemantle–Ungar formula and the Satopää–Ungar synthetic result (5.01 for five independent sources), both described above.\n\n## 7. Citations\n\n1. Dietrich, F. and List, C. (2016/2017). Probabilistic opinion pooling. In Hájek, A. and Hitchcock, C. (eds.), *The Oxford Handbook of Probability and Philosophy*, ch. 25, pp. 519–542. Oxford University Press. DOI 10.1093/oxfordhb/9780199607617.013.37. Preprint: https://philsci-archive.pitt.edu/11349/1/OpinionPoolingReview.pdf\n2. Genest, C. and Zidek, J. V. (1986). Combining probability distributions: a critique and an annotated bibliography. *Statistical Science* 1(1), 114–135. DOI 10.1214/ss/1177013825.\n3. Dietrich, F. (2010). Bayesian group belief. *Social Choice and Welfare* 35(4), 595–626. DOI 10.1007/s00355-010-0453-x.\n4. Ranjan, R. and Gneiting, T. (2010). Combining probability forecasts. *Journal of the Royal Statistical Society B* 72(1), 71–91. DOI 10.1111/j.1467-9868.2009.00726.x.\n5. Satopää, V. A., Baron, J., Foster, D. P., Mellers, B. A., Tetlock, P. E. and Ungar, L. H. (2014). Combining multiple probability predictions using a simple logit model. *International Journal of Forecasting* 30(2), 344–356. DOI 10.1016/j.ijforecast.2013.09.009.\n6. Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E. and Ungar, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. *Decision Analysis* 11(2), 133–145. DOI 10.1287/deca.2014.0293. PDF: https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015---two-reasons-to-make-aggregated-probability-forecasts_1.pdf\n7. Satopää, V. A., Pemantle, R. and Ungar, L. H. (2016). Modeling probability forecasts via information diversity. *Journal of the American Statistical Association* 111(516), 1623–1633. DOI 10.1080/01621459.2015.1100621. Preprint: https://www2.math.upenn.edu/~pemantle/papers/forecasting.pdf\n8. Satopää, V. A. and Ungar, L. (2015). Combining and extremizing real-valued forecasts. arXiv:1506.06405.\n9. Neyman, E. and Roughgarden, T. (2022). Are you smarter than a random expert? The robust aggregation of substitutable signals. *EC '22*, 990–1012. DOI 10.1145/3490486.3538243. arXiv:2111.03153.\n10. Lichtendahl, K. C., Grushka-Cockayne, Y., Jose, V. R. and Winkler, R. L. (2022). Extremizing and antiextremizing in Bayesian ensembles of binary-event forecasts. *Operations Research* 70(5), 2998–3014. DOI 10.1287/opre.2021.2176.\n11. Tetlock, P. E., Mellers, B. A., Rohrbaugh, N. and Chen, E. (2014). Forecasting tournaments. *Current Directions in Psychological Science* 23(4), 290–295. DOI 10.1177/0963721414534257.\n12. Clemen, R. T. and Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. *Operations Research* 33(2), 427–442. DOI 10.1287/opre.33.2.427. PDF: https://people.duke.edu/~clemen/bio/Published%20Papers/01.Limits-Clemen&Winkler-OR-85.pdf\n13. Winkler, R. L. (1981). Combining probability distributions from dependent information sources. *Management Science* 27(4), 479–488. DOI 10.1287/mnsc.27.4.479.\n14. Morrison, D. G. and Schmittlein, D. C. (1991). How many forecasters do you really have? Mahalanobis provides the intuition for the surprising Clemen and Winkler result. *Operations Research* 39(3), 519–523. DOI 10.1287/opre.39.3.519.\n15. Winkler, R. L., Grushka-Cockayne, Y., Lichtendahl, K. C. and Jose, V. R. (2019). Probability forecasts and their combination: a research perspective. *Decision Analysis*, online 6 Nov 2019. DOI 10.1287/deca.2019.0391. Working paper (the version I read): https://www.hbs.edu/ris/Publication%20Files/19-039_8af5ea92-6191-4774-aeb6-5eee0a5fd4c2.pdf\n16. Domingos, P. and Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. *Machine Learning* 29, 103–130. DOI 10.1023/A:1007413511361. PDF: https://ics.uci.edu/~pazzani/Publications/mlj97-pedro.pdf\n17. Zadrozny, B. and Elkan, C. (2001). Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. *ICML 2001*, 609–616. https://cseweb.ucsd.edu/~elkan/calibrated.pdf\n18. Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. *KDD '02*. DOI 10.1145/775047.775151.\n19. Niculescu-Mizil, A. and Caruana, R. (2005). Predicting good probabilities with supervised learning. *ICML 2005*. DOI 10.1145/1102351.1102430.\n20. Bennett, P. N. (2000). Assessing the calibration of naive Bayes' posterior estimates. Technical report CMU-CS-00-155, Carnegie Mellon University. http://reports-archive.adm.cs.cmu.edu/anon/2000/CMU-CS-00-155.pdf\n21. Sevilla, J. (2021). When pooling forecasts, use the geometric mean of odds. EA Forum. https://forum.effectivealtruism.org/posts/sMjcjnnpoAQCcedL2/when-pooling-forecasts-use-the-geometric-average-of-odds (secondary source, used only for the a = 1.161–3.921 range).\n"}