{"path":"research/priors-as-standing-work/research-report-2026-10-24.md","content":"# Research report: machine pre-answers as a capped evidence kind, bot-seeded encyclopedias, and pre-annotation anchoring\n\n*A research helper's report (Sonnet, web research, 2026-10-24), written to disk verbatim before it was read\nfurther, for `../priors-as-standing-work.md`. Its brief named three angles the corpus lacked and the covered\nangles to skip. Read-status tags are the helper's own. Numbers in it are the helper's claims until checked\nagainst the sources it names.*\n\n---\n\nRESEARCH REPORT: machine pre-answers as a capped evidence kind, bot-seeded encyclopedias, and pre-annotation anchoring\n\nRead-status tags: [full text] = I read the source's text myself. [part] = I read excerpts. [abstract only]. [secondary] = I read only a summary or a mirror. Nothing here is invented; \"not found\" marks gaps. The Pejaver numbers come from the March 2022 bioRxiv preprint, which I read from the PDF. The published AJHG version (Nov 2022) could not be fetched (reCAPTCHA wall), so thresholds may differ slightly.\n\n---\n\n## 1. Machine predictions as a capped evidence kind in variant classification\n\n**The 2015 rule.**\n- ACMG/AMP 2015 (Richards et al., Genet Med) lets computational predictors count only as \"Supporting\" evidence (PP3 for pathogenic, BP4 for benign). The criterion applies once per variant. [secondary]\n- Per Pejaver's intro, the 2015 guideline required two or more algorithms and counted them as Supporting only if all tested algorithms agree. Pejaver adds that \"many methods overlap and thus do not offer independent assessment\". [part: preprint intro]\n- I could not retrieve the exact 2015 sentence usually quoted (that each algorithm \"cannot be counted as an independent criterion\" because they share similar input). A 2023 splicing-adaptation table repeats the idea, so treat it as [secondary].\n\n**Why \"one line of evidence.\"**\n- Shared inputs and training sets mean correlated errors.\n- Ghosh, Oak & Plon 2017 (Genome Biol 18:225) [part, via full-text excerpts] took 14,819 ClinVar missense variants (7,346 benign, 7,473 pathogenic) and 25 algorithms (18 with public thresholds).\n  - Full concordance across all 18 held for only 5.2% of benign and 39.2% of pathogenic variants.\n  - Five popular tools (PolyPhen, SIFT, CADD, PROVEAN, MutationTaster) agreed on 79% of pathogenic and only 33% of benign variants.\n  - Concordance for combinations of 2–5 tools ran 83.4% down to 64.5%.\n  - They named a new error, \"false concordance\": tools agree and are wrong. The best trio for benign variants had a 2.8% false-concordance rate; the best pathogenic trio had up to 25.8% on benign variants.\n  - They found no combination with 0% false concordance.\n  - The ACMG \"all must agree\" rule led to inconsistent lab practice and more VUS.\n\n**Pejaver et al. 2022 (AJHG), the calibration.** [part: preprint, 40 pages]\n- Method: local positive likelihood ratios, with the prior probability of pathogenicity set at 0.0441. This replaced Tavtigian's 0.10.\n- Table 1, posterior and likelihood ratio per strength for PP3:\n  - Supporting: 0.0999, LR 2.406\n  - Moderate: 0.2108, LR 5.790\n  - Strong: 0.6073, LR 33.53\n  - Very Strong: 0.9811, LR 1,124\n  - For BP4 the posteriors of benignity are 0.9812, 0.9921, 0.9986 and 1.0000, with the reciprocal ratios.\n- Data:\n  - ClinVar 2019 set: 11,834 variants from 1,914 genes, after removing VUS, conflicting entries and zero-star entries.\n  - gnomAD set: 363,894 variants from 3,640 genes.\n  - ClinVar 2020 test set: 9,114 variants from 2,197 genes.\n  - Any variant in a tool's training set was removed. For meta-predictors such as REVEL and BayesDel, their constituent tools' training variants were removed as well.\n- Thresholds (Table 2), selected rows:\n  - REVEL: Supporting ≥0.644, Moderate ≥0.773, Strong ≥0.932. Benign: ≤0.290 Supporting, ≤0.183 Moderate (pathogenic Supporting starts at 0.644), ≤0.016 Strong, ≤0.003 Very Strong.\n  - BayesDel: Strong ≥0.50.\n  - MutPred2: Strong ≥0.932.\n  - VEST4: Strong ≥0.965.\n  - Only these four tools reached Strong for pathogenicity. 9 of 13 tools topped out at Moderate, and several only at Supporting.\n  - REVEL alone reached Very Strong, and only for benign calls.\n- Fractions of variants:\n  - For tools that reach Strong pathogenic, only 1.4–1.7% of gnomAD variants land in the Strong interval.\n  - Most tools output an Indeterminate score (no evidence at all) for 26–50% of variants. FATHMM, GERP++ and MPC were indeterminate for more than half.\n- ClinGen recommendation, in the preprint:\n  - Laboratories should pick ONE tool genome-wide (BayesDel, MutPred2, REVEL or VEST4) and choose it before seeing prediction results, \"to avoid biases such as those from multiple trials\". Scanning many tools for the strongest result is explicitly the failure.\n  - The sum of PP3 and PM1 is capped at Strong. Reason: overlapping data, so double-counting of conservation and domain information.\n  - Use of PM2/BS1 with a frequency-blind tool is allowed.\n  - Published points mapping, from a search summary [secondary]: PP3 up to +4 (Strong), BP4 down to −2 (Moderate).\n  - A 2025 follow-up (Bergquist et al., Genet Med) calibrated 3 more tools [secondary].\n\n**Documented failures.**\n- Grimm et al. 2015 (Hum Mutat 36:513) [part]: tested 10 tools on 5 datasets and found two kinds of circularity.\n  - Type 1: the same variants appear in training and evaluation.\n  - Type 2: different variants from the same protein appear in both. Labels cluster by gene, so \"predict from other variants in the same protein\" scores well without learning anything.\n  - Numbers: ~91% of pathogenic and ~68% of neutral variants in one benchmark sit in proteins also in HumVar/ExoVar. FatHMM-W's strong results were attributed to type-2 circularity.\n  - Conclusion: circular tools can look best, even better than optimized combinations.\n- Pejaver acknowledges the same risk in its own text (\"validation and calibration data sets present potential sources of error\").\n- Its remedy was filtering out training variants and testing on post-2019 ClinVar entries.\n\n---\n\n## 2. Bot-generated mass content in encyclopedias\n\n**Lsjbot.** [secondary: Wikipedia pages, Vice 2020, ABC 2025; no peer-reviewed study found]\n- Created 454,000 articles by June 2013, about half of Swedish Wikipedia then (Wikimedia blog via the en-wiki page).\n- 2.7M articles by July 2014, about 8.5% of all Wikipedia (WSJ via the sv-wiki page). 9.5M by January 2019, about two-thirds Cebuano and a third Swedish. It created 80–99% of the articles in the Swedish, Waray and Cebuano editions.\n- Cebuano Wikipedia:\n  - Vice (Wilson, Feb 2020) cites Morandini's count: 5,331,028 of 5,378,570 articles were Lsjbot's, 99.12%.\n  - Over 24M of 29.5M edits were Lsjbot's (the unit is missing in the Vice text).\n  - All but five of the top 35 editors are bots; no human is in the top 10.\n  - En-wiki now lists about 6.1M articles and 257 active users. 95.8% of 1.21M articles (2015 Wikidata analysis) concern living organisms and 3.3% cities.\n  - By end of 2015, about 99% of its 1.4M articles were bot-created.\n  - The same page says articles \"are not maintained or improved often\" given the low ratio of editors to articles (citing ABC 2025).\n  - A 2018 proposal to delete the whole edition was rejected after strong opposition from Philippine Wikimedians.\n- Quality evidence is anecdotal:\n  - A sample of 1,000 random articles reviewed by Motherboard was \"surprisingly well constructed\", with no count given.\n  - Cebuano speakers reported grammatical and sometimes factual errors from imperfect translation.\n  - I found no measured error rate.\n- Waray: editors could not reach consensus on automatic creation, so Johansson stopped there.\n- Swedish Wikipedia:\n  - Lsjbot became inactive there in November 2016.\n  - Total article count peaked at nearly 3.8M in December 2017 and fell to about 2.5M by September 2022, linked on the sv-wiki page to a review and cleanup (an interpretation of the page, not a measured Lsjbot-only figure).\n  - Deletions were reportedly 20,000–40,000 a year from 2017, then accelerated after July 2020 to sometimes over 5,000 a day. A mirror of the en-wiki page also claims about 900,000 bot articles deleted since early 2017 and about 200,000 pending in December 2021. That mirror marks the claim \"citation needed\".\n  - Deletion grounds named: GeoNames data-quality problems and doubtful relevance (\"small hills\", \"small watercourses\").\n  - I could not find a single formal community vote with a result. The claim of \"a vote\" appears only in a summary.\n  - Lsjbot has been largely inactive since 2021; Johansson cited the debate as part of his reason for retiring it.\n- Did humans later expand them? A Meta-Wiki closure debate has a Cebuano editor saying place articles are being improved with Cebuano-speaking editors (a community claim). I found no systematic measure of human edits per bot stub.\n- A 2025 review (Reeves & Simperl, PACM HCI 9(7), arXiv 2509.22443) [part: text pulled but no hit on this question]: no study found on stubs attracting versus deterring later editors. Zheng et al. (CSCW 2019, \"The Roles Bots Play in Wikipedia\") classified 1,601 bots including \"Generator\" bots and examined newcomer survival; I did not see its generator-bot result [abstract only].\n\n**Rambot (English Wikipedia, October 2002).** [secondary: First Monday interview 2016 [part]; Wikipedia project pages]\n- About 30,000 US city/town articles were created in about a week (18–26 Oct 2002) from Census and CIA data, when the wiki had about 50,000 articles (+60%).\n- A data error corrupted around 2,000 new articles (Wikipedia:History of Wikipedia bots).\n- The flood caused the first bot policy.\n- Contemporary reactions were split. One editor said a page for every one-horse town would not project a good image. Another called the articles \"the foundation for the encyclopedia of the future. Use them. Improve them.\"\n- Fate: a Wikipedia growth-modelling page says \"most of the Rambot articles have since been substantially expanded\". It gives no counts, so treat it as an uncited assertion.\n- The English case differs from Cebuano in one respect: the English wiki had a large human community to expand the stubs.\n\n---\n\n## 3. Pre-filled answers anchoring human reviewers\n\n**Fort & Sagot 2010** (LAW IV, ACL; POS tagging on Penn Treebank Section 23) [full text]\n- Two annotators. Pre-annotators were MElt taggers with accuracies of 66.5%, 81.6%, 86.7%, 92.1%, 93.6%, 95.8%, 96.2% and 96.4%.\n- Time: with the 96.4% pre-annotation, ten sentences took about 2.5 minutes for both annotators. From scratch it took about 7.5 minutes (Annotator1) and 11.5 minutes (Annotator2).\n- Only the worst tagger (66.5%) was significantly slower. The gain flattens above roughly 81.6% pre-annotation accuracy.\n- Accuracy against the reference: no-pre-annotation baselines were 98.1–98.4% (A1) and 94.1–94.6% (A2).\n  - A2 gained: 94.1 → 97.1 with the best tagger, 94.6 → 95.2 with the 81.6% tagger.\n  - A1, the most accurate annotator, fell from 98.1 to 95.8 with the 81.6% tagger, with errors the authors call obvious lapses in concentration. With the best tagger A1 stayed stable (98.4 → 98.2).\n- The best pre-annotation shifted errors from random to systematic: A1 kept the tagger's systematic JJ-for-VBN confusion. Both annotators said they concentrated less with good pre-annotations.\n- Earlier work they cite (Marcus et al. 1993): manual tagging took twice as long as correcting pre-tags and had about 50% higher error and twice the disagreement. Dandapat et al. 2009 found untrained annotators more influenced than trained ones. Fort et al. 2009 reported anchoring in named-entity annotation: unmarked entities go unnoticed. Rehbein et al. 2009 found noisy pre-annotation did not overall corrupt human judgment.\n\n**Wang et al. CHI 2024, \"Human-LLM Collaborative Annotation\"** [part: excerpts]\n- SNLI human accuracy: 70.44% with no assistance, 72.07% with LLM labels, 74.15% with labels plus explanations.\n- No significant difference on the stance dataset.\n- When the LLM was wrong, showing its label hurt human accuracy. Manual annotation beat any assistance in almost all conditions, for annotators who saw a highly accurate LLM subset (over 80%) as well as a mostly inaccurate one (under 30%). The authors conclude people are easily misled \"regardless of LLM accuracy\".\n- LLM accuracy was 79.5% on SNLI and 66.3% on SemEval.\n\n**Schroeder, Roy & Kabbara, ACL Findings 2025** [part: full text]\n- 410 annotators, over 7,000 annotations, two models, two datasets, pre-registered.\n- Pre-filled labels did not make annotators faster, but raised their confidence.\n- Full 5-of-5 consensus rose from 8% (no assistance) to 38% (pre-highlighted label), p = .001.\n- Measured LLM F1 against crowd labels rose by an average of +0.32 (GPT-4) and about +0.35 (Llama), inflating reported model performance.\n- This is a subjective multi-label task, so \"errors\" are not defined here as in the others.\n\n**Beck, Eckman, Kern & Kreuter 2025, \"Bias in the Loop\"** (arXiv 2509.08514) [part]\n- 2,784 Prolific participants. The task was checking AI-suggested greenhouse-gas values in tables. The suggestions were hand-made (Wizard of Oz) and framed as AI. Each person saw 10 tables, and in the 7 randomized ones 4 were wrong and 3 correct.\n- Overall accuracy was 66–68%.\n- Requiring a correction value for each flagged error lowered corrections and raised acceptance of wrong suggestions (accuracy 68% vs 66% overall, p = .028, because fewer correct items were over-corrected).\n- Errors that were easy to see were caught more often: spelling errors in 82% of cases, wrong-cell errors in 77%. Errors needing conceptual understanding of the rules were caught only 31% of the time, which means about 69% were accepted. One technically correct item that looked wrong was caught at only 21%.\n- Whether the first three suggestions were all correct or all wrong made no meaningful difference to later behavior.\n- Attitude toward AI was the strongest predictor: AI-favorable annotators undercorrected more; AI-skeptical ones corrected more but also overcorrected more. Financial incentive had no consistent effect.\n\n**Clinical automation bias.**\n- Goddard, Roudsari & Wyatt 2012 (JAMIA 19:121), 74 studies [part]:\n  - Pooled risk ratio for following erroneous advice vs controls: 1.26 (95% CI 1.11–1.44), 4 healthcare studies.\n  - \"Negative consultations\" (a correct decision switched to an incorrect one after advice) ran 6–11% of cases.\n  - Decision support is typically 80–90% accurate.\n  - Mitigators: accountability, showing confidence levels, offering information rather than a recommendation, and where the advice sits on screen.\n- Goddard et al. 2014 (Int J Med Inform; abstract only): 26 GPs, 20 scenarios, advice wrong in 6. Doctors changed 22.5% of prescriptions. Accuracy rose from 50.38% to 58.27%. Automation bias, meaning a correct decision switched to an incorrect one, was 5.2% of all cases (net +8%). Less clinical experience went with more switching.\n- Dratsch et al. 2023 (Radiology; coverage only [secondary]): 27 radiologists read mammograms with suggested BI-RADS categories, and the AI suggestions were wrong for 12 of 40 test cases.\n  - Inexperienced readers' accuracy dropped from about 80% to under 20% (19.8%) on wrong suggestions.\n  - Moderately experienced readers scored 24.8% on wrong suggestions.\n  - Experienced readers (average over 15 years) dropped from 82% to 45.5%.\n  - Secondary outlets quote relative-decline percentages that do not match these raw figures, so quote the raw ones.\n\n**Does the effect depend on machine accuracy?** The evidence is mixed.\n- Fort & Sagot: a 66.5% pre-annotator cost time, but at 81.6% accuracy the time gain was already near the maximum. Quality effects depended on the annotator. A weak pre-annotator hurt the strongest annotator.\n- Wang et al.: wrong labels hurt humans whether the LLM subset was accurate or inaccurate.\n- Beck et al.: a visible run of early errors did not change later acceptance. Seeded-error acceptance depended mostly on the error type: 69% of conceptual errors accepted, 18–23% of surface errors.\n- Goddard's review: 80–90% accurate advice still produced a 6–11% rate of right-to-wrong switching.\n- I found no study giving a clean curve of \"fraction of seeded errors accepted vs machine accuracy\".\n\n---\n\n## What would most surprise a designer who assumes machine pre-answers are harmless if they weigh nothing\n\n1. **\"Weighs nothing\" does not protect the reviewer.** In Schroeder 2025, merely showing the label moved full agreement from 8% to 38%, and the people were meant to be judging independently. The prefill steers the human, who then makes the signal that does count.\n2. **A prefill does not need to be mostly wrong to harm.** Wang 2024: a wrong LLM label lowered human accuracy even for annotators who had been shown a highly accurate (over 80%) subset. Goddard: advice that is 80–90% accurate still produces 6–11% right-to-wrong switches.\n3. **Errors that need conceptual judgment are the ones accepted.** Beck 2025 caught only 31% of rule-based errors, against 82% of typos. Critical-question answers are exactly the conceptual kind.\n4. **The best annotators can get worse.** In Fort & Sagot the most accurate annotator lost 2.3 points with a mediocre pre-annotator, and shifted from random to systematic errors with a good one. Attention declined as the machine improved.\n5. **Many review designs gave no speed benefit.** Schroeder 2025: no speed gain from prefilled labels; Wang: wrong labels slowed people. The saving that justifies seeding may not appear.\n6. **Correlated tools are one witness, and the ceiling held up.** Clinical genetics capped in-silico tools at Supporting, told labs to commit to one tool before looking at results, and even then required a calibration set stripped of every tool's training variants and cut the number of variants that reach even Strong to about 1.4–1.7%. The same logic applies to several machine passes about one claim, and to checking machine output against data the same machine shaped.\n7. **Bot-seeded corpora become the corpus.** In Cebuano, 99.1% of articles and nearly all edits are one bot's, 257 active human users maintain 6.1M articles, and the content is \"not maintained or improved often\". On English Wikipedia (Rambot) the stubs were expanded, but that wiki had a large human community. Swedish Wikipedia deleted a large share after 2017; the total fell from about 3.8M to 2.5M by 2022. Cheap seeding may leave a maintenance debt that only a large human community can service.\n8. **No clean study of whether seeded stubs attract or deter human editors exists that I could find.** Do not assume the stubs attract contributions (\"not found\").\n\nSource URLs, as retrieved:\n- Pejaver preprint: https://www.biorxiv.org/content/10.1101/2022.03.17.484479.full.pdf (AJHG version: PMC9748256, PubMed 36413997)\n- Grimm 2015: https://pmc.ncbi.nlm.nih.gov/articles/PMC4409520/\n- Ghosh 2017: https://link.springer.com/article/10.1186/s13059-017-1353-5\n- Lsjbot: https://en.wikipedia.org/wiki/Lsjbot, https://sv.wikipedia.org/wiki/Lsjbot, https://en.wikipedia.org/wiki/Cebuano_Wikipedia, https://www.vice.com/en/article/the-worlds-second-largest-wikipedia-is-written-almost-entirely-by-one-bot/, https://www.abc.net.au/news/science/2025-04-15/wikipedia-cebuano-lsjbot-ai-article-generation-non-english/105123090\n- Rambot: https://firstmonday.org/ojs/index.php/fm/article/view/6027, https://en.wikipedia.org/wiki/Wikipedia:History_of_Wikipedia_bots, https://en.wikipedia.org/wiki/Wikipedia:Modelling_Wikipedia%27s_growth\n- Fort & Sagot: https://aclanthology.org/W10-1807/\n- Wang CHI 2024: https://dl.acm.org/doi/fullHtml/10.1145/3613904.3641960\n- Schroeder 2025: https://aclanthology.org/2025.findings-acl.1323.pdf\n- Beck 2025: https://arxiv.org/abs/2509.08514\n- Goddard 2012: https://pmc.ncbi.nlm.nih.gov/articles/PMC3240751/; Goddard 2014: https://pubmed.ncbi.nlm.nih.gov/24581700/\n- Dratsch 2023 coverage: https://www.rsna.org/news/2023/may/ai-bias-may-impair-accuracy\n"}