{"path":"research/the-literature-as-a-distributed-data-structure/report-tweet-and-ai-review-2026-11-04.md","content":"# Helper report, verbatim: Dan Roy's JMLR tweet and the measured state of AI peer review (2026-11-04)\n\n*A research helper on Claude Sonnet, briefed by the session for `the-literature-as-a-distributed-data-structure.md` § 4. Its report is kept here as it arrived, unedited. Every number carries the helper's own read-status tag; the session recomputed the ones it uses in `cases.py` and read whole the sources marked so in the note. The helper's saved pages were in its scratchpad and are not committed; the URLs at the end are the record.*\n\n---\n\n# Research report: Dan Roy's JMLR tweet and the measured state of AI peer review (as of 2026-11-04)\n\nTags: [full text] I read the primary text. [part] I read part of it or an extracted excerpt. [abstract only]. [secondary] a report of the source. Saved pages are in the helper's scratchpad directory. File names are in the source list at the end. Six saved files are stubs or empty (X pages, Science, the Medium post, Zancanaro, Bhargava, New Scientist). Their retrieved text is in `_exa-and-browser-excerpts.txt` in that directory.\n\n## Short version\n- Roy was a JMLR Action Editor, not editor-in-chief. The asterisk means he \"actually resigned ~2 years ago\" and kept handling resubmissions. I found no longer statement from him.\n- The newest evidence supports a narrower claim than the tweet makes. AI reviews beat human reviews as feedback to authors, but that is measured on helpfulness, not on decisions or ground truth. AI reviewers also miss many inserted errors, are gameable, and homogenize.\n- Human accept/reject is measurably noisy. NeurIPS 2014 and 2021: about 23–26% of decisions flipped between two independent committees.\n- TMLR already unbundles soundness from significance and runs AI soundness reviews. It still ends in one accept/reject per submission.\n- Nothing I found describes the \"distributed data structure\". The one reply I could retrieve asks \"But what is the solution?\".\n\n---\n\n## PART A. The tweet in context\n\n### A1. Who Roy is, what the asterisk means, and reactions\n- **Roy.** Professor of Statistical Sciences at the University of Toronto with a cross-appointment in Computer Science, CIFAR Canada AI Chair, Vector Institute founding faculty. He stepped down as Vector Research Director on 31 Dec 2025 and is on leave as a Visiting Researcher at Google DeepMind in London. His profile says \"Views are my own\". His own page lists \"Action Editor, Journal of Machine Learning Research\". [full text: danroy.org]\n- **JMLR role.** Action Editor. He is listed in Wayback snapshots of the JMLR board page from 11 Jan 2024 and 27 Dec 2024. He is absent from the live board page I fetched today, and from the 2019 snapshot. A grep for \"Roy\" on the live page finds only Roy Frostig. His own page is therefore stale, which fits \"resigned ~2 years ago\". I did not find the start date of his term. [full text]\n- **TMLR.** His page source contains a commented-out line \"Action Editor, Transactions of Machine Learning Research\". He is not on the TMLR board page I fetched. I found no evidence he served at TMLR. [full text, raw HTML]\n- **Verbatim tweet (2 Nov 2026).** \"Just fully* resigned from JMLR. I still believe in science, but journals needs a complete rethink. AI reviews by top models are now much better than time-pressed human ones. OTOH, one-time gate keeping (accept / reject) doesn't make sense. The literature is going to be moving faster than ever. We need to think about it as a distributed data structure, and start over.\"\n- **Timing.** The tweet is 13:24 UTC. The mirror shows 10:24 in UTC−3. [full text, mirror and AGI Hunt timestamp]\n- **Asterisk follow-up, three minutes later.** \"*Fully because I actually resigned ~2 years ago, and have been dealing with resubmissions.\" [full text, mirror and Digg]\n- **Engagement.** Built-in browser, logged out, 2026-11-04: 27 replies, 53 reposts, 527 likes, 61K views. [part]\n- **Other posts the same day or next.**\n  - Reply to @jason_of_cs, 2 Nov: \"Context matters. My chatbots have seen dozens of my reviews, etc., and I've given them a lot of guidelines in the past, and so when I ask them to assess work, it's always pretty good.\" The parent post was not retrieved, so the link to the resignation is inferred.\n  - Reply, 3 Nov, parent not shown: \"At 1 paper a month, I don't think quality would suffer… 1 paper a day, for sure, you would be getting a lot of junk. But more, who is consuming this? That's the real question.\" [full text, mirror]\n- **Reactions found.** Only @ziv_ravid, replying: \"But what is the solution?\" [Digg]. I found no reply from Roy to it. Other reactions: not found.\n- **Longer statement, blog or thread from Roy (Oct–Nov 2026):** not found.\n- **Context for a careful reader.** Roy works at Google DeepMind. On 1 Oct he posted that he contributed to Gemini 4 Argon's math capabilities. Google's Paper Assistant Tool is used by authors at NeurIPS 2026, and ICML 2026 piloted it. [full text, NeurIPS blog; part, X]\n\n### A2. JMLR and the 2001 resignation\n- **Date of founding.** JMLR was founded in 2000 by Leslie Kaelbling. The web site went live on 23 Jun 2000, and the announcement went out on 8 Aug 2000. The 2001 resignation therefore came a year after JMLR existed, as an endorsement of it. It was not the origin. [full text: jmlr.org/history]\n- **Model.** Reviews were promised within six weeks, and final versions were published electronically \"immediately upon receipt\". There was an annual MIT Press print volume until the agreement ended at the end of 2004. Content is free \"without limits and without conditions\", and authors keep copyright (CC-BY from 2017). The 2000 announcement was built on JAIR's example. [full text]\n- **The letter.** Michael Jordan circulated a resignation letter on 8 Oct 2001 from 40 named editors of *Machine Learning* (Kluwer). Signers included Bengio, Bishop, Dietterich, Ghahramani, Hinton, Jaakkola, Koller, Kaelbling, Mitchell, Russell, Schölkopf, Sutton and Thrun. Infotoday: two-thirds of the board. A Boston Globe piece says 39. The letter cites the 1999 resignation of the editors of the journal *Logic Programming*, who moved to *Theory and Practice of Logic Programming*. [full text]\n- **Rationale.** The letter's concern is access: \"immediate and universal access… at a cost that excludes no one\". MLJ charged $1,050 a year to institutions and $120 to individuals. [full text]\n- **Wording of the letter.** It does not say reviewing needed redesign. Roy's tweet is a different kind of resignation, about the review and decision model.\n- **Hinton and TMLR.** JMLR's history says Hinton once proposed the name \"Transactions in Machine Learning Research\" for a different idea. [full text]\n\n### A3. TMLR\n- **Founders.** Raia Hadsell, Kyunghyun Cho and Hugo Larochelle. Announced 12 Dec 2021 (Larochelle's Medium post). Open to submissions 24 Mar 2022. Hosted on OpenReview. [part: Medium via Exa; full text: launch post]\n- **Model.**\n  - Rolling submission with a per-paper timeline. Double-blind.\n  - The announcement promised reviews in 4 weeks and a decision in 2 months.\n  - All submissions stay public on OpenReview, including rejected and retracted ones. Only desk rejects are hidden.\n  - Decisions are Accept, Accept with minor revision, or Reject. A rejected paper can return only as a new submission linked to the old one.\n  - Acceptance rests on two questions. Are the claims supported by accurate evidence? Would some of TMLR's audience be interested? Significance and novelty are explicitly excluded. [full text: editorial policies, acceptance criteria]\n- **Certifications.** Larochelle's announcement describes them as \"aimed at separating editorial statements on submitted work from their claim-based scientific assessment\". They are Outstanding, Featured, Reproducibility, Survey, Expert and, since Oct 2025, Journal-to-Conference. [part]\n- **2026 changes.**\n  - Submissions tripled in a year, with single-author submissions up about 13-fold.\n  - Desk rejection rose from 6% (2023) to about 53%.\n  - Submission quotas are in place. 159 authors received one-year suspensions.\n  - Each submission receives an AI soundness review from October 2026. The AE decides, the EiCs approve, and authors may respond. A July post had said August. [full text]\n- **The AI-reviewer evaluation (CMU: Rao and Shah).**\n  - Table: share of papers where the reviewer said \"claims supported\" (a \"yes\" counts as a miss on the error rows).\n\n| | Accepted papers (higher is better) | AE-rejected on claims only (lower is better) | AE-rejected on both criteria (lower is better) | Inserted errors (Gemini-inserted / OpenAI-inserted; lower is better) |\n|---|---|---|---|---|\n| CSPaper (chosen) | 92.7% (n=41) | 9.1% (n=44) | 0.0% (n=46) | 82.1% / 70.0% |\n| GPT-5.5 | 82.9% | 65.9% | 32.6% | 76.1% / 80.0% |\n| Opus 4.7 | 82.9% | 34.1% | 23.9% | 86.6% / 94.0% |\n\n  - The authors write that \"none of the systems is a strong detector of the inserted errors\". [full text]\n  - Caveats: the vendor's prompts were calibrated on TMLR data, and the last author is a TMLR co-EiC.\n- **Roy and TMLR.** No involvement found (see A1).\n- **What TMLR already unbundles.** Rolling review, a soundness-only gate with significance moved into certifications, a public review record, and a separate AI soundness check.\n- **What it does not.** I found no statement that a decision is revisited after publication, and no versioned or annotated paper. Each submission still ends in one accept/reject. A rejected paper can only be resubmitted as a new submission. I did not check for a post-acceptance annotation mechanism.\n\n### A4. Roy's earlier statements\n- **Bluesky, 23 Nov 2024:** \"Reviewing is dead. I just read some ICLR reviews. Absolutely junk… Only 5% (if that!) of the community has worthwhile opinions.\" Later that day: \"No one gets tenure by reviewing well… Once we fix _that_, we will get useful peer review.\" Kunal Talwar replied that in small communities most reviews are reasonable and that 5% \"is perhaps more a sign of paper matching failure\". [full text, mirrored] The date is close to his \"~2 years ago\" resignation. No source links the two.\n- **31 Jul 2026,** on ICLR 2027's 20-paper cap: a satirical \"petition\" saying the cap \"will slow down AI progress… especially since all reviewing is now handled by LLMs (…admit it)\". [secondary: 36Kr and Marsbit; mirror]\n- **His own page,** \"marginalia\": \"I believe that errata, clarifications, missed citations, links to follow-on work, retractions, and other 'marginalia' are important, but underappreciated, contributions to the scientific literature. Ultimately, the nature of a scientific document needs to be rethought, but until then I am slowly collecting marginalia in a simple wiki.\" This is the closest thing to a precursor of the \"distributed data structure\" remark. The wiki link returned a 404. [full text]\n\n---\n\n## PART B. Measured state of AI peer review\n\n### B1. Liang et al. (GPT-4 feedback; NEJM AI per your cite). I read arXiv v1, Oct 2023. [full text]\n- **Overlap of GPT-4 with human reviewers:** 30.85% for Nature-family journals (3,096 papers) and 39.23% for ICLR (1,709 papers).\n- **Overlap between two human reviewers:** 28.58% and 35.25%.\n- **Comments raised by at least one human:** 57.55% of GPT-4's comments (Nature) and 77.18% (ICLR).\n- **ICLR by outcome, body text:** GPT-4–human overlap 30.63% for orals, 32.12% for spotlights, 47.09% for rejected papers. Human–human overlap 23.54%, 24.52% and 43.80%. The abstract cites 43.80% for GPT-4–human overlap on rejected papers. The body text gives that figure as human–human.\n- **User study:** 308 researchers from 110 institutions. 57.4% found the feedback helpful or very helpful, and 82.4% found it more beneficial than at least some human reviewers.\n- **Limitations stated:** GPT-4 \"struggles to provide in-depth critique of method design\".\n\n### B2. ICLR 2025 Review Feedback Agent (Thakkar et al.; Nature Machine Intelligence 2026 per Hartline [secondary]). [full text, arXiv v1]\n- **Design:** 44,831 reviews on 11,553 papers. 22,467 were randomized to feedback and 22,364 to control. 18,946 reviews (42.3% of all) received feedback, 15 Oct–12 Nov 2024. The agent gave feedback only on vague comments, misunderstandings or unprofessional remarks.\n- **Update rates:** 26.6% of reviewers who received feedback updated their review, about 17 percentage points above control.\n- **Incorporation:** 12,222 suggestions were incorporated. 17.7% of 69,836 items given were incorporated. 89% of updaters incorporated at least one item.\n- **Effects:** Updated reviews gained about 80 words. Two blinded researchers preferred the modified review in 89% of 100 examples. Rebuttals were 6% longer.\n- **Score edits:** 31.7% of reviewers who received feedback edited scores, against 30.6% of those who did not.\n\n### B3. AAAI-26 AI review pilot (Biswas et al., arXiv 2604.13940). [full text]\n- **Scale:** Every one of the 22,977 full-review main-track papers got one labelled AI review in under 24 hours at under $1 per paper (OpenAI in-kind credits). No scores or recommendations were given, and no human reviewer was replaced.\n- **Survey:** 5,834 voluntary responses, with self-selection acknowledged. AI reviews were rated higher than human on 6 of 9 criteria.\n  - Technical errors identified: +0.67\n  - New points raised: +0.61\n  - Presentation suggestions: +0.54\n  - Research-design suggestions: +0.49\n  - Thoroughness: +0.48\n  - Against: more \"overemphasized minor issues\" (−0.38), more technical errors (−0.22), more wrong suggestions (−0.11). Effects were larger for authors than for program committee members.\n- **Usefulness:** 53.9% called the AI reviews useful and 20.2% did not. Among reviewers, 46.6% said the AI found concerns a human would struggle to catch, and 49.4% said it overlooked points a human would catch. Only 13.8% said it changed their evaluation.\n- **Top negative themes:** weak big-picture judgment (9.1%), nitpicking (8.5%), verbosity (8.3%).\n- **Benchmark:** On the authors' SPECS synthetic-perturbation benchmark, the system beat a single-prompt baseline by an average recall gain of +0.19. It was not compared to human reviewers.\n\n### B4. Agents4Science 2025 (Stanford, 22 Oct 2025)\n- **Design:** 315 submissions (48 accepted, 196 rejected, 10 withdrawn, 61 desk-rejected), with an AI as first author. Three LLM reviewers scored 1–6: GPT-5 (mean 2.30), Gemini 2.5 Pro (4.23), Claude Sonnet 4 (3.0). 80 papers above a threshold went to human review. [part: Science news, organizers' LinkedIn, a third-party audit]\n- **Reviewer behaviour:** Zou reports GPT-5 most negative, Gemini most positive and sometimes sycophantic (\"groundbreaking\"), and strong catching of technical inconsistencies. [secondary]\n- **Reviewer agreement:** All three gave the same score on 19 of 250 papers (7.6%), and 123 of 250 had a spread of 3 or more points. [secondary, one analyst's repo]\n- **Fabricated references:** The organizers' automated check flagged 56% of submissions. A third-party audit of 241 reviewed papers found a wholly invented reference in 23.7% and a fabricated reference in 37.8%. An LLM review asserted fabricated references in only 4 of 91 affected papers, and no human review did. [secondary: audit preprint, not independently verified]\n- **Human reviewers:** Risa Wechsler: papers \"technically correct but… neither interesting nor important\". [part]\n- **Related:** BadScientist, an accepted paper there (see B7).\n\n### B5. 2026 venue policies and pilots\n- **ICLR 2026:**\n  - Reviews were 76,139 by 18,054 reviewers. 19,525 valid submissions. 779 desk-rejected and 5,042 withdrawn. 13,763 papers decided. [full text]\n  - ICLR ran two LLM detectors and flagged reviews that both labelled fully LLM-written, passing the flags to area chairs. Papers with hallucinated references were desk-rejected. [full text]\n  - Pangram (vendor): 21% of reviews (15,899) fully AI-generated, and more than half had some AI involvement. Pangram says higher AI involvement goes with higher scores. [full text; vendor result]\n- **ICLR 2027:**\n  - A 20-paper cap per author and a one-paper limit for new authors. [full text]\n  - Reviewers may use AI with mandatory disclosure, including their original self-written assessment. No official AI reviewer. The area chair guide states that LLMs are not used for consensus decisions. [full text]\n- **ICML 2026:**\n  - Two reviewer policies: A (no LLMs) and B (limited LLM help). [full text]\n  - Watermarking by hidden PDF prompts flagged 795 Policy-A reviews (~1% of all reviews) by 506 reviewers. 497 papers (398 reciprocal reviewers) were desk-rejected. Every flag was checked by a human. The post says the method is easy to circumvent. [full text]\n  - Randomized experiment (24,661 papers, 17,886 reviewers):\n    - Policy assignment had near-zero effects on decisions (accepted 27.0% vs 26.5%) and on scores (3.31 vs 3.32).\n    - Reviews were 5.5–7% longer under the permissive policy.\n    - In the survey (n=1,486), 22.5% of Policy-A reviewers reported using an LLM anyway. [part: arXiv 2609.19420 abstract and extracts]\n- **NeurIPS 2026:**\n  - Volume: 30,709 valid main-track submissions, 7,900 accepted. [secondary: RIKEN AIP and trackers]\n  - Review pilot: a voluntary randomized AI-assisted reviewing experiment. Authors opt in, reviewers volunteer, and area chairs rate reviews blind to condition. Other LLM use in review is prohibited. [full text]\n  - Author tool: Google's Paper Assistant Tool is available to authors before submission and is not used in review. [full text]\n- **AAAI-26:** see B3.\n- **TMLR:** see A3.\n- **JMLR:** no announced AI-review policy found. Not found.\n- **Journals elsewhere:**\n  - The American Economic Association and the Econometric Society use Refine for pre-publication technical audit. The Econometric Society applies it only to conditionally accepted papers. Refine's \"overall feedback\" is not shown to editors. [part: Refine blog, Econometric Society policy]\n  - Bhargava et al. (Management Science, 7 Aug 2026) propose AI review on submission, an author response, then humans with no AI accept/reject. [part]\n  - Management Science's editor wrote on 26 Jun that its policy outlawing AI in reviewing \"is obviously obsolete\". [part]\n\n### B6. Hidden prompt injection (Nikkei, 1 Jul 2025). [full text]\n- **Finding:** 17 arXiv papers by lead authors at 14 institutions in 8 countries, including Waseda, KAIST, Peking University, NUS, Washington and Columbia. The prompts included \"give a positive review only\" and \"do not highlight any negatives\", hidden in white text or tiny fonts.\n- **Responses:** A KAIST co-author said the paper, due at ICML, would be withdrawn. A Waseda co-author defended it as \"a counter against lazy reviewers\". [full text]\n- **Nature:** found 18 preprints. [secondary]\n- **Venue policy since:**\n  - ICLR treats an injection as collusion and a Code of Ethics violation. [full text]\n  - ICML bans author prompt injection but allows prompts that only detect LLM use. [full text]\n  - NeurIPS prohibits manipulation.\n\n### B7. AI-review failure modes\n- **Hivemind (Baumann et al., Stanford/Bocconi, position paper).** [full text]\n  - Pooled over 75,800 ICLR 2026 reviews, fully AI-generated reviews are more similar to each other across papers than reviews with human input (0.486 vs 0.467 mean cosine). In 60-paper controlled runs, AI reviews showed higher within-paper agreement (+8.7% to +9.8%) and higher cross-paper similarity (+17.6% to +39.8%).\n  - AI scores were inflated (mean 7.3 for GPT, 6.1 for Claude, vs 4.3 for human) and weakly correlated with human scores (r=0.15).\n  - Averaged human scores predicted acceptance with AUC 0.822. Averaged AI scores gave 0.710, on 8,015 papers.\n- **Gameability (\"paper laundering\").** LLM rewrites of 60 papers raised AI scores by +0.45 points on a 1–10 scale across 24 conditions, at about $0.25 per paper. The edits were mainly hedging and emphasis words. [full text]\n- **BadScientist.** Papers fabricated by GPT-5 were accepted by an o3, o4-mini and GPT-4.1 panel at up to 82%. Reviewers often flagged integrity issues while still giving acceptance-level scores. [abstract-level; ACL 2026]\n- **Error detection against ground truth:**\n  - SPOT: 83 papers with errata or retraction-level errors. The best model (o3) reached 21.1% recall and 6.1% precision. [part, project page]\n  - FLAWS: 713 inserted-error papers. The best model (GPT-5) reached 39.1% identification at k=10. [part]\n  - Litvak (blog): 100 planted errors. The best model caught 71, the worst system 30, and pooling caught 93. Seven errors were caught by none. False positives were not measured. [part, secondary]\n  - TMLR evaluation: see A3.\n- **Zou et al. GPT-5 checker on 2,500 published papers.** [full text]\n  - 4.66 flagged mistakes per paper. 99.2% of papers had at least one. The mean for NeurIPS rose from 3.8 (2021) to 5.9 (2025).\n  - Precision was 83.2%, with 263 of 316 flags confirmed in 60 papers selected for already having a flagged substantive mistake. Recall on 90 injected mistakes was 60.0%.\n- **Black Spatula / YesNoError.**\n  - Nick Brown found 14 false positives among 40 YesNoError flags. [secondary: Nature via HN, Pivot to AI]\n  - Black Spatula has no published quantitative benchmark. [secondary]\n  - A count of real errors found: not found.\n- **Length bias.** King & Worthington report the AI win persisted on length-matched pairs and when human reviews were rewritten in AI style. [part]\n\n### B8. Human review baseline\n- **NeurIPS 2014 experiment:** 166 duplicated papers, 43 disagreements (25.9%). Random baseline 37.5%. Of papers the first committee accepted, 49.5% were rejected by the second (n=116). [full text, NeurIPS 2021 paper]\n- **NeurIPS 2021 experiment:** 882 duplicated papers, 203 inconsistent (23.0%). Random baseline 35.1%. 50.6% of accepted papers would be rejected by the other committee. 14.9% of rejected papers would be accepted. [full text]\n  - If borderline cases are not counted as inconsistent, 16.0%.\n  - Pearson correlation between committees' average scores: 0.575 initial, 0.586 final.\n  - 48% of area-chair survey answers called the paper borderline with \"no real disagreement\".\n  - More selective cutoffs raise arbitrariness: accepting only orals and spotlights gave 88.8% of accepted papers rejected by the other committee.\n- **Volume:**\n  - NeurIPS main track 9,467 (2020), 21,575 (2025), 30,709 (2026). 2025 had 20,518 reviewers and the chairs said the process was \"noisier\". 11 papers were reversed because a co-author was grossly negligent as a reviewer. [full text]\n  - ICLR 2025: 11,603 submissions. ICLR 2026: 19,525.\n  - AAAI-26: 22,977 full-review papers.\n  - ICLR 2026 reviews per reviewer: 76,139 ÷ 18,054 ≈ 4.2 (my arithmetic).\n- **Time per review:** a median of 5 hours is quoted in the Publons 2018 report, which ties it to a 2016 estimate. [part] A software-engineering survey found 88% of journal reviewers spend over 2 hours. [part] An ML-conference time-per-review figure: not found.\n- **Reviewer load comparison:** Rao and Shah write that conferences require 3–6 papers in a short duration, while TMLR assigns one at a time.\n\n### B9. Sakana AI Scientist-v2 (March 2025)\n- **What happened:** Three fully AI-generated manuscripts were submitted to the ICLR 2025 ICBINB workshop, among 43 submissions. Reviewers knew some might be AI-generated, but not which. One paper scored 6, 7, 6 (mean 6.33, roughly the top 45%). The other two fell short. [full text]\n- **Withdrawal:** Sakana withdrew the paper after review, by prior agreement. No meta-review was done.\n- **Caveats stated by Sakana:**\n  - Workshop acceptance rates run 60–80%, against 20–30% at main conferences.\n  - Sakana's internal assessment found none of the three met the main-conference bar.\n  - The system makes citation errors. Sakana also lists dataset-overlap issues and inaccuracies in figure captions.\n  - The authors selected the 3 papers to submit.\n\n---\n\n## PART C. What the sources support, per claim (no opinion)\n\n**1. \"Journals need a complete rethink.\"**\n- *Supports:*\n  - Submission volume: NeurIPS grew 9,467→30,709 (2020→2026), and TMLR tripled in a year.\n  - Desk rejection at TMLR rose from 6% to 53%.\n  - The chairs call review \"noisier\", and a fifth of ICLR 2026 reviews were fully AI-generated (Pangram, vendor).\n  - Policy is moving: AI-first workflows are proposed (Bhargava), AI soundness review is deployed (TMLR), and AAAI and NeurIPS run field experiments.\n  - A Management Science editor calls its ban on AI reviewing obsolete.\n- *Contradicts or qualifies:*\n  - Most numeric evidence is from conferences, not journals.\n  - The responses so far are mostly quotas, detection and pilots, not structural redesign.\n  - ICML's randomized policy test found near-zero effects on decisions.\n- *Unknown:* JMLR's own volume or policy for 2026. Not found.\n\n**2. \"AI reviews by top models are now much better than time-pressed human ones.\"**\n- *Supports:*\n  - King & Worthington (working paper, 22 Oct 2026, not peer-reviewed): 134 AI/human pairs on 124 papers, Claude Opus 4.8 against paid, well-run human reviews.\n    - The GPT-5.5 judge chose AI in 132 of 132 decided pairs. The text says 132 of 132. A figure in the paper labels it 134 of 134 (noted by another source).\n    - Three former editors chose AI in 26 of 30 judgments.\n    - Incorrect claims about the manuscript: 5.5% (AI) vs 15.9% (human). No fabricated sources among 651 checked.\n  - AAAI-26 (6 of 9 criteria), Zancanaro (n=40, one cardiology journal, concordance with final decision 67.5% vs 71.9%, and AI higher on 5 of 7 domains), and Liang (overlap parity with a second human).\n- *Contradicts or qualifies:*\n  - The strongest comparison is against non-time-pressed humans, which the authors say understates AI's edge.\n  - Its limits: the editors did not read the manuscripts, the judge is another AI, the scope is social science, and it measures helpfulness to authors. The authors say it does not address decision-making.\n  - AAAI's respondents also rated AI worse on nitpicking and errors, and only 13.8% of reviewers changed their evaluation.\n  - A survey of authors who received both found human reviews rated higher on scientific understanding and insight, with AI on par for clarity and politeness. [abstract-level]\n  - A 119-article BMJ Open comparison found ChatGPT raised fewer and less varied substantive points. [secondary, via King]\n  - Hivemind, gameability and low error recall (SPOT 21.1%, FLAWS 39.1%, TMLR evaluation 70–94% miss on inserted errors) all weigh against it.\n  - Averaged AI scores predicted acceptance worse than human scores (0.710 vs 0.822).\n- *Unknown:* A head-to-head of top-model reviews against time-pressed conference reviews on ground truth. Whether AI reviews improve decisions or published quality. How these results hold once authors adapt to the AI reviewer.\n\n**3. \"One-time gatekeeping (accept/reject) doesn't make sense.\"**\n- *Supports:* the NeurIPS 2014 and 2021 consistency results, about a quarter of decisions flipping and about half of accepted papers swapped. TMLR is built around a soundness gate with significance moved into certifications.\n- *Contradicts or qualifies:*\n  - The experiments show noise, not uselessness. Committees' scores correlate at 0.575, and the 2021 result is a 35% improvement over random.\n  - No retrieved evidence that a non-gated model performs better.\n  - TMLR still gates once per submission.\n- *Unknown:* Whether continuous annotation instead of a gate changes reader behaviour.\n\n**4. \"The literature as a distributed data structure.\"**\n- *Supports (indirectly):*\n  - Roy's marginalia statement says documents \"need to be rethought\".\n  - Zou et al.'s average of 4.66 flagged mistakes per paper implies a large post-publication corrections stream.\n  - TMLR keeps the full review record public.\n  - Refine-style audits separate technical verification from acceptance.\n- *Contradicts:* Nothing directly.\n- *Unknown:* Roy gave no specification, and no retrieved source tests such a design.\n\n## Not found\n- Reactions to the tweet beyond @ziv_ravid.\n- A Roy follow-up post or blog on the proposal.\n- Roy's JMLR start date and exact resignation date.\n- Any JMLR announcement on AI review.\n- A real-error count from Black Spatula.\n- An ML-specific time-per-review figure.\n- The JMLR editorial-board change date.\n\n## Sources (local files in the helper's scratchpad)\nA1–A4\n- danroy.org: `danroy-org.txt`\n- Roy post mirror: `dailyjournal-roy.txt`\n- Digg: `digg-roy.txt`\n- X views: `_exa-and-browser-excerpts.txt`\n- JMLR board snapshots: `jmlr-board-wayback-2024-01.txt`, `jmlr-board-wayback-2024-12.txt`, `jmlr-editorial-board.txt`\n- JMLR history: `jmlr-history.txt`\n- MLJ resignation letter: `mit-jmlr-statement.txt`, `sigir-ml-resignation.txt`, `infotoday-forty-resign.txt`\n- TMLR: `tmlr-launch-2022.txt`, `tmlr-ai-reviews-soundness-blog.txt`, `tmlr-blog-2026-10-15.txt`, `tmlr-annual-quota.txt`, `tmlr-editorial-board.txt`, `cspaper-tmlr-eval-rao-shah-2026.pdf`/`.txt`\n- Roy Bluesky: `_exa-and-browser-excerpts.txt`\n\nB\n- Liang: `liang-2023-llm-feedback.pdf`/`.txt`\n- Thakkar: `thakkar-2025-iclr-feedback-agent.pdf`/`.txt`\n- AAAI-26: `aaai26-pilot-biswas-2604.13940.pdf`/`.txt`\n- Agents4Science: `bianchi-agents4science-2511.15534.pdf`, `_exa-and-browser-excerpts.txt`\n- ICLR: `iclr-2026-retrospective.txt`, `iclr-2026-response-llm.txt`, `iclr-2027-submission-policies.txt`, `iclr-2027-reviewer-ai-policy.txt`, `pangram-iclr-21pct.txt`\n- ICML: `icml-2026-violations-blog.txt`, `icml-2026-llm-policy.txt`, `icml-2026-llm-rct-2609.19420.pdf`\n- NeurIPS: `neurips-2026-ai-reviewing-experiment.txt`, `neurips-2026-pat-blog.txt`, `neurips-2025-reflections.txt`\n- Nikkei: `nikkei-hidden-prompts.txt`, `the-register-hidden-prompts.txt`, `the-guardian-hidden-prompts.txt`\n- Baumann: `baumann-2026-stop-automating.pdf`/`.txt`\n- Zou checker: `zou-to-err-is-human-2512.05925.pdf`/`.txt`\n- King & Worthington: `king-worthington-2026-peerless.pdf`/`.txt`, `the-times-higher-ed-king.txt`\n- NeurIPS consistency: `neurips-2021-consistency-blog.txt`, `beygelzimer-2023-neurips-consistency.pdf`\n- Sakana: `sakana-ai-scientist-first-pub.txt`, `sakana-ai-scientist-v2-paper.pdf`\n- Refine: `refine-aea-econometric.txt`\n- Bhargava, Zancanaro, BadScientist, SPOT, Litvak: excerpts in `_exa-and-browser-excerpts.txt` only\n\nPrimary URLs: danroy.org; jmlr.org/history.html; jmlr.org/statement.html; jmlr.org/tmlr/; blog.tmlr.org (3 posts); arxiv.org/abs/2310.01783, 2504.09737, 2604.13940, 2605.03202, 2512.05925, 2609.19420, 2306.03262, 2511.15534; gking.harvard.edu/files/peerless.pdf; blog.icml.cc/2026/03/18/on-violations-of-llm-review-policies/; neurips.cc/Conferences/2026/ai-reviewing-experiment; blog.iclr.cc (2025/11/19, 2026/03/31, 2026/09/02); asia.nikkei.com (Positive review only); sakana.ai/ai-scientist-first-publication/; blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/; pubsonline.informs.org/doi/full/10.1287/mnsc.2026.00184; doi.org/10.1093/ehjimp/qyag097.\n"}