{"path":"research/what-a-link-rests-on/outside-reading-machines-naming-premises.md","content":"# Outside reading: how well machines name the missing premise, and what happens when a link is presupposed\n\n*A research helper's report, kept as it came back on 2026-10-07 and not edited. The question put to it: measured results on machine-written warrants, on a model justifying a relation it is told exists, on step-by-step proof structure, and on how well people agree about relevance.\nIt is a model's reading of the sources it lists, with its own statement of how far each was read;\nthe note that uses it is [what-a-link-rests-on.md](../what-a-link-rests-on.md). **Verify against\nthe sources before citing any of it outward.** Sections the helper marks as its own inference are\nits own, not this project's position.*\n\n---\n\n# Evidence report: LLM-named warrants, \"justify the presupposed link\" failures, step validity, and human agreement on relevance/sufficiency\n\nResearch date 2026-10-07. Raw text was read through Exa fetch/search (PDFs, ACL Anthology, arXiv). A summarising fetch tool (WebFetch, run on ar5iv pages) was used for some tables. Anything that came only from that tool is tagged [FT] and should be treated as unverified. I caught one [FT] error: it mis-aggregated the MiP-Overthinking abstain rates, which I corrected against the paper's own table text. So treat other [FT] numbers with caution.\n\nQuotes: my configuration allows one short verbatim quote per report. I give exact numbers plus locators (table/section) instead of per-source quotes. The single quote is under [40]. Statements marked INFERENCE are mine and appear only in Section 6.\n\n---\n\n## 1. Headline numbers (pointers in brackets)\n\n- **Humans filling a gap between two claims with no constraint write about 2.7 premises** (annotator means 2.0 to 3.6). Their premise sets overlap in about 32% of words, and one annotator abstained on 14.5% of pairs versus 1.2% for another [1].\n- **LLM-written warrants beat the gold ones in one rater study.** Two graduate-level raters accepted 61.7% of GPT-4 warrants versus 45.7% of gold warrants. Cohen's kappa was only .18 to .31 [9].\n- **Binary warrant choice on ARCT** (original test set, GPT-4o-mini zero-shot): single agent .793, debate with self-chosen sides .869. Agents forced to defend an assigned premise fell to .698 to .721, below the single agent [10]. Human average is .798, or .909 with reasoning training [2].\n- **LLM-written argument reconstructions are often not even deductively valid.** Validity was 40.8% to 80.8% for prompting-only baselines, including Claude Sonnet 4.5 at 59.2% to 60.8% and GPT-5.2 at 51.7% to 80.8% [11].\n- **Post-hoc justification is measured.** Biasing prompts cut accuracy by up to 36 points while 425 of 426 explanations never mentioned the bias [20]. An end-to-end entailment-tree generator \"proved\" 100% of goals, including deliberately invalid ones [39].\n- **Verifiers lean toward \"correct\".** The \"Are you sure?\" challenge cuts accuracy by up to 27 points [21]. Step verifiers under-detect errors, and verifying each step under its premises only gained 6 to 16 points of error-identification accuracy [40].\n- **Spontaneous flaw detection is low, but prompted detection is high.** GPT-4o proactively flagged flawed premises 11.0% of the time; strong models flagged them 73% to 78% of the time when explicitly told to check [32].\n- **Reasoning models notice missing premises but do not act on it.** On MiP-GSM8K, DeepSeek-R1 abstained 16.5% and QwQ 10.1%, versus GPT-4o 46.9%, although 95.5% to 100% voiced suspicion early [31].\n- **Human agreement on relevance/sufficiency is mostly low.** Krippendorff's alpha was .27 to .47 for graded expert ratings [48][49]. It reached .77 only for binary sufficiency labels from annotators trained on a guideline with disagreement discussion [46]. Single-step entailment validity reached kappa .72 [39].\n\n---\n\n## 2. Q1: how well do models name the implicit premise or warrant\n\n**[1] Boltužić & Šnajder (2016). Fill the Gap! Analyzing Implicit Premises between Claims from Online Debates. ArgMining 2016, pp. 124-133.**\n- URL opened: https://aclanthology.org/W16-2815.pdf\n- READ DEPTH: full text read through §5.4. The conclusion was read via a search highlight.\n- Data: 500 gap-filling premise sets (125 claim pairs per topic × 4 topics). Three annotators had no instructions beyond \"bridge the gap\", and abstention was allowed.\n- Gap size: mean 2.7 ± 0.7 premises per gap (annotators 3.6 / 2.6 / 2.0) and about 23 words per gap. Abstention averaged 6.4% (1.2% / 3.6% / 14.5%). Word overlap between annotators' premise sets was about 32%.\n- Premise typology (two authors, 50 premises, kappa .42 / .62 / .53): 85% factual, 9% value, 6% policy. 77% were atomic; 62% universal versus 38% claim-specific.\n- Human similarity judgment versus gap size: Pearson -0.22.\n- Oracle premises raised claim-matching macro-F1 from 13.84 to 47.43 (Table 8). Annotators A2 and A3 gave 33.97 and 32.91, so premise quality matters.\n- Authors state they cannot recover the arguer's genuine premises. Locators: Tables 3-8, §4.2-4.3.\n\n**[2] Habernal, Wachsmuth, Gurevych, Stein (2018). The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants. NAACL-HLT 2018, pp. 1930-1940.**\n- URL opened: https://arxiv.org/pdf/1708.01425 (v4, pre-print of the NAACL version).\n- READ DEPTH: full text read of the main text. Supplementary material was not opened.\n- Source: NYT Room for Debate, 188 debates, 5,000 comments used. The eight-step MTurk pipeline ran Dec 2016 to Apr 2017.\n  1. Stance annotation (kappa .58 with MACE and 5 workers).\n  2. Reason spans (alpha_u .51).\n  3. Reason gists.\n  4. Reason disambiguation (kappa .42). This keeps only pairs where the reason implies the claim: 1,955 pairs.\n  5. Workers write an alternative warrant (AW) for the opposite claim: 5,342 written.\n  6. AW validation. Workers must pick the reason out of the reason plus a distractor: 3,791 pass. A logic score of at least .68 keeps 2,613.\n  7. Workers write the warrant by minimally editing the AW: 2,447.\n  8. Final validation, with experts resolving disputes: 1,970 instances.\n- Cost was about $6,000. Year-based splits are 1,210 / 316 / 444.\n- Humans: 173 AMT participants, 10 items each. Mean 79.8% (SD .162); with extensive formal reasoning training 90.9%; 30 of 173 scored 100%. Education had negligible effect.\n- Models: random .473 / .491 (dev/test). Best neural .637 dev / .560 test. Language model about .500 on test.\n- Authors note an AW cannot be written for every reason/claim pair, and that free-form gap-filling [1] was abandoned because of large variance. Locators: §3-§5.1, Tables 1-2.\n\n**[3] Habernal et al. (2018). SemEval-2018 Task 12: The Argument Reasoning Comprehension Task. pp. 763-772.**\n- URL opened: https://aclanthology.org/S18-1121.pdf\n- READ DEPTH: substantial extract read, through Table 3. The §5 analysis was not read.\n- 21 systems took part. The best was GIST at .712 and the random baseline .527. Only three systems were significantly above random (approximate randomization test); most were near .5 to .6.\n\n**[4] Niven & Kao (2019). Probing Neural Network Comprehension of Natural Language Arguments. ACL 2019.**\n- URL opened: https://arxiv.org/pdf/1907.07355 (updated version).\n- READ DEPTH: full text read.\n- BERT-Large peaked at 77% on the ARCT test (median 71.2%, mean 67.1 ± 9; 5 of 20 seeds failed to train).\n- The cue \"not\" in the warrant picks the right one 61% of the time and applies to 64% of items overall (test: 52% / 77%).\n- Probing (max / mean): warrant-only 71.2 / 65.6; reason+warrant 75.0 / 60.0; claim+warrant 73.2 / 53.2. The whole gain over about 71% is explained by cues (Table 3).\n- Adversarial set (negate the claim, swap warrants; exploits the design where reason + alternative warrant implies the opposite claim): BERT max 53.3, mean/median 50.4 / 50.5 (Table 4).\n\n**[5] Chakrabarty, Trivedi, Muresan (2021). Implicit Premise Generation with Discourse-aware Commonsense Knowledge Models. EMNLP 2021, pp. 6247-6252.**\n- URL opened: https://aclanthology.org/2021.emnlp-main.504.pdf\n- READ DEPTH: full text read.\n- BART-large fine-tuned on ART (abductive narratives), optionally with PARA-COMET xIntent.\n- Test sets: D1 ARCT (1,654 triples), D2 [1] (494), D3 Becker microtext support pairs with a single implicit premise (112).\n- Human evaluation: 50 items per set (150 total), 3 MTurkers each, binary plausible/not. Plausible meant grammatical, relevant, commonsense-coherent, and completing the argument. Krippendorff alpha .56.\n- Plausible rate, base ART / +PARA-COMET: D1 50% / 58%; D2 48% / 66%; D3 38% / 56% (Table 5).\n- Zero-shot BART was judged unable to produce plausible premises (no figure given). Caveat: pre-LLM generator.\n\n**[6] Stahl et al. (2023). Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments. Findings of EMNLP 2023.**\n- URLs opened: https://aclanthology.org/2023.findings-emnlp.312.pdf (search highlights including Appendix D) and https://ar5iv.labs.arxiv.org/html/2310.18098 ([FT] extraction). Author list not captured in the retrieved text.\n- READ DEPTH: substantial extract read.\n- Corpus: ICLEv3-derived, 40,089 instances (17,762 positive, 22,327 negative). Removed units were 63.28% premises, 30.97% claims, 5.75% major claims.\n- Gap validity: annotator agreement was Kendall tau .25. [FT] The full method reduced sufficiency in 61.0% of arguments versus 53.5% for random removal.\n- Naturalness (3-point scale): modified 2.22 versus original 2.28 (20/38/42% versus 18/36/46%), tau .27.\n- Baselines [FT]: DeBERTa detection accuracy 72%, F1 68%; BART reconstruction ROUGE-1 21.83.\n- Appendix D (three annotators, 30 arguments, 5-point scales), mean clarity / strength / coherence: original unit 3.79 / 3.64 / 3.84; BART-augmented 3.39 / 3.13 / 3.47; Gurcke-style baseline 3.17 / 2.86 / 2.99. Majority-agreement ratios were 40% to 90%.\n- Authors' limits: no comparison with human-made enthymemes; no guarantee the created gaps match the traditional notion.\n\n**[7] Becker, Korfhage, Frank (2020). Implicit Knowledge in Argumentative Texts: An Annotated Corpus (IKAT). LREC 2020, pp. 2316-2324.**\n- URL opened: https://aclanthology.org/2020.lrec-1.282.pdf\n- READ DEPTH: search snippet only (multi-paragraph highlights of §3.3-3.4).\n- Protocol: decide whether the link between a unit pair is explicit. If not, add as few simple sentences as possible, one fact per sentence. Two trained experts each write a version and a third adjudicates.\n- Agreement was measured by Word Mover's Distance between the two versions: 1.97. For comparison, the earlier multi-annotator protocol had 2.2, 3.08 and then 1.89 by round. Concept-span overlap was 76.98% (Jaccard) and 84.87% (Dice).\n- Authors say annotators' intuitions about what is missing differ even when phrasing could match.\n\n**[8] Becker, Liang, Frank (2021). Reconstructing Implicit Knowledge with Language Models. DeeLIO 2021, pp. 11-24.**\n- URL opened: https://aclanthology.org/2021.deelio-1.2.pdf\n- READ DEPTH: search snippet only (abstract, intro, conclusion). The manual-evaluation numbers were not seen.\n- GPT-2, XLNet and BART were fine-tuned on Generics-KB and e-SNLI and tested on IKAT. Authors report automatic NLG metrics are insufficient for this task, and BART gave the most informative explanations.\n\n**[9] Gupta, Zuckerman, O'Connor (2024). Harnessing Toulmin's theory for zero-shot argument explication. ACL 2024, pp. 10259-10276.**\n- URL opened: https://aclanthology.org/2024.acl-long.552.pdf\n- READ DEPTH: full text read of the main body. Appendices were not in the fetched text.\n- Task: output a claim/reason/warrant triple from a prompt that names Toulmin's model. Evaluation sets are ARCT test (445 pairs) and Microtext (112 arguments, Becker warrants).\n- Prompt compliance (ARCT / MCT): GPT-4 99.0 / 100; GPT-3 94.6 / 95.4; Llama-2-70B 75.2 / 90.2; Falcon-40B 35.0 / 42.5.\n- Generic prompts such as \"explain the logical steps\" touched all three components in only 38.4% / 24.1% / 21.4% of GPT-4 responses. Many responses just paraphrased the input.\n- GPT-4 reason extraction against gold (FactScore recall/precision): ARCT .88 / .87, MCT .83 / .86.\n- Warrant acceptability: two Upwork freelancers with graduate-level composition and rhetoric expertise, 150 pairs (75 ARCT, 75 MCT), 300 judgments per warrant type. Gold 45.7%, GPT-4 61.7%, Llama-2-70B 26.3%. Cohen's kappa .18 (gold), .24 (GPT-4), .31 (Llama-2).\n- BERTScore of GPT-4 warrants versus gold was only .3 (ARCT) and .4 (MCT).\n- Acceptability criteria: relevant and fully explains the link; non-trivial (not a bare \"if reason then claim\"); must hold for the claim to follow, even if the rater disagrees with it.\n- Named failure modes:\n  - Gold warrants were rejected when they restated the claim, had wording errors, were irrelevant, or did not explain the link.\n  - GPT-4 warrants were rejected when they repeated the reason or claim, or took the \"if reason then claim\" form.\n  - Llama-2 mostly repeated the reason.\n- Giving the original argument as context made no difference for warrants (§4, footnote 5).\n\n**[10] Ku et al. (2025). Multi-Agent LLM Debate Unveils the Premise Left Unsaid. 12th Workshop on Argument Mining, pp. 58-73.**\n- URL opened: https://aclanthology.org/2025.argmining-1.6.pdf\n- READ DEPTH: full text read.\n- Setup: ARCT test (444 items), GPT-4o-mini, temperature .5, no few-shot or chain-of-thought.\n- Accuracy: random .500; replicated LSTM .7050; fine-tuned RoBERTa .7564; single-agent LLM .7928.\n- Debate with agents choosing their own premise: sequential .8694, simultaneous .8446.\n- Debate with agents assigned a premise to defend: sequential .6982, simultaneous .7207. This is below the single agent (McNemar p<.0001 and p<.01). Example contingency for Given-sequential versus single agent: 275 both right, 77 single only, 35 Given only, 57 both wrong.\n- Debate rounds 5 to 20 had no effect. Temperature changed which items were right, not overall accuracy (.851 to .869).\n- Authors attribute the drop to rhetorical rigidity and overcommitment, and the opposing agent sometimes deferred.\n- Limits: binary choice only; original (non-adversarial) test set [4]; possible contamination (my note).\n\n**[11] Ryu, Chu, Betz, Yang, Rosé, Welleck (2026). Argument Reconstruction as Supervision for Critical Thinking in LLMs. arXiv 2603.17432 (preprint, March 2026).**\n- URL opened: https://www.arxiv.org/pdf/2603.17432\n- READ DEPTH: substantial extract read, through Tables 2-3. Appendices were not read; §4.2 was seen only via a search highlight.\n- Validity was measured by LLM-to-first-order-logic translation plus a SAT solver. Prompting-only results: gpt-oss-120b 40.8; Qwen3-235B-Thinking 39.2; Claude Sonnet 4.5 no-think 59.2; GPT-5.2 none 51.7; Claude Sonnet 4.5 think 60.8; GPT-5.2 xhigh 80.8.\n- The authors' engine reaches 100% validity by construction. Faithfulness, as baseline win-rate against the engine with ties excluded: prompting baselines 26.4 to 48.8; earlier AAR engine 21.4.\n- The judge was validated by humans: NL-to-logic 99.0%, pairwise faithfulness 89.5%.\n- Failure taxonomy found in reconstructions: (1) misrepresented premises, including over-generalised ones; (2) missing essential premises; (3) unnecessary premises, including rhetorical or paraphrased content.\n- Remedy: accuracy, completeness and parsimony criteria, plus SAT-based pruning of premises used in no valid proof.\n- Corpus analysis (snippet): implicit premises are about 41.28% of premises, and LLM reconstructions use fewer implicit premises than human ones.\n- Caveat: validity is deductive validity of an LLM formalisation, not human plausibility.\n\n**[12] Sviridova, Cabrio, Villata (2026). Better than the Gold Standard? Evaluating Human vs. LLM Enthymeme Reconstruction in Natural Language Argumentation. COMMA 2026, FAIA vol. 428, pp. 77-88.**\n- URLs opened: https://ebooks.iospress.nl/DOI/10.3233/FAIA260802 and the /pdf/doi/ variant. Both returned only the abstract.\n- READ DEPTH: abstract only.\n- Claims: new human-reconstructed dataset from real-world arguments; evaluation by automatic metrics, human evaluation, and LLM-as-judge; human experts often find LLM reconstructions indistinguishable from human-authored ones. No numbers available to me.\n\n**[13] Sviridova, Cabrio, Villata (2026). Is There Anything More Deceptive than an Obvious Fact? Investigating Implicitness in User-Generated Argumentative Text. LREC 2026.**\n- URL opened: https://aclanthology.org/2026.lrec-1.657.pdf\n- READ DEPTH: search snippet only (results discussion).\n- 78 Change-My-View discussions and 112 Microtexts annotated for implicature, ambiguity and presupposition. This is a detection task, not generation.\n- LLM binary detection on CMV (argument component and enthymeme detection) was close to random; GPT-4o was better but still below fine-tuned encoders. Some models returned empty fields.\n- Presupposition was hardest to detect (RoBERTa macro-F1 .53 on CMV and .41 on Microtext).\n\n**[14] Sviridova, Cabrio, Villata (2026). Mining implicit arguments for reasoning: A survey. Argument & Computation 17(1):3-27.**\n- URL opened: https://sage.cnpereading.com/doi/10.1177/19462174251344764\n- READ DEPTH: search snippet only. It restates that humans reconstruct premises easily but use entirely different premises for the same claims.\n\n**[15] Saadat-Yazdi & Kökciyan (2024), Beyond Recognising Entailment (ACL 2024, Kialo-PTA24), and Saadat-Yazdi, Pan, Kökciyan (2023), Uncovering Implicit Inferences for Improved Relational Argument Mining (EACL 2023).**\n- URLs opened: https://aclanthology.org/2024.acl-long.520.pdf and https://aclanthology.org/2023.eacl-main.182/\n- READ DEPTH: search snippet only (abstracts and a few paragraphs).\n- Kialo-PTA24: 760 topics; annotators also judge whether the lever (the link) is valid. Argument canonicalisation is hard for FLAN-T5 and Llama-2.\n- The EACL paper reports that COMET-generated inference chains improve support/attack/neutral classification by 2 to 5 F1.\n\n---\n\n## 3. Q2: \"given a presupposed relation, the model justifies it\", and designs that reduce it\n\n### 3a. Evidence on the failure\n\n**[16] Kim, Htut, Bowman, Petty (2023). (QA)²: Question Answering with Questionable Assumptions. arXiv 2212.10003.**\n- URLs opened: https://arxiv.org/pdf/2212.10003 (raw text through §4.3) and https://ar5iv.labs.arxiv.org/html/2212.10003 ([FT] for Table 4).\n- READ DEPTH: substantial extract read.\n- Data: 602 search queries, half with a questionable assumption (246 false, 55 unverifiable). 12,000 queries were crowd-screened by 5 workers each, and only about 6% were flagged by at least one worker. 12% of expert labels changed after adjudication.\n- Headline, verified in the text: best models reach 56% human-judged acceptability, 64% on assumption detection and 72% on assumption verification.\n- Human rating: 100 sampled items, 5 Prolific raters, majority vote. No inter-rater statistic is reported.\n- [FT] Zero-shot detection was about 46% to 52% (near chance) for most models. text-davinci-003 acceptability was 48% zero-shot (40% on questionable items versus 56% on valid ones) and 56% in-context (50% versus 62%). Flan-PaLM was 28% zero-shot (8% on questionable items).\n- The paper stresses detection often needs narrow topical knowledge and that accommodation can be adversarial in information-seeking QA. Models tested are 2022 to 2023 vintage.\n\n**[17] Yu, Min, Zettlemoyer, Hajishirzi (2023). CREPE: Open-Domain Question Answering with False Presuppositions. arXiv 2211.17257 (v1 read).**\n- URLs opened: https://arxiv.org/pdf/2211.17257 and the ar5iv page ([FT] for results).\n- READ DEPTH: substantial extract read.\n- 8,466 labeled Reddit ELI5 questions; 26.0% (2,202) contain a false presupposition (25.0% in the test split).\n- Types in a 50-item sample: false predicate 30%, false property 22%, false causal relation 22%, false clause 14%, existential 6%.\n- Gold labels rely on the most-upvoted comment. The authors report inherent ambiguity about whether the asker really presupposed the claim.\n- [FT] Human detection macro-F1 was 85.1 with the comment and 70.9 without. Best model was 75.6 with the gold comment and 66 to 67 in the open track. Best model correction quality was 0.8 versus 2.7 for humans (0-3 scale).\n- Models are pre-chat-LLM (RoBERTa/T5/FiD class).\n\n**[18] Hu et al. (2023). Won't Get Fooled Again: Answering Questions with False Premises. ACL 2023.**\n- URL opened: https://aclanthology.org/2023.acl-long.309/ and the .pdf.\n- READ DEPTH: search snippet only (abstract plus a table fragment).\n- FalseQA: 2,365 human-written false-premise questions. Raw MACAW-11B flagged only 8.7% of false-premise items (precision 91.5%). After fine-tuning on 256 examples recall was 81.3%, and on full data 88.8%.\n- The authors argue the knowledge exists and must be activated. ChatGPT is said to handle these questions, noted as untested.\n\n**[19] Sieker, Lachenmaier, Zarrieß (2025). LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High. arXiv 2505.22354.**\n- URL opened: https://arxiv.org/pdf/2505.22354v2.pdf\n- READ DEPTH: search snippet only.\n- GPT-4o rejected 84.08% of false presuppositions in political contexts. Llama-3-8B accommodated 50.03% and Mistral-7B accommodated 91.51%.\n- The snippet's prose and table disagree on Llama's rejection rate (50.05% versus 15.55%), so treat that figure as unreliable.\n\n**[20] Turpin, Michael, Perez, Bowman (2023). Language Models Don't Always Say What They Think. arXiv 2305.04388.**\n- URL opened: https://arxiv.org/pdf/2305.04388\n- READ DEPTH: substantial extract read (abstract through §3.1 setup). Result figures and tables were not seen.\n- Models: GPT-3.5 (text-davinci-003) and Claude 1.0. Data: 13 BIG-Bench Hard tasks, 3,299 items. Biasing features: answer-is-always-A ordering in few-shot examples, and a suggested answer in the prompt.\n- Accuracy falls by as much as 36 points toward wrong answers. Of 426 explanations supporting biased predictions, only 1 mentions the bias (Appendix B).\n- In BBQ, explanations weight the same weak evidence inconsistently in line with stereotypes. Chain-of-thought reduces bias sensitivity in few-shot but hurts more than it helps zero-shot (Fig. 1 caption).\n\n**[21] Sharma et al. (2023/2024). Towards Understanding Sycophancy in Language Models. arXiv 2310.13548 (v4).**\n- URL opened: https://arxiv.org/html/2310.13548v4 (search highlights including results paragraphs).\n- READ DEPTH: substantial extract read.\n- The Are-you-sure challenge cuts accuracy by up to 27 points (Claude 1.3), averaged over six datasets. Models change their first answer 32% (GPT-4) to 86% (Claude 1.3) of the time, and admit a mistake 42% (GPT-4) to 98% (Claude 1.3), including when they were right. Confidence barely moves (GPT-4 98.9 to 98.9).\n- Preference-model analysis: the Claude 2 PM prefers sycophantic over baseline truthful responses 95% of the time. For the hardest misconceptions it prefers the sycophantic response 45% of the time against a helpful truthful one (Fig. 7).\n- Best-of-N against the PM reduces sycophancy less than a PM prompted to be non-sycophantic.\n\n**[22] Wei, Huang, Lu, Zhou, Le (2023). Simple synthetic data reduces sycophancy in large language models. arXiv 2308.03958.**\n- URL opened: https://arxiv.org/abs/2308.03958 (and the html v2).\n- READ DEPTH: search snippet only (abstract and results passages).\n- Scaling PaLM 8B to 62B raises opinion-repeating sycophancy by 19.8 points, and 62B to 540B by a further 10.0. Instruction tuning adds 26.0 (Flan-PaLM-8B versus PaLM-8B). Models also agree with objectively false addition statements when the user does.\n- A lightweight synthetic-data fine-tune lowers repeating the user's opinion by 4.7 to 10.0 points. Filtering out claims the model does not know matters.\n\n**[28] Acquiescence Bias in Large Language Models. Findings of EMNLP 2025; arXiv 2509.08480. Authors not captured in the retrieved text.**\n- URLs opened: https://arxiv.org/html/2509.08480 and https://aclanthology.org/2025.findings-emnlp.607/\n- READ DEPTH: search snippet only.\n- Five LLMs (Llama-3.1-8B, Mistral-Small-24B, Gemma-2-27B, Llama-3.3-70B, GPT-4o), nine tasks, three languages.\n- No classic acquiescence. In English, switching a neutral A/B question to yes/no raised no-type answers by 31% to 203% across models, even for disagree-phrased questions. The authors conclude the bias is toward no, not toward agreeing.\n\n**[27] Tjuatja et al. (2023/2024). Do LLMs exhibit human-like response biases? A case study in survey design.**\n- URL opened: https://www.cs.cmu.edu/~sherryw/assets/pubs/2023-biasmonkey.pdf\n- READ DEPTH: search snippet only.\n- Nine models, five biases including acquiescence. LLMs generally do not show human-like response biases and react to non-bias perturbations. Vanilla models are more human-like than instruction-tuned or RLHF ones.\n\n**[29] Eisape, Tessler, Dasgupta, Sha, van Steenkiste, Linzen (2024). A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models. NAACL 2024, pp. 8425-8444.**\n- URL opened: https://aclanthology.org/2024.naacl-long.466.pdf\n- READ DEPTH: search snippet only.\n- PaLM 2 models: larger means more logical, and above the human average of about 50%. LMs rarely answer that nothing follows, even for the 37 syllogisms where that is correct (accuracy near 0%). This is worst under zero-shot chain-of-thought.\n\n**[30] Dasgupta et al. (2022). Language models show human-like content effects on reasoning tasks. arXiv 2207.07051.**\n- URL opened: https://arxiv.org/html/2207.07051v4\n- READ DEPTH: search snippet only.\n- Belief bias: both humans and LMs tend to call an argument valid when its conclusion is believable, regardless of validity. Natural-language inference showed modest content effects.\n\n### 3b. Related abstention / missing-premise evidence\n\n**[31] Fan et al. (2025). Missing Premise exacerbates Overthinking. arXiv 2504.06514.**\n- URL opened: https://arxiv.org/pdf/2504.06514 (search highlights including Tables 2-4).\n- READ DEPTH: substantial extract read. These numbers are from the paper's table text; my [FT] summary was wrong.\n- MiP-GSM8K abstain rate on missing-premise questions: GPT-4o 46.9, Qwen2.5-32B-Instruct 44.0, Gemini 1.5 54.5, o1 55.7, o1-mini 40.0, o3-mini 23.7, DeepSeek-R1 16.5, QwQ-32B 10.1. On MiP-MATH: GPT-4o 15.4, R1 3.8, QwQ 1.9.\n- Response length on MiP-GSM8K: R1 3,246 tokens on missing-premise questions versus 1,226 on well-defined ones. GPT-4o was 202 versus 212.\n- Reasoning models voice suspicion early (in-process suspicion 95.5% to 100%, first suspicion at step 1.16 to 3.90) yet rarely abstain.\n\n**[32] \"Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of LLMs\" (PCBench). Findings of EMNLP 2025, paper 44.**\n- URL opened: https://aclanthology.org/2025.findings-emnlp.44.pdf\n- READ DEPTH: search snippet only. Authors not captured.\n- 15 LLMs. Proactive critique rate (no hint to check): GPT-4o 11.0%, DeepSeek-V3 40.5%, o4-mini 4.0%.\n- Assisted rate (told to check): GPT-4.1 73.2%, Gemini-2.0-flash-thinking 77.9%, o4-mini 74.2%.\n- Flawed premises inflate reasoning-model output length (o3-mini cost ratio 3.90). Direct contradictions are easiest; flawed solution completion is hardest.\n\n**[33] AbstentionBench (arXiv 2506.09038).**\n- URL opened: https://arxiv.org/abs/2506.09038v1\n- READ DEPTH: search snippet only. Authors not captured.\n- 20 datasets, 20 LLMs. Reasoning fine-tuning lowers abstention by 24% on average. Models hallucinate missing context. A system prompt helps but does not solve it, and scaling helps little.\n\n**[34] Sufficient Context (ICLR 2025).**\n- URL opened: https://www.researchgate.net/publication/385721878_Sufficient_Context_A_New_Lens_on_Retrieval_Augmented_Generation_Systems\n- READ DEPTH: search snippet only. Authors not captured.\n- An autorater (Gemini 1.5 Pro, 1-shot) labels context sufficiency at about .935 F1/accuracy. SOTA LLMs still answer 35% to 62% of insufficient-context questions correctly.\n- Retrieval lowers abstention: Claude 3.5 Sonnet 84.1% to 52%, GPT-4o 34.4% to 31.2%, Gemini 1.5 Pro 100% to 18.6%.\n- Selective generation using the sufficiency signal improves the correct fraction by 2 to 10%.\n\n**[35] Bridging the Detection-to-Abstention Gap in Reasoning Models under Insufficient Information (arXiv 2605.28070, May 2026).**\n- URL opened: https://ar5iv.labs.arxiv.org/html/2605.28070\n- READ DEPTH: search snippet only.\n- Base DeepSeek-R1-Distill: detection 45.3% but abstention 18.6%. Qwen3-30B-Thinking: detection 52.9%, abstention 21.1%.\n- Plain RL raises abstention to 52.7% to 63.6% with a persistent gap. The authors' method reaches 72.4% to 88.5%.\n\n**[36] Identify-then-Verify (arXiv 2512.06476).**\n- URL opened: https://arxiv.org/pdf/2512.06476\n- READ DEPTH: search snippet only.\n- Single-run what-is-missing judgments are unstable: disagreement across runs jumps from 1 to 2 runs and plateaus after about 4.\n- Method: 5 samples, consensus claim about the gap, then a separate call checks that the claimed missing information is really absent.\n- Gains over an autorater: MuSiQue-Ans 64.85 to 71.86, MuSiQue-Full 59.49 to 72.13, CouldAsk-BBC 63.89 to 78.78. Verification gives up to 37% relative error reduction.\n- It is worse on strict-faithfulness sets under default settings (SQuAD v2 82.87 to 69.50; FaithEval 78.08 to 57.61). A stricter verification prompt recovered more than 15 points on FaithEval.\n\n### 3c. Designs and their measured effects\n\n| Design | Measured effect | Source and caveat |\n|---|---|---|\n| Contrastive warrants: choose between a warrant and an alternative implying the opposite claim | Humans 79.8% (90.9% trained). GPT-4o-mini zero-shot .793. A fine-tuned BERT scored 77% but 53% on the mirrored set | [2][4][10]. The mirrored set controls cues; the LLM result is on the original set. |\n| Forcing the model to defend an assigned warrant | .698 to .721 versus .793 single-agent. The model's own choice of side gave .845 to .869 | [10] |\n| Both-sides explanations plus consistency inference (Maieutic) | Abstract: up to 20% better accuracy on 3 commonsense true/false sets. [FT] dev ablations: 72.5 to 68.4 without both-sides explanations; to 65.6 with likelihood-only consistency | Jung et al. 2022, EMNLP 2022, aclanthology 2022.emnlp-main.82. GPT-3 175B, binary statements. [FT] values unverified. |\n| Debate versus single advocate (consultancy) | Human judges 88% versus 78% versus 60% naive. LLM judges 76% versus 54% versus 48%. [FT] More persuasive advocates hurt judges in consultancy | [24]. Reading comprehension with verified quotes. |\n| Generate the conclusion from premises alone and compare | No gain. Premises-only macro-F1 .875, with generated conclusion .878, with true conclusion .885 | [25]. BART-era; the data barely test sufficiency, per the authors. |\n| Judge with anchoring information removed | Warrant-only BERT reaches 71% against 77% full input | [4]. Isolates answer-option cues. |\n| Verify each step under its premises only | Error identification accuracy up 6 to 16 points. Cost: lower accuracy on correct steps | [40] |\n| Swap order and accept only if consistent | GPT-4 consistent in only 65.0% of swaps. Few-shot examples raised it to 77.5% | [26]. Consistency is not accuracy. |\n| Repeat runs | Disagreement on gap judgments stabilises after about 4 runs | [36] |\n| Explicit instruction to look for flaws | Detection rises from 4% to 40% unprompted to 73% to 78% prompted | [32] |\n| Separate verification call on a claimed gap | Up to 37% relative error reduction | [36] |\n| Fine-tune against sycophancy | 4.7 to 10.0 points less repeating of the user's opinion | [22] |\n| Separate trained classifier instead of generative confidence | AUROC .97 / .95 (flag classifier) versus .57 / .62 (generation likelihood) | [39] |\n\n**[24] Khan et al. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv 2402.06782.**\n- URLs opened: https://arxiv.org/abs/2402.06782, https://arxiv.org/html/2402.06782 (highlights), and ar5iv ([FT]).\n- READ DEPTH: abstract plus search highlights, verified.\n- QuALITY with information asymmetry. Human non-expert judges: debate 88%, consultancy 78%, naive 60%. LLM judges: 76%, 54%, 48%.\n- [FT] More persuasive optimisation raised judge accuracy in debate and lowered it in consultancy.\n\n**[25] Gurcke, Alshomary, Wachsmuth (2021). Assessing the Sufficiency of Arguments through Conclusion Generation. ArgMining 2021, pp. 67-77.**\n- URL opened: https://aclanthology.org/2021.argmining-1.7.pdf (raw text to §5; Table 4 via search highlight).\n- READ DEPTH: substantial extract read.\n- Data: 1,029 arguments from 402 essays, 66.2% sufficient.\n- Manual evaluation of generated conclusions: 100 arguments × 3 versions, 5 annotators on 5-point scales. Krippendorff alpha .19 to .50. For the sufficiency question: .23 (ground truth), .43 (unsupervised BART), .35 (supervised BART). Majority agreement 57% to 64%.\n- Sufficiency classification macro-F1: human upper bound .883 (432 arguments), CNN .831, RoBERTa .876, premises-only .875, conclusion-only .553, generated-only .532, premises+true conclusion .885, premises+generated .878, conclusion+generated .571, all .885.\n\n**[26] Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets & Benchmarks.**\n- URL opened: https://arxiv.org/html/2306.05685v4 (search highlights).\n- READ DEPTH: substantial extract read.\n- Swap consistency on two near-identical answers: Claude-v1 23.8% (75% favour first), GPT-3.5 46.2%, GPT-4 65.0% (66.2% with renamed assistants).\n- Few-shot examples raised GPT-4 to 77.5%, Claude-v1 to 63.7% and GPT-3.5 to 55.0%, but moved GPT-3.5's bias to the second position.\n- Math grading, wrongly accepting an incorrect answer: default 14 of 20, chain-of-thought 6 of 20, reference-guided 3 of 20.\n- GPT-4 versus human agreement was 85% (non-tie), against 81% human-to-human.\n\n---\n\n## 4. Q3: multi-step structure\n\n**[37] Dalvi et al. (2021). Explaining Answers with Entailment Trees (EntailmentBank). EMNLP 2021.**\n- URLs opened: https://arxiv.org/pdf/2104.08661 and https://arxiv.org/html/2104.08661v3 (highlights covering §3.1 and §6.3).\n- READ DEPTH: substantial extract read (§3.1 and §6.3 raw text; some tables [FT]).\n- Step validity definition (§3.1): three graduate/undergraduate annotators. Each step is an entailment, meaning a conclusion a person would typically infer, whose content reasonably follows from its immediate children. Each step is one fine-grained inference. The tree is explicit, aiming to include all the knowledge a young child would need. It is also compositional and relevant. A tree is valid if every node is entailed by its children. Intermediate nodes are new sentences, not input sentences.\n- Annotators could add missing facts, and a second author reviewed each step.\n- Size: 1,840 trees (1,313 / 187 / 340), 5,881 steps (4,175 / 597 / 1,109), about 600 paid hours, about 20 minutes per question.\n- Reasoning types in 100 sampled steps [FT]: substitution 42%, rule application 33%, further specification 15%.\n- Automatic validity is approximated by matching the gold tree. Human re-rating of 50 Task-1 trees gave Overall AllCorrect 58% against 35.6% automatic, so the metric underestimates by about 20 points.\n- [FT] T5-11B Overall AllCorrect, no distractor / distractor / full corpus: 35.3 / 25.6 / 2.9.\n- Error analysis of 100 steps in imperfect trees: 30% were correct entailments (13% nearly correct). Invalid steps split as repetition 41% (the conclusion restates an input), invalid entailment 47% (uses knowledge that is not among the step's inputs but exists elsewhere in the context), and mis-evaluation or irrelevance 12%.\n- Error analysis of 50 trees: about 50% incorrect or missing leaves (for example the key fact was not used), about 25% valid trees scored invalid, about 20% correct leaves but invalid steps, about 5% disconnected, under 5% correct step but wrong intermediate.\n- Scores fall as the gold tree gets longer (Appendix Table A3).\n\n**[38] Yang, Deng, Chen (2022). Generating Natural Language Proofs with Verifier-Guided Search (NLProofS). EMNLP 2022.**\n- URLs opened: https://aclanthology.org/2022.emnlp-main.7.pdf (highlights) and ar5iv ([FT]).\n- READ DEPTH: substantial extract read.\n- T5-large stepwise prover plus a RoBERTa verifier that scores each step's validity.\n- [FT] Verifier negatives are generated automatically (drop or replace premises with BM25 distractors, copy a premise as the conclusion, negate the conclusion). Scores are min-aggregated.\n- EntailmentBank distractor task, Overall AllCorrect: EntailmentWriter T5-large 20.9, T5-11B 25.6, MetGen 27.7, NLProofS 33.3 ± 1.5. Leaves AllCorrect rose from 35.6 to 58.8.\n- Ablations: stepwise without search 31.8; without verifier score 31.9; without prover score 29.7; without stepwise 27.1.\n- Human check of 100 sampled steps per system: valid steps were 43% (EntailmentWriter), 65% (NLProofS without search), 77% (NLProofS).\n- [FT] Oracle prover gives a much larger gain than oracle verifier. Prompting GPT-3/Codex in one shot did worse (App. H, not read). Missing-step detection is not measured.\n\n**[39] Bostrom et al. (2022). Natural Language Deduction through Search over Statement Compositions. Findings of EMNLP 2022.**\n- URLs opened: https://aclanthology.org/2022.findings-emnlp.358.pdf (highlights) and ar5iv 2201.06028.\n- READ DEPTH: substantial extract read.\n- Setup: EntailmentBank-derived tasks with a 1:1 mix of valid goals and invalid goals (goal swapped from another example).\n- End-to-end T5-3B: always emits the hypothesis marker, so it \"proves\" 100% of goals. Using its generation likelihood to separate valid from invalid goals gives AUROC 0.57 (Task 1) and 0.62 (Task 2), about chance.\n- A variant with a trained true/false flag gets AUROC .97 / .95 but can still produce a nonsensical tree after the flag, because trees are post-hoc.\n- SCSearch (generate one step from a premise pair, then check against the goal): proves 48.8% / 52.9% of valid goals.\n- Human step validity (300 steps, three blind annotators, macro kappa .72, judging whether the conclusion can be inferred with minimal world knowledge): SCSearch 82.3%, Learned(Goal) 74.0%, end-to-end 65.0%.\n\n**[40] Mukherjee et al. (2025). Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs. ICML 2025 (PMLR 267).**\n- URLs opened: https://proceedings.mlr.press/v267/mukherjee25a.html and https://arxiv.org/html/2502.02362v3\n- READ DEPTH: search snippet only (abstract plus results paragraphs).\n- Open-source LLMs identify each step's premises with at least 90% recall. Verifying each step under its premises only improves error-identification accuracy by 6 to 16 points absolute, at some cost on correct steps (smaller for larger models). Accumulation errors, where a step is valid but built on a wrong premise, are hardest.\n- Single quote of this report, from the authors' results discussion: the models are \"inherently biased toward marking a step as correct\".\n\n**[41] Tafjord, Dalvi, Clark (2022). Entailer.**\n- URL opened: https://ar5iv.labs.arxiv.org/html/2210.12217\n- READ DEPTH: search snippet only. It over-generates backward-chaining steps, then filters by the model's own belief that (a) the premises are true and (b) the step is valid. Numbers not read.\n\n**[42] Ribeiro et al. (2022). Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (IRGR).**\n- URL opened: https://arxiv.org/html/2205.09224v2\n- READ DEPTH: search snippet only. It builds one entailment step at a time and reports about a 300% gain in Overall AllCorrect on the full-corpus task.\n\n**[43] Gao, Kitamura, Inoue (2024). Exploring the Challenges of Multi-Step Logical Reasoning with Language Models. NLP2024 (Japan).**\n- URL opened: https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/P6-19.pdf\n- READ DEPTH: search snippet only. Domestic proceedings, small sample.\n- GPT-3.5 few-shot step-by-step 28% and all-at-once 27% Overall AllCorrect, versus 35% for fine-tuned T5.\n- Error analysis on 40 samples: about 27% used the model's own knowledge instead of the given facts. In one case it skipped a needed fact because it judged it irrelevant. About 27% had invalid reasoning, and about 20% gave useless conclusions that repeat a given fact.\n\n**[44] Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with LLMs. EMNLP 2024, aclanthology 2024.emnlp-main.854.**\n- URL opened: https://aclanthology.org/2024.emnlp-main.854.pdf\n- READ DEPTH: search snippet only. Authors not captured.\n- GPT-3.5, GPT-4 and Llama-3-70B with structure-aware demonstrations and pruning on EntailmentBank, AR-LSAT and PrOntoQA. A probe withholds two evidence sentences, and models find them more easily when they sit in different subtrees. No numbers read.\n\n**[45] Song, Wu, Cai (2024), SPEH (LREC-COLING 2024), and FRVA (ACL Findings 2024).**\n- URLs opened: https://aclanthology.org/2024.lrec-main.1329/ and https://aclanthology.org/2024.findings-acl.540/\n- READ DEPTH: abstract only.\n- SPEH targets false feasible steps that look feasible but give wrong intermediates, with error detection and backtracking. FRVA uses fact retrieval plus verification.\n\nNo source I opened measures whether a model notices that an inference needs an intermediate conclusion. The closest evidence is the missing-premise and insufficiency studies [31]-[36] and the withheld-evidence probe in [44].\n\n---\n\n## 5. Q4: relevance and sufficiency, and how well people agree\n\n**[46] Stab & Gurevych (2017). Recognizing Insufficiently Supported Arguments in Argumentative Essays. EACL 2017, pp. 980-990.**\n- URL opened: https://aclanthology.org/E17-1092.pdf\n- READ DEPTH: substantial extract read, through §5.2.\n- RAS criteria (relevance, acceptability, sufficiency). Relevance is judged per premise and sufficiency over the premises jointly. A sufficient argument may still contain irrelevant premises.\n- 1,029 arguments from 402 essays, three non-native annotators. Guideline built on 64 arguments, with 20 arguments for collaborative training sessions.\n- On the 433-argument evaluation set: observed agreement 91.07%, Fleiss kappa .7672, Krippendorff alpha .7673. The two best annotators reached alpha .815 on the rest. Final labels were 66.2% sufficient and 33.8% insufficient.\n- Disagreement causes: modal verbs in the claim, long arguments perceived as sufficient, vague quantifiers such as many and some.\n- Abstract: a CNN reaches 84% accuracy identifying insufficient arguments.\n\n**[47] Stab & Gurevych (2014). Annotating Argument Components and Relations in Persuasive Essays. COLING 2014.**\n- URL opened: https://aclanthology.org/C14-1142.pdf\n- READ DEPTH: search snippet only.\n- Three annotators, 90 essays. Alpha_U .72 for components. Relations: alpha .81 overall (support .812, attack .807).\n- Markables are 5,137 candidate pairs, 25.5% support and 3.1% attack.\n\n**[48] Wachsmuth, Naderi, Hou, Bilu, Prabhakaran, Thijm, Hirst, Stein (2017). Computational Argumentation Quality Assessment in Natural Language. EACL 2017.**\n- URL opened: https://aclanthology.org/E17-1017.pdf\n- READ DEPTH: substantial extract read (definitions, Table 3, agreement discussion via search highlights).\n- 320 arguments, 15 dimensions on a 1-3 scale, three annotators chosen from a pool of seven after a pilot.\n- Table 3 (304 arguments judged argumentative by all): Krippendorff alpha of the most agreeing annotator pair was local acceptability .46, local relevance .47, local sufficiency .44, cogency .44, global sufficiency .27, overall .51.\n- Full three-way agreement: local acceptability 27.0%, relevance 32.6%, sufficiency 37.2%. Majority agreement was 90.8% / 92.4% / 92.8%.\n- In the pilot with seven annotators, alpha was at most .22 on every dimension.\n- Majority scores: local relevance got the top score 124 times, local sufficiency the bottom score 172 times.\n\n**[49] Mirzakhmedova, Gohsen, Chang, Stein (2024). Are Large Language Models Reliable Argument Quality Annotators? MISDOOM 2024; arXiv 2404.09696.**\n- URLs opened: https://arxiv.org/pdf/2404.09696, https://arxiv.org/html/2404.09696v1 (Tables 2-4), and the Springer page.\n- READ DEPTH: substantial extract read.\n- Same 320-argument corpus as [48]. Three experts versus three novices drawn from 108 students. LLMs were gpt-3.5-turbo-0613 and PaLM 2 text-bison@001. GPT-4 was not used for cost reasons. Llama-2-13B had near-zero run-to-run agreement.\n- Human within-group alpha (expert / novice): local relevance .41 / .36; local sufficiency .27 / .35; local acceptability .33 / .43; overall .40 / .37.\n- LLM run-to-run agreement over 3 runs: PaLM 2 about .96 to .99 without explanation prompts. GPT-3.5 .59 to .79 on relevance and sufficiency.\n- Human-LLM agreement, computed only on items where all humans agreed: moderate, maximum .71 (PaLM 2, appropriateness and global relevance). GPT-3.5 disagrees systematically with experts on local and global sufficiency.\n- Adding five PaLM 2 ratings lifts overall alpha to .43 to .45. Adding GPT-3.5 ratings does not help (.32 to .40, against a human-only .37 to .40). PaLM 2 refused about 23% of requests under content policy.\n- Models are 2023 vintage.\n\n**[50] Wachsmuth et al. (2024). Argument Quality Assessment in the Age of Instruction-Following LLMs. LREC-COLING 2024; arXiv 2403.16084.**\n- URL opened: https://ar5iv.labs.arxiv.org/html/2403.16084\n- READ DEPTH: substantial extract read via [FT] only. It is a position paper with no agreement numbers; it flags hallucination and bias.\n\n**[51] Gorur, Rago, Toni (2025). Can Large Language Models perform Relation-based Argument Mining? COLING 2025; arXiv 2402.11243.**\n- URLs opened: https://aclanthology.org/2025.coling-main.569.pdf and https://arxiv.org/html/2402.11243v1\n- READ DEPTH: search snippet only.\n- Binary support/attack micro-F1: Mixtral-8x7B 82, Llama2-70B 81, GPT-4o-mini 78 (4,000-sample subset), Mistral-7B 75, GPT-3.5-turbo 71. Best RoBERTa baseline 76.\n- Ternary micro-F1 (attack/support/neutral): Mixtral 68, Mistral-7B 64, GPT-4o-mini 63 (2,000 samples), GPT-3.5 60, Llama2-70B 54. Best RoBERTa baseline 59.\n\n**[52] KialoPrime (COMMA 2024, FAIA 388).**\n- URL opened: https://discovery.dundee.ac.uk/ws/portalfiles/portal/145044735/FAIA-388-FAIA240324.pdf\n- READ DEPTH: search snippet only. Authors not captured.\n- 5,687 discussion graphs and 1,088,801 relations. Neutral pairs are random claims from different discussions (362,933 pairs), so they are unrelated by construction. Reported relation F1 is .899 within discussions.\n\nI found no 2025-2026 result that tests a frontier LLM on judging whether a premise is relevant to a conclusion against a human-agreement ceiling. [49] and [51] are the nearest, and both use older models.\n\n---\n\n## 6. What transfers to your design and what does not\n\n### 6a. Evidence (no inference)\n\n1. **Naming the premise is something models can do at about human-acceptable quality, but \"acceptable\" is itself a noisy label.** GPT-4 warrants were accepted 61.7% against 45.7% for gold, with kappa .18 to .31 [9]. BART-era plausibility ranged from 38% to 66%, alpha .56 [5]. Raters disagree about what is missing, and humans write different premises for the same claims [1][7][14].\n2. **The \"two or three extra claims\" is the human baseline under open instructions.** Mean 2.7 premises [1]. Setups that obtained one premise constrained the task. ARCT asks for a single missing piece of common knowledge [2], IKAT asks for as few sentences as possible, one fact each [7], and Ryu et al. prune premises used in no valid proof [11].\n3. **Named failure modes of generated premises:**\n   - Restating the claim or reason, or a bare \"if reason then claim\" [9].\n   - Over-generalised, missing, or unnecessary premises [11].\n   - Repetition of an input (41% of invalid steps in the EntailmentBank error analysis [37], about 20% for GPT-3.5 few-shot [43]).\n   - Using unstated knowledge silently (47% of invalid steps [37], about 27% for GPT-3.5 [43]).\n4. **Conditioning on an asserted relation degrades judgment, with numbers.**\n   - Generators \"prove\" 100% of goals including invalid ones, and their likelihood cannot tell them apart (AUROC .57 / .62) [39].\n   - Biased prompts cut accuracy by up to 36 points and are almost never admitted [20].\n   - Challenges flip correct answers (up to 27 points) [21].\n   - Assigned advocacy is worse than neutral choice (.698 to .721 versus .793) [10]. Single-advocate setups mislead judges more than two-sided debate [24].\n   - Step verifiers lean toward calling steps correct [40]. Models avoid the nothing-follows answer [29].\n5. **A generic yes-bias is not what the literature finds.** In English yes/no prompts LLMs lean toward no [28], and LLMs mostly do not copy human acquiescence [27]. The documented acceptance failures are presupposition accommodation [16][17][19], sycophancy [21][22], and silently supplying the missing piece [31][33][43].\n6. **Instruction matters more than most designs.** Unprompted flaw detection can be 4% to 40%, but explicitly prompted detection is 73% to 78% for strong models [32]. Reasoning models suspect missing premises early but do not abstain [31][35]. Separating naming the gap from verifying it helped (up to 37% relative error reduction) [36].\n7. **Designs with positive measured effect** are in the table in §3c: contrastive choice [2][4][10], both-sides explanation with consistency [23], chosen-stance debate [10][24], premises-only verification [40], order-swap consistency [26], and a separate trained classifier [39].\n8. **Designs with no measured benefit:** generating the conclusion from the premises and comparing [25]. Run-to-run self-consistency among LLM raters is high (alpha up to .98) [49] but is not agreement with humans.\n9. **Human agreement ceilings:**\n\n| Task | Agreement | Source |\n|---|---|---|\n| Binary sufficiency, trained annotators | alpha .77 (best pair .815) | [46] |\n| Support/attack relations within essays | alpha .81 | [47] |\n| Single-step entailment validity | kappa .72 | [39] |\n| Graded relevance / sufficiency / acceptability, experts | alpha .47 / .44 / .46 (best pair) | [48] |\n| Same dimensions in a pilot with 7 annotators | at most .22 | [48] |\n| Relevance / sufficiency, experts (3) | .41 / .27 | [49] |\n| Relevance / sufficiency, novices (3) | .36 / .35 | [49] |\n| Sufficiency of human-written conclusions, 5 raters | alpha .23 | [25] |\n| Plausibility of a generated premise | alpha .56 | [5] |\n| Acceptability of warrants | kappa .18 to .31 | [9] |\n| Reason implies claim (ARCT step 4) | kappa .42 | [2] |\n\n   Willingness to accept a link also varies by person, from 1.2% to 14.5% abstention [1]. Disagreement causes in [46] are modal verbs, argument length, and vague quantifiers.\n\n### 6b. INFERENCE (mine, not from a source)\n\n- Two-sided or neutral prompts are better evidenced than assigned-side prompts. A pass that asks the model to name the bridge for the stated link and for its opposite, then compares, matches the ARCT twist [2], the both-sides explanation in [23], and the self-chosen-side debate in [10]. A pass that asks it to justify the stated link matches the assigned-stance condition that did worse [10].\n- Because accuracy on gap judgments rises sharply when the model is explicitly told to look for flaws [32] and when verification is a separate call [36], the evaluative step is best done by a separate call that does not know a link was asserted, sees only the two claims plus the named premise [40], and is run in both orders [26].\n- The over-listing is probably an instruction artifact (human baseline 2.7 [1]). A one-premise cap, a necessity test (remove the premise and check whether the link still fails), and a parsimony check [11] are the evidenced mechanisms.\n- An explicit none / not-a-reason option is supported: the ARCT pipeline allows \"impossible\" [2], [1] allows abstention, and models avoid nothing-follows [29].\n- Expect disagreement with careful readers on borderline links. Score the model against human-human agreement (alpha .27 to .47 for graded relevance/sufficiency [48][49], .77 for binary sufficiency with trained annotators [46]), not against 1.0.\n- The literature has no measurement of what you saw (the human-named premise appears in the model's list). It is consistent with [9] and [1] but is not replicated by any study I found.\n- ARCT has been public since 2018, so contamination of the 79% to 87% figures [10] cannot be excluded.\n\n### 6c. What does not transfer\n\n- No study used a Claude-class model for premise generation. Claude appears only as Sonnet 4.5 inside an engine [11] and as Claude 1.x in [20][21].\n- Premise-generation human ratings are small (150 items in [5]; 150 pairs and 2 raters in [9]; 30 arguments in [6]).\n- ARCT items are filtered to pairs where an opposite warrant could be written [2]. Your cross-text support links may lack a natural opposite.\n- Relation datasets build \"neutral\" negatives by random pairing [52], which is far easier than links a careful reader judges not to be reasons.\n- Debate [24], missing-premise [31], and premise-critique [32] results are on reading comprehension and math, not argument links.\n- [39] and [37] use grade-school science entailment with T5-era generators.\n- Human step-validity agreement (kappa .72 [39]) is for steps with minimal world knowledge; argument links need contested background assumptions.\n\n---\n\n## 7. Could not verify\n\n- Full text of [12]. Only the abstract was served, so the claim that experts often cannot tell LLM from human reconstructions is unquantified.\n- [8]: manual-evaluation numbers not seen.\n- [7]: only highlights seen.\n- Perez et al. (2022) on sycophancy, Poliak et al. (2018) and Gururangan et al. (2018) on hypothesis-only baselines, Wang et al. (2023) on LLM evaluator order bias, Lanham et al. (2023) on chain-of-thought faithfulness, Laban et al. (2023) on answer flipping, and Tyen et al. (2023) on mistake finding were not opened. I state nothing about them.\n- [FT] numbers: (QA)² per-model values [16], CREPE human/model values [17], Maieutic ablations [23], NLProofS verifier negatives and oracle results [38], Stahl sufficiency and detection percentages [6]. Headline values were checked in raw text where marked.\n- [19]: Llama's rejection rate is internally inconsistent in the snippet.\n- [28], [32], [33], [34], [44], [52]: author lists were not in the retrieved text.\n- A third-party summary of an ACL 2025 paper on argument mining in the LLM age showed numbers I did not trust and did not use.\n- I found no study measuring recall of the human-named premise inside an LLM's multi-premise list.\n- I found no study validating LLM gap-size or reasoning-scheme judgments for argument links against humans. Related pointers are [1] (gap size) and [15] (scheme canonicalisation difficulty).\n- Venue details for [16], [17], [20], [21], [23], [24] come from the arXiv or Anthology pages as retrieved. I did not independently confirm every proceedings volume.\n"}