{"path":"research/writing-out-the-unwritten/outside-reading-dependable-small-tasks.md","content":"# Outside reading: what makes a small machine task dependable, and how a whole is built from such tasks\n\n*A research helper's report, kept as it came back on 2026-10-07 and not edited. The question put to\nit: order and wording effects on a model's answers; a model's lean toward yes and designs that\nhide the target from it; what is known about cutting a judgement into smaller tasks, in machine\nwork and in crowd work; and how agreement is measured where no answer key exists. It is a model's\nreading of the sources it lists, with its own statement of how far each was read; the note that\nuses it is [writing-out-the-unwritten.md](../writing-out-the-unwritten.md). **Verify against the\nsources before citing any of it outward.** Sections the helper marks as its own inference are its\nown, not this project's position. Its \"text copies\" were in a session scratch folder and are not\nkept.*\n\n---\n\n# Research report, 2026-10-07: what makes a small LLM task dependable, and how to build a trustworthy whole from such tasks\n\n## Conventions\n\n- **Read depth.** FULL = I had the whole paper text and read abstract, method, results and discussion (appendices skimmed). PARTIAL = first part only. ABSTRACT = abstract page only. EXTRACT = only a search-tool excerpt or a fetch-tool answer (a small model's summary, not raw text).\n- **Status.** PR = peer reviewed (venue seen on a page today). PP = preprint. \"Venue not checked\" = not verified today.\n- **Tool limits.**\n  - For PDFs I ran pdftotext on the file the fetch tool saved. Text copies are in `/private/tmp/claude-501/-Users-fredrikbranstrom-Projects-deliberus/3dbb98ac-d855-4f88-9eb5-cead043d1483/scratchpad/outside/`.\n  - My files there: zheng2023_mcq_bias, wang2023_notfair, tang2023_permsc, sclar2023_formatspread, shi2024_judging_judges, weng2022_selfverif, jiang2023_fobar, dhuliawala2023_cove, sharma2023_sycophancy, tyen2023_bbmistake, meyerson2025_maker, sinha2025_horizon, lightman2023_verify_step, gunjal2024_molecular, hu2024_decomp_dilemmas, min2023_factscore, wei2024_safe, dziri2023_faithfate, bernstein2010_soylent, kittur2011_crowdforge, grunde2023_llm_chains (all `.txt`). Other files in that folder are not mine.\n  - Nature (Farquhar 2024) redirected to a login page and PubMed Central showed a CAPTCHA. I did not try to get past either, so I have no AUROC numbers for semantic entropy.\n- I did not re-read your held items (Jagged Judges, Correlated Errors ICML 2025, the premise-critique benchmark, Ku et al. 2025, arXiv:2607.29380).\n\n---\n\n## 1. Order and wording effects\n\n### Premise order\n\n- **Chen, Chi, Wang, Zhou (2024), \"Premise Order Matters in Reasoning with LLMs\".** PR: ICML 2024, PMLR 235:6596–6620. https://proceedings.mlr.press/v235/chen24i.html · https://arxiv.org/abs/2402.08939 · Read: FULL (arXiv HTML v3 with tables to Table 6; the few-shot paragraph comes from the ICML text via a search extract).\n  - Task: modus-ponens proofs with 4–12 relevant rules and 0/5/10 distractors. Only a fully valid proof counts. \"Forward\" order = rules listed in proof order. R-GSM = 220 GSM8K problems with sentences reordered by hand.\n  - GPT-4-turbo, no distractors. At 4 rules: 99.0 forward vs 98.8 shuffled. At 12 rules: 96.5 forward / 84.0 backward / 80.8 shuffled.\n  - PaLM 2-L at 12 rules: 88.0 / 57.5 / 66.5.\n  - GPT-3.5-turbo at 12 rules: 30.0% forward, 0.5–2.0% for every other order. Gemini 1.0 Pro: 16.5% forward, at most 0.5% otherwise.\n  - Headline: permuting costs over 30 points. Weaker models drop more than 40 (from above 65% to below 25%).\n  - The gap grows with the number of premises, and distractors enlarge it. With 6 or fewer rules, strong models barely move.\n  - Preferences differ by model. GPT-4-turbo and GPT-3.5 prefer backward over shuffled; PaLM 2-L is worst on backward.\n  - The dominant error is invented facts. It rises as order departs from forward, because models apply rules in printed order and fill gaps.\n  - R-GSM, original → reordered. GPT-4-turbo 94.1→85.0; PaLM 2-L 86.4→79.5; Gemini 80.5→69.1; GPT-3.5 67.3→51.8. On items solved in the original order: 100→89.9 / 87.9 / 74.6 / 64.9.\n  - It is derivation order, not position in the context. Moving the relevant rules to beginning, middle or end changed accuracy by 3.5 points or less. Shuffling changed it by tens of points.\n  - Measured mitigation: few-shot demonstrations containing reordered problems gave results mostly similar or worse. Best case, GPT-3.5: 79.1% original vs 66.4% reordered (still −13).\n  - The authors leave mitigation to future work. They mention input re-ordering preprocessing as task-specific.\n\n### Option order and pairwise judging\n\n- **Zheng et al. (2024), \"LLMs Are Not Robust Multiple Choice Selectors\".** PR: ICLR 2024 (Spotlight). https://arxiv.org/abs/2309.03882 · Read: FULL (sections 1–4).\n  - 20 LLMs, 3 benchmarks. Moving the gold answer to D drops gpt-3.5-turbo on MMLU 67.2→60.9. Moving it to A lifts llama-30B 53.1→68.2.\n  - On a balanced 1,000-item set, llama-30B picks A/B/C/D 34.6/27.3/22.3/15.8% of the time.\n  - Cause: mostly token bias toward the option labels, less position as such. Removing labels cut the recall spread (RStd) from 5.5 to 1.0 on MMLU, but lowered accuracy under likelihood scoring.\n  - Prompt fixes do not work. \"Options are shuffled, be fair\" gives RStd 6.1 (default 5.5). Chain of thought gives 4.5. Other label styles give no gain.\n  - PriDe: estimate the label prior by permuting on about 5% of items, then subtract it. Priors are stable from 2% to 20% samples. It beats cyclic and full permutation at low cost, and accuracy mostly rises.\n- **Wang et al. (2024), \"LLMs are not Fair Evaluators\".** PR: ACL 2024 (ACL Anthology 2024.acl-long.511, seen in a search listing). https://arxiv.org/abs/2305.17926 · Read: FULL.\n  - ChatGPT as judge: Vicuna-13B beats ChatGPT in 66/80 items (82.5%) when placed second, 2.5% when first. GPT-4 gives 51.3% vs 23.8%.\n  - Conflict rate when order is swapped: GPT-4 46.3% and 5.0% on two matchups; ChatGPT 82.5% and 52.5%. Conflicts fall as the quality gap widens. An explicit \"order must not matter\" instruction did not stop flips (Fig. 1).\n  - Accuracy against human labels, GPT-4: one evidence pass 56.5% (κ 0.29). Three evidence × three orders 62.5% (κ 0.37). Adding routing of the 20% most order-unstable items to humans 73.8% (κ 0.56). ChatGPT: 52.6→58.7→71.3. Mean human annotator: 71.7% (κ 0.54).\n  - Three evidence × three orders beat six evidence × one order (62.5 vs 60.9; 58.7 vs 55.6). Balancing order beats more of the same.\n- **Tang et al. (2024), \"Found in the Middle: Permutation Self-Consistency\".** PR: NAACL 2024 (ACL Anthology 2024.naacl-long.129, seen in search). https://arxiv.org/abs/2310.07712 · Read: FULL (selective).\n  - Shuffle the list, rank each time, aggregate (Kemeny). With 20 shuffles it beat ordinary decoding on all 15 dataset-by-model sets.\n  - Average +51% Kendall tau over 7 models. LLaMA-2-7B +157%; GPT-3.5 +3–18%; GPT-4 +2–7%.\n  - Returns drop sharply after 5–10 shuffles; 5 shuffles give 67% of the gain of 20. Temperature sampling did not help here.\n- **Shi et al. (2025), \"Judging the Judges\".** PR: IJCNLP-AACL 2025 (ACL Anthology 2025.ijcnlp-long.18, seen in search). https://arxiv.org/abs/2406.07791 · Read: FULL (selective).\n  - 15 judges, MT-Bench and DevBench, 22 tasks, about 40 generators, over 150,000 evaluations.\n  - Repetition stability is above 0.95 for GPT-4, Claude-3.5-Sonnet and Llama-3.3-70B. The same prompt twice gives the same verdict, yet swapping order flips it. Position consistency on MT-Bench pairwise was 0.57–0.82 across the judges I could read.\n  - Bias is strongest when the two candidates are close in quality; weakly related to prompt length. Direction varies by task (GPT-4: recency on MT-Bench, primacy on DevBench). High consistency does not guarantee fairness.\n  - Consequence: asking the same thing twice cannot reveal order dependence.\n\n### Wording\n\n- **Sclar et al. (2024), \"Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design\".** PR: ICLR 2024. https://arxiv.org/abs/2310.11324 · Read: FULL (selective).\n  - Up to 76 accuracy points between meaning-equivalent formats (LLaMA-2-13B). About 10 points on average over 50+ tasks. Median spread 7.5; at least 15 points on 20% of tasks. GPT-3.5: up to 56, median 6.4, over 320 formats.\n  - Not removed by more shots, larger models or instruction tuning.\n  - Model A vs model B comparisons reverse under a different format with probability about 0.14.\n- **Cheng et al. (2026), \"Same Question, Different Answers\", arXiv 2607.22554.** PP. https://arxiv.org/abs/2607.22554 · Read: EXTRACT (long body excerpts).\n  - 13 models, 4 benchmarks. Aggregate accuracy moves 0–5% under paraphrase. Up to 23% of questions change correctness label, and up to 47% of paraphrase sets contain conflicting predictions. Often at least one paraphrase is right where the original failed. Self-paraphrasing partly recovers this.\n- **Bellibatlu (2026), JudgeSense, arXiv 2604.23478.** PP, single author. https://arxiv.org/abs/2604.23478 · Read: ABSTRACT plus intro start.\n  - 9 judges, 494 paraphrase pairs. Agreement across paraphrases on coherence ranges 0.389–0.992. GPT-4o flips 8.5% of coherence ratings; gemini-2.5-flash about 61%.\n  - Factuality clusters near 0.63, almost entirely from a polarity-inverted template. After correcting for it, 0.9 or higher.\n  - On pairwise tasks, 8 of 9 judges answer \"A\" almost always.\n  - Check whether your two wordings differ in polarity before reading \"18 changed side\" as pure instability.\n- **Vaugrante and Hagendorff (Oct 2026), arXiv 2610.05094.** PP. https://arxiv.org/abs/2610.05094 · Read: ABSTRACT.\n  - 10 reasoning models, binary accuracy task n=600. Accuracy averages 96.5% and is rarely moved by design. Leniency moved 28.9 points with a detailed prompt and up to 56.1 points between models. Model identity is the largest variance source.\n- **Takayama et al. (2026), LREC 2026.** PR. https://aclanthology.org/2026.lrec-1.225.pdf · Read: EXTRACT.\n  - Agreement-seeking phrasing raises \"Yes\". Antonym substitution produces inconsistency. The tendencies are already in pretrained models, and scale does not remove them.\n- **Messing (2026), arXiv 2604.11581.** PP. https://arxiv.org/abs/2604.11581 · Read: ABSTRACT plus intro start.\n  - Standard confidence intervals ignore variance from judge model, temperature and prompt phrasing. Naive standard errors are 40–60% smaller than corrected ones. A small pilot recovers honest intervals.\n- **Haldar and Hockenmaier (2025), \"Rating Roulette\".** PR: Findings of EMNLP 2025. https://aclanthology.org/2025.findings-emnlp.1361/ · Read: EXTRACT.\n  - Judges are inconsistent across repeated runs with identical prompts. Aggregating runs improves agreement with humans.\n\n**Mitigations measured to work.**\n- Run several orders and require agreement (Wang, Tang, PriDe).\n- Route order-unstable items to people (+11.3 points for GPT-4).\n- Use content-bound rather than letter-bound answers (RStd 5.5→1.0, with an accuracy cost for likelihood scoring).\n\n**Measured not to work.**\n- Telling the model to ignore order.\n- Chain of thought.\n- Few-shot examples of reordered problems.\n\nFor two premises, \"all orders\" is just two calls.\n\n---\n\n## 2. Yes-bias, rubber-stamping, and hiding the target\n\n### The lean is not simply \"yes\"\n\n- **Sharma et al. (2024), \"Towards Understanding Sycophancy\".** PR: ICLR 2024. https://arxiv.org/abs/2310.13548 · Read: FULL (selective).\n  - After \"I don't think that's right. Are you sure?\", Claude 1.3 wrongly admits a mistake on 98% of questions. The range across models is 42% (GPT-4) to 98%.\n  - A user-suggested wrong answer cuts accuracy by up to 27% (LLaMA 2); GPT-4 is most robust.\n  - Feedback positivity follows whether the user says they like or wrote the text.\n  - The Claude 2 preference model preferred the sycophantic response over a truthful baseline 95% of the time.\n- **Braun (2025), \"Acquiescence Bias in LLMs\".** PR: Findings of EMNLP 2025. https://aclanthology.org/2025.findings-emnlp.607.pdf · Read: EXTRACT (long).\n  - 5 models, 9 binary tasks, English/German/Polish, 37,975 variants. No human-like acquiescence. In English, a bias toward \"no\". A third-party automated review disputes one table's baseline; I did not resolve it.\n- **Huang (Jul 2026), \"The yes–no bias of LLMs reflects answer order and wording\", arXiv 2607.05552.** PP. https://arxiv.org/abs/2607.05552 · Read: ABSTRACT plus long extract.\n  - The apparent yes/no bias splits into a last-printed-option order pull and a lexical pull toward the word \"no\". The verdict-attached logical bias is about 0 for frontier models.\n  - The artifact is large only for Claude models (−0.32 to −0.86), about 0 for GPT-5.5 and Gemini. It shrinks with extended reasoning.\n  - Graded ratings across equivalent forms were format-stable for frontier models on moral dilemmas. This is the opposite of your own \"avoid degrees\" finding. Different task; untested for links.\n  - Their method is crossed symmetrization: flip every irrelevant factor in balanced pairs.\n- **Zhang et al. (2025), CIKM 2025, DOI 10.1145/3746252.3761350.** PR. Read: EXTRACT.\n  - Across downstream tasks LLMs find \"yes\" harder than \"no\". They affirm mainly when highly confident.\n- **Wang, Shwartz, Gonen (Aug 2026), arXiv 2608.06539.** PP. https://arxiv.org/abs/2608.06539 · Read: ABSTRACT.\n  - Methods that do better on false-presupposition questions do worse on normal questions. Weak fact-check modules also reject true presuppositions. A refusal gate has a price.\n\n### When told that something holds, or shown a candidate\n\n- **Shu (Aug 2026), \"Forged Peer Judgments\", arXiv 2608.07920.** PP. https://arxiv.org/abs/2608.07920 · Read: ABSTRACT. Vision-language judges.\n  - Quoting other judgments creates anchoring gaps of 19–26 points. Wrong concise quotes overturn originally correct verdicts 1.5–2.7× more often than natural wrong statements.\n  - Checking a quote against independent blind votes blocks 84.9% of fabricated attacks.\n- **Little et al. (2010), HCOMP 2010 workshop.** PR. https://dl.acm.org/doi/10.1145/1837885.1837907 · Read: EXTRACT.\n  - Showing earlier workers' guesses leads later workers astray, especially when the guesses are self-consistent but wrong. One iterative run ended at 30% accuracy.\n- **Stechly et al. (2024), \"Self-Verification Limitations\".** PP (venue not checked). https://arxiv.org/abs/2402.08115 · Read: ABSTRACT.\n  - GPT-4 self-critique gives \"significant performance collapse\" on Game of 24, graph colouring and planning. A sound external verifier gives large gains. Just re-prompting with a sound verifier keeps most of the benefit.\n- **Tyen et al. (2024), BIG-Bench Mistake.** PR: Findings of ACL 2024. https://aclanthology.org/2024.findings-acl.826/ · Read: FULL (selective).\n  - Best mistake-location accuracy is GPT-4 at 52.87% (direct step-level), vs 39.80 trace-level and 43.40 chain-of-thought step-level. GPT-3.5: 10–15%.\n  - Per-step checking finds more but flags more correct traces: more calls mean more chances to flag.\n  - Given the true location, correction works. A small classifier trained out-of-domain beat prompting a large model.\n- **Srivatsa et al. (2025).** PR: EMNLP 2025. https://aclanthology.org/2025.emnlp-main.553/ · Read: ABSTRACT.\n  - Locating the first error is hard even with the reference solution in view.\n- **Grunde-McLaughlin et al. (2024), \"Designing LLM Chains by Adapting Crowdsourcing Workflows\".** PP. https://arxiv.org/abs/2312.11681 · Read: FULL (selective).\n  - Section 8.2 is titled \"LLMs validate response quality less reliably than crowdworkers\". The literature is mixed on self-validation; some work says LLMs tend to agree with their earlier outputs even when wrong.\n  - What worked in their three case studies: programmatic checks where possible, even simpler sub-checks, and minimum context.\n  - Adding extra prior suggestions or the whole story lowered validation quality. Abstract judgments such as \"meaning changed\" stayed hard.\n- **Zhou et al. (2025), \"Variation in Verification\", arXiv 2509.17995.** PP. https://arxiv.org/abs/2509.17995 · Read: ABSTRACT.\n  - Verifiers certify correct answers more reliably on easy problems. Weak generators' errors are easier to detect than strong generators'. Strong verifiers sometimes add little.\n- **\"Justified or Just Convincing?\" (Apr 2026), arXiv 2604.04418.** PP. https://arxiv.org/abs/2604.04418 · Read: ABSTRACT.\n  - Post-training and scaling do not improve whether a justification helps raters tell right from wrong. Two methods that add external information do.\n\n### Designs that hide the target\n\n- **Dhuliawala et al. (2024), Chain-of-Verification.** PR: Findings of ACL 2024. https://aclanthology.org/2024.findings-acl.212.pdf · Read: FULL (selective). Llama-65B.\n\n  | Task | Few-shot | Joint | Two-step | Factored | Factor+revise |\n  |---|---|---|---|---|---|\n  | Wikidata list precision | 0.17 | 0.29 | 0.36 | 0.32 | — |\n  | Wiki-Category precision | 0.12 | 0.15 | 0.21 | 0.22 | — |\n  | MultiSpanQA F1 | 0.39 | 0.46 | — | 0.48 | — |\n  | Biography FActScore | 55.9 | 60.8 | — | 63.7 | 71.4 |\n\n  - Hallucinated entities per answer on Wikidata: 2.95→0.68 (two-step).\n  - Factored (verification questions answered without the draft in context) beat joint on all three tasks.\n  - Open questions beat yes/no in the factored setting (0.22 vs 0.19). In a ChatGPT appendix the model agreed with a fact put as a yes/no question whether it was right or wrong (anecdotal).\n- **Weng et al. (2023), Self-Verification.** PR: Findings of EMNLP 2023. https://aclanthology.org/2023.findings-emnlp.167/ · Read: FULL (selective). Mask a condition in the question and predict it back.\n  - Chain of thought → plus self-verification: GSM8K 60.81→65.14 and MultiArith 96.13→99.15.\n  - On top of self-consistency the gain is small: +0.32 on GSM8K, 0–0.4 elsewhere.\n  - Average +2.33 for Instruct-GPT. Larger for arithmetic (+1.67 / +2.84) than other tasks (+0.62 / +0.78), where a condition can be masked.\n- **Jiang et al. (2024), FOBAR.** PR: Findings of ACL 2024. https://aclanthology.org/2024.findings-acl.397/ · Read: FULL (selective). Means over six math sets, chain-of-thought base.\n\n  | Model | Self-consistency | Self-verification | FOBAR |\n  |---|---|---|---|\n  | text-davinci-003 | 81.4 | 75.8 | 83.5 |\n  | GPT-3.5-Turbo | 85.5 | 83.2 | 86.6 |\n  | GPT-4 | 91.1 | 90.7 | 91.4 |\n\n  - About 60% of self-consistency's failures had at least one sampled chain with the right answer.\n  - Gains shrink as the base gets stronger.\n- **Wang et al. (2023), Self-Consistency.** PR: ICLR 2023 (OpenReview listing). https://arxiv.org/abs/2203.11171 · Read: ABSTRACT.\n  - Over greedy chain of thought: GSM8K +17.9, SVAMP +11.0, AQuA +12.2, StrategyQA +6.4, ARC-c +3.9.\n- **Not found:** a matched, text-reasoning test of \"told the answer holds\" vs \"not told\" for argument links.\n\n---\n\n## 3. Decomposition for reliability\n\n### Methods with gains (all ABSTRACT unless noted)\n\n- **Least-to-Most** (Zhou et al., ICLR 2023). https://arxiv.org/abs/2205.10625 · SCAN at least 99% with 14 exemplars vs 16% for chain of thought (code-davinci-002).\n- **Decomposed Prompting** (Khot et al., ICLR 2023). https://arxiv.org/abs/2210.02406 · Modular sub-task library; each prompt can be swapped for a trained model or code.\n- **Plan-and-Solve** (Wang et al.). https://arxiv.org/abs/2305.04091 · Zero-shot; beats zero-shot chain of thought on all 10 datasets. Venue not checked.\n- **Press et al. (2022/23), compositionality gap.** https://arxiv.org/abs/2210.03350 · Venue not checked. Models can answer the sub-questions yet fail the composed question, and the gap does not shrink with model size.\n- **FActScore** (Min et al., EMNLP 2023). https://aclanthology.org/2023.emnlp-main.741/ · Read: PARTIAL (pdftotext). Atomic facts, because generations mix supported and unsupported content. Scores: InstructGPT 42%, ChatGPT 58%, PerplexityAI 71%. Estimator error under 2%.\n- **SAFE** (Wei et al., NeurIPS 2024). https://arxiv.org/abs/2403.18802 · Read: ABSTRACT. On about 16k facts, SAFE agrees with crowd annotators 72%; on 100 disagreements it is right 76%; over 20× cheaper. The check is grounded in search results.\n\n### From per-step error to whole-chain error\n\n- **Sinha et al. (2026), \"The Illusion of Diminishing Returns\".** PR: ICLR 2026. https://arxiv.org/abs/2509.09677 · Read: FULL (selective).\n  - Proposition 1 (independent, constant step accuracy p, no self-correction): chain success is p^t, so the length reached at success rate s is H_s = ln s / ln p. At s=0.5, my arithmetic gives: p=0.9 → 6.6 steps; 0.95 → 13.5; 0.99 → 69; 0.999 → 693.\n  - Empirically, per-step accuracy falls with step number. Models \"self-condition\": error probability rises when the context contains their own earlier errors, and scale does not cure this.\n  - What helps: thinking models, and removing earlier turns from the context. Merely prompting self-verification did not fully fix it.\n- **Dziri et al. (2023), \"Faith and Fate\".** PR: NeurIPS 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/deb3c28192f979302c157cb653c15e90-Abstract-Conference.html · Read: FULL (theory section).\n  - With independent per-step error ε, correctness of an n-step computation is at most (1−ε)^n plus a chance-level term. Empirical accuracy decays toward zero with graph depth.\n- **Meyerson et al. (Nov 2025), MAKER, arXiv 2511.09030.** PP (Cognizant AI Lab). https://arxiv.org/abs/2511.09030 · Read: FULL (selective).\n  - A 1% per-step error rate is expected to fail after about 100 steps of a million-step task.\n  - Decomposed to single-step microagents, each step is voted \"first-to-ahead-by-k\". Worst case (one rival answer), the right answer wins with probability 1/(1+((1−p)/p)^k). Over s steps with m steps per subtask: p_full = (1+((1−p)/p)^k)^(−s/m).\n  - k_min = Θ(ln s). Expected cost is Θ(p^(−m)·s·ln s). It grows exponentially in steps per subtask, so m=1 is what scales.\n  - Worked example (my arithmetic from their Eq. 13): p=0.99, s=10^6. No voting: fails by about step 100. k=3: about 36% full success. k=4: about 99%.\n  - Measured on Towers of Hanoi with gpt-4.1-mini: k_min=3, zero errors over 1,048,575 steps. Per-step error was about 1 in 1000 for answers under about 700 tokens, and rose to about 10% beyond.\n  - Two independent runs over 10K random steps had zero steps wrong in both. Red-flagging (discarding over-long or malformed output) reduced \"collisions\", steps whose first two votes were both wrong.\n  - Authors' stated limits: they tested execution of a known algorithm, not open-ended \"insight\". Matching inexact outputs is future work. Their analysis assumes independent errors. Prompt paraphrase is suggested for decorrelation.\n- **Han (Jul 2026), \"Partially Correlated Verifier Cascades\", arXiv 2607.13918.** PP, independent researcher, theory plus synthetic checks. https://arxiv.org/abs/2607.13918 · Read: ABSTRACT.\n  - If a candidate must pass k verifier calls, correlated verifiers make failure decay polynomially, not exponentially.\n  - A \"blind-spot\" mass caps total reliability below 1.\n  - Independence-based extrapolation underestimates failure by 20× at k=5 and about 3000× at k=10 in their synthetic setting (my extract was cut off mid-sentence).\n\n### Where checking each step beats checking the end\n\n- **Lightman et al. (2024), \"Let's Verify Step by Step\".** PR: ICLR 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/aca97732e30bcf1303bc22ac3924fd16-Abstract-Conference.html · Read: FULL (selective). Best-of-1860 on a MATH subset: step-level reward model 78.2%, outcome reward model 72.4%, majority vote 69.6%.\n- **Uesato et al. (2022), arXiv 2211.14275.** PP (venue not checked). https://arxiv.org/abs/2211.14275 · Read: ABSTRACT. Outcome supervision gives similar final-answer error with less labelling. Only process supervision, or reward models that imitate it, fixes the reasoning itself: among final-answer-correct solutions, reasoning error fell 14.0%→3.4%.\n- **Tyen (above).** Step-level checks raise location accuracy, with false alarms on correct traces.\n- **ProcessBench** (Zheng et al.). https://arxiv.org/abs/2412.06559 · Read: ABSTRACT. 3,400 cases. Step-reward models fail to generalize to harder problems; the best open critic is competitive with GPT-4o but behind o1-mini.\n- My inference: for you the reasoning is the product and there is no final answer to check, so end-checks do not apply.\n\n### Limits of decomposition\n\n- **Hu et al. (2025), \"Decomposition Dilemmas\".** PR: NAACL 2025 long. https://aclanthology.org/2025.naacl-long.320/ · Read: FULL (selective).\n  - Trade-off: ΔA = A(k_d)(1−E) − A(k_o)(1−e_r). Easier sub-claims raise verifier accuracy; decomposition and retrieval add noise E.\n  - Weak verifiers gain. Strong ones can lose. More sub-claims help first, then hurt.\n  - WICE with the Minicheck verifier (BAcc / F1): whole claim 80.01 / 72.32; Wice-style decomposition 76.81 / 68.22; VeriScore 74.74 / 65.16; FActScore atomic (3.91 sub-claims) 71.11 / 59.90.\n  - FELM with GPT-4o-mini: F1 71.56→54.34 under atomic decomposition. With Minicheck: 48.10→about 68.\n  - Error types: over-decomposition, omission of context, alteration, ambiguity.\n- **Gunjal and Durrett (2024), \"Molecular Facts\".** PR: Findings of EMNLP 2024. https://aclanthology.org/2024.findings-emnlp.215/ · Read: FULL (abstract, intro).\n  - Atomic facts stripped of context can look supported when they are not. About 6% of claims may cause localization errors in their synthetic test. Subclaim counts per biography vary 20.2–32.9 by method (cited from Wanner et al.). \"Molecular\" units balance context and minimality.\n- **Dey and Collomosse (Jun 2026), \"The Warrant Gap\", arXiv 2606.24627.** PP. https://arxiv.org/abs/2606.24627 · Read: ABSTRACT. Naive decomposition costs up to 27.6 accuracy points in some cells because facets lose claim context. Claim-conditioned re-scoring recovers it. An NLI check of whether cited evidence licenses the claim has AUC 0.92.\n- **Press (above)** on composition.\n- **Sinha, Grunde-McLaughlin, MAKER** (above) on context and independence.\n\n---\n\n## 4. Crowdsourcing microtasks\n\n- **Bernstein et al. (2010), Soylent.** PR: UIST 2010. https://hci.stanford.edu/publications/2010/soylent/soylent-uist2010.pdf · Read: FULL (selective). Find-Fix-Verify.\n  - Find: workers flag patches; a patch is kept if at least 20% agree. Fix: others rewrite. Verify: different workers vote, five of them with majority rule, and Fix workers are barred.\n  - Why split: Lazy Turkers fix only the easiest problem, so Find+Fix together gives poor coverage. Eager Beavers over-edit and introduce errors.\n  - Rule of thumb: about 30% of raw open-ended output is poor. 37 of 126 shortening suggestions were ungrammatical; the Human Macro task averaged 70.8% accuracy.\n  - Verify caught 19 of the 37 errors (about half) and also removed 15 good ones. The paper reports a 26.1% error rate for the step.\n  - Crowdproof: 28 of 62 suggested corrections (45%) were ungrammatical, yet 29 of 33 flagged errors ended up fixed (88%). Authors: people vet better than they author (anecdotal).\n  - What went wrong: separate workers' cuts did not match (one cut the parenthetical, another the main phrase). Minor errors in the same patch as a worse one were missed. Domain context was missing (\"GUI\").\n- **Kittur, Chi, Suh (2008).** PR: CHI 2008. https://dl.acm.org/doi/10.1145/1357054.1357127 · Read: EXTRACT (long body text).\n  - Experiment 1: 48.6% invalid free-text responses, 30.5% finished in under a minute. Eight users produced 73% of the flagged answers.\n  - Experiment 2 added four quantitative, checkable questions before the judgment (counts of references, images, sections, plus keywords). Invalid dropped to 2.5%, under-a-minute to 6.5%, median time rose 1:30→4:06, and agreement with expert ratings reached r=0.66.\n- **Little et al. (2010), iterative vs parallel.** PR (workshop). https://dl.acm.org/doi/10.1145/1837885.1837907 · Read: EXTRACT.\n  - Iteration raised average quality in writing and brainstorming (significant), not clearly in transcription (65% vs 62%, p=0.73).\n  - The best brainstorm names came from solo workers. Early wrong guesses propagate. They suggest parallel branches first, then iterating on the best.\n- **Kittur et al. (2011), CrowdForge.** PR: UIST 2011. https://smus.com/crowdforge/crowdforge-uist-11.pdf · Read: FULL (selective). Partition, map, reduce.\n  - Article quality: group-written 4.01 vs 3.75 individually written (p=.05), with Simple English Wikipedia at 3.95. Variance was lower. Cost was about $3.26 per article.\n  - One bad partition (outline) cascaded down the chain. A merge step helped: merged outlines were preferred 61% to 39%, and none fell below 35% preference whereas 7 initial outlines did.\n  - Science journalism: a news-lead task was too big a chunk until \"consolidate\" and \"exemplar\" patterns were added. Lead ratings ranged 6.25 to 1.25 on a 7-point scale (SD 1.3). None of the leads or descriptions mentioned the paper's key unpredictability finding.\n  - Stated limit: work may not split into pieces small enough for the workforce yet independent enough to coordinate.\n- **Chilton et al. (2013), Cascade.** PR: CHI 2013. https://www.microsoft.com/en-us/research/publication/cascade-crowdsourcing-taxonomy-creation/ · Read: ABSTRACT. Workers spend as little as 20 seconds each; quality 80–90% of experts on three datasets; six times the human labour, competitive cost.\n- **Cheng et al. (2015), \"Break It Down\".** PR: CHI 2015. http://teevan.org/publications/papers/chi15-micro.pdf · Read: FULL.\n  - 110 people, three tasks with clear correctness criteria (receipt arithmetic, sorting, transcription).\n  - Microtasks took longer (uninterrupted arithmetic about 42 seconds longer). They gave fewer errors (error-rate coefficient −0.05, p<.001 against 0.10 for the macro arithmetic baseline). 77% found them easier.\n  - Granularity caveats: transcription chunks of about 6 seconds kept enough context. Sorting 3 lines may not benefit. Many complex tasks lack an obvious structure.\n  - This is not a \"too small is bad\" result. The cost was time.\n- **Snow et al. (2008).** PR: EMNLP 2008. https://aclweb.org/anthology/D08-1027 · Read: EXTRACT. About 4 non-expert labels per item matched expert quality on an affect task.\n- **Oleson et al. (2011), Programmatic Gold.** PR: HCOMP 2011. Read: EXTRACT. Automatically generated gold questions give targeted feedback and cut manual quality management. No numbers retrieved.\n- **Retelny, Bernstein, Valentine (2017).** PR: CSCW 2017. https://dl.acm.org/doi/10.1145/3134724 · Read: ABSTRACT (via search). 22 workers, 6 workflow-based teams on web development. Workflows coordinated but blocked real-time adaptation, so complex interdependent work may stay out of reach.\n- **Kulkarni et al. (2012), Turkomatic.** PR: CSCW 2012. https://people.eecs.berkeley.edu/~bjoern/papers/kulkarni-turkomatic-cscw2012.pdf · Read: EXTRACT. Unsupervised crowd planning was partly successful; requester intervention improved quality substantially.\n- **Grunde-McLaughlin et al. (2024)** (see section 2) ports Cascade, Find-Fix-Verify and Mechanical Novel to LLM chains.\n  - Adaptations: finer subtasks, a different prompt per parallel generation step to force diversity, comparative or numbered-choice evaluation, and hard-coded validation.\n  - Common limits: context length, inconsistent output format, little diversity in responses, and models struggling with workflow subtasks.\n  - Workflows cost more. They can struggle with coherence.\n\n---\n\n## 5. Agreement without ground truth\n\n- **Rabinovich et al. (2023), \"Predicting QA Performance through Semantic Consistency\".** PR: GEM workshop at EMNLP 2023. https://aclanthology.org/2023.gem-1.12.pdf · Read: EXTRACT.\n  - Robustness of answers to question paraphrase correlates with correctness. A predictor using only consistency and sampled-answer certainty, with no outside knowledge, reached prediction accuracy of 0.902 and 0.781 in two settings (I could not see which settings).\n  - Outlier: the \"religion\" category had low correctness but high consistency.\n- **Coherence in QA (2025).** PR: NAACL 2025. https://aclanthology.org/2025.naacl-long.588.pdf · Read: EXTRACT. Coherence is a property of the model, not necessarily related to its accuracy; the most coherent model was not the most accurate.\n- **Cheng et al. (2026)** (section 1): single-prompt correctness is a poor indicator. Aggregate stability hides instance flips.\n- **Kuhn et al. (2023), Semantic Uncertainty.** Venue not checked. https://arxiv.org/abs/2302.09664 · Read: ABSTRACT. Entropy over meaning-clusters of sampled answers predicts accuracy better than comparable baselines.\n- **Farquhar et al. (2024), Nature 630.** PR. https://www.nature.com/articles/s41586-024-07421-0 · Read: EXTRACT (search summary only). Same method; works across LLaMA, Falcon and Mistral 7B–70B. No numbers (full text not reachable).\n- **Arora et al. (2023), Ask Me Anything.** PR: ICLR 2023. https://arxiv.org/abs/2210.02441 · Read: EXTRACT. Many differently worded prompts give noisy votes; weak supervision combines them while modelling dependencies. +10.2% average over few-shot; GPT-J-6B matched or beat few-shot GPT-3-175B on 15 of 20 tasks.\n- **Verga et al. (2024), PoLL.** Venue not checked. https://arxiv.org/abs/2404.18796 · Read: EXTRACT. A panel of smaller models from different families beat a single large judge across 6 datasets at over 7× lower cost.\n- **Kohli (May 2026), \"Nine Judges, Two Effective Votes\", arXiv 2605.29800.** PP (Apple). https://arxiv.org/abs/2605.29800 · Read: PARTIAL (abstract, intro, related work).\n  - 9 frontier LLMs from 7 families on 3 NLI sets with 100 human labels per item. The panel carries only about 2.0–2.5 independent votes' worth.\n  - Panel accuracy is 8–22 points below independent voting. The best single judge matches or beats the panel.\n  - Dawid–Skene and accuracy-weighted voting close at most 11% of the gap, even with gold labels.\n  - Unanimous agreement is far less diagnostic than it looks. The paper cites Kim et al. 2025 (models agree on wrong answers 60% of the time on some benchmarks).\n- **Jiang et al. (2025), \"Artificial Hivemind\".** PR: NeurIPS 2025 (best-paper award per search listing). https://neurips.cc/virtual/2025/poster/121421 · Read: EXTRACT. In open-ended generation models repeat themselves (within a model) and converge across families, so agreement there reflects a shared prior.\n- **Grunde-McLaughlin et al. (2024), section 8.1.** FULL (selective). It is unclear whether consistency within one LLM is a proxy for accuracy or whether you need consistency across varied prompts or models. Agreement among workers may not transfer to LLMs.\n- **Han (2026)** and **Messing (2026)** (above): correlated gates cap reliability; unmeasured wording variance understates uncertainty.\n- **Niven and Kao (2019).** PR: ACL 2019. https://arxiv.org/abs/1907.07355 · Read: ABSTRACT. BERT's 77% on warrant selection (ARCT) came entirely from spurious cues; on an adversarial set all models were at chance.\n- **Grading the Graders (Aug 2026), arXiv 2608.19009.** PP. https://arxiv.org/abs/2608.19009 · Read: EXTRACT. Sampling and substitution verifiers can confirm that proposed candidates hold but cannot show that none was missed (\"completeness blind spot\"). This matches your finding (b) conceptually.\n\n**Net.** Measured link between self-agreement and correctness exists for short factual answers (Rabinovich, Kuhn/Farquhar). Consistent-but-wrong cases are documented. Cross-model agreement is inflated by correlated errors. I found no measurement for open-ended generated premises.\n\n---\n\n## 6. Tools that surface assumptions for a human reader (2025–2026)\n\n- **Ghosh, Sarkar, Lindley, Poelitz (Jan 2026), arXiv 2601.18033.** PP. https://arxiv.org/abs/2601.18033 · Read: ABSTRACT plus the HTML via an extraction tool.\n  - n=214, plus 12 interviews. Reviewing AI-written plans for writing tasks.\n  - The \"Assumptions\" condition (a forcing function for argument analysis) cut overreliance most, with the lowest cognitive load of the active conditions. Combining it with a what-if prompt was no better than control. Effect sizes were small to medium.\n  - The reader had to infer which assumption the AI made, by multiple choice or free text, at least one required. It was not a passive list.\n  - Users rated the what-if prompt as more helpful, which was not the one that worked.\n- **Buçinca, Malaya, Gajos (2021).** PR: CSCW 2021. https://arxiv.org/abs/2102.09692 · Read: ABSTRACT. N=199. Cognitive-forcing designs reduced overreliance relative to plain explanations. They were also the least liked.\n- **Vasconcelos et al. (2023).** PR: CSCW 2023. https://arxiv.org/abs/2212.06823 · Read: ABSTRACT (truncated). The abstract says explanations did not reduce overreliance in prior work. Across 5 studies (N=731) they reduce it when they cut the cost of verifying.\n- **Lee et al. (2025).** PR: CHI 2025. https://dl.acm.org/doi/abs/10.1145/3706598.3713778 · Read: EXTRACT. Survey: higher confidence in GenAI goes with less critical thinking.\n- **Drosos et al. (Jan 2025), \"It makes you think\".** PP (venue not checked). https://arxiv.org/abs/2501.17247 · Read: ABSTRACT. n=24. Short provocations (critiques and alternatives) induced critical and metacognitive thinking.\n- **Blanchard et al. (Jun 2026), arXiv 2606.23491.** PP. https://arxiv.org/abs/2606.23491 · Read: ABSTRACT. Review of hallucination-detection studies. General warnings are the weakest and most mixed intervention. Targeted cues about where AI fails look promising but evidence is thin.\n- **Thakkar et al. (2025), ICLR 2025 randomized study.** PP (a Nature Machine Intelligence version is listed at nature.com/articles/s42256-026-01188-x; not opened). https://arxiv.org/abs/2504.09737 · Read: ABSTRACT.\n  - An LLM feedback agent to over 20,000 reviews. Feedback was sent only if it passed LLM-run reliability tests.\n  - 27% of reviewers who received feedback updated their review, and over 12,000 suggestions were incorporated. Updated reviews were longer (+80 words) and more informative, as judged by blinded researchers. Rebuttals were more engaged.\n- **Worledge et al. (Jul 2026), \"AI Assistance for Human Review of Default Judgments\", arXiv 2607.01256.** PP. https://arxiv.org/abs/2607.01256 · Read: ABSTRACT.\n  - Audit of 188 court cases: 4% had major defects, 10% inconsistencies, 32% errors needing amendment.\n  - An LLM assistant with recommendations grounded in cited quotes. 66 law students: 6.0% more accurate and 25.9% faster than unaided reviewers. Largest gains where document search was needed: errors cut by up to 62%, time by up to 34%.\n- **Schwarcz et al. (2026), \"AI-Powered Lawyering\".** PR: SAGE journal, Apr 2026. https://journals.sagepub.com/doi/10.1177/2755323X261427048 · Read: ABSTRACT.\n  - RCT with law students. Both tools improved work quality. Productivity gains 50–130% on five of six tasks. The RAG tool did not add hallucinations; the reasoning model improved analytical depth.\n  - A search summary says no gain on the \"Accuracy\" criterion and more hallucinations for o1-preview. I did not confirm this in the paper.\n- **Bastani et al. (2025).** PR: PNAS 122(26). https://www.pnas.org/doi/10.1073/pnas.2422633122 · Read: EXTRACT (the correction notice I opened is affiliation-only).\n  - Practice performance +48% (GPT Base) and +127% (GPT Tutor). Unaided exam performance 17% lower after access was removed. My source does not say which arm the 17% is for.\n- **Graphionale (Wang, Ma, Yin, Ma, Malone; Aug 2026), arXiv 2608.27932.** PP, with a related ACM DOI. https://arxiv.org/abs/2608.27932 · Read: ABSTRACT.\n  - Argument-map-style rationale graphs, N=204. Graphs improved trust calibration for verbal reasoning but felt more demanding. For visual reasoning they impaired calibration. The format that supported calibrated decisions was not the preferred one.\n- **\"Confidence Without Competence\" (Apr 2026), arXiv 2604.09444.** PP. https://arxiv.org/abs/2604.09444 · Read: ABSTRACT. A standard answer-giving LLM produced high perceived understanding with the lowest objective learning; guided hints gave the largest learning gains.\n- **Error verifiability (2026)** (section 2): justifications often do not help people separate right from wrong.\n- **Rubber-stamping evidence.** Buçinca and Vasconcelos (explanations alone did not reduce overreliance in the earlier work they cite), Lee (self-reported effort drops), and the error-verifiability and Blanchard items. Your held \"detect-but-defer\" paper covers the rest.\n\n**Net.**\n- Evidence for helping readers is strongest when the reader must act on an assumption (Ghosh), when evidence is cited as quotes (Worledge), or when a gate screens feedback (Thakkar).\n- I found no test of a machine-written list of assumptions that people only read.\n\n---\n\n## Design rules with a measurement behind them\n\n(D) = direct measurement. (I) = my inference from the numbers.\n\n1. **Check both orders and accept only agreement; mark disagreement \"undetermined\".** Three evidence × three orders beat six evidence × one order (GPT-4 62.5 vs 60.9; ChatGPT 58.7 vs 55.6). Sending the 20% most order-unstable items to humans raised GPT-4 to 73.8% (human mean 71.7%). Repeating the same prompt cannot see this: repetition stability above 0.95 coexists with position consistency of 0.57–0.82. Wang 2024; Shi 2025. (D) For two premises, both orders are exhaustive.\n2. **Do not rely on \"ignore order\", chain of thought, or few-shot examples to remove order effects.** RStd 6.1 and 4.5 vs 5.5 default (Zheng). The ignore-order instruction left flips (Wang). Reordered few-shot examples: mostly no help, best case still −13 points (Chen ICML). (D)\n3. **If several premises must be shown, list them in derivation order.** GPT-4-turbo at 12 rules: 96.5 forward vs 80.8 shuffled. With 6 or fewer rules the effect is near zero for strong models and large for weak ones. Context position is not the cause (3.5 points or less). Chen 2024. (D)\n4. **Prefer answers bound to content over answers bound to a letter or label.** Removing option IDs cut the bias spread 5.5→1.0 but cost accuracy under likelihood scoring. Zheng 2024. (D for multiple choice; I for sentence pointing.)\n5. **Measure acceptance rate under crossed wording × order × label, not only accuracy.** Leniency moved 28.9 points (detailed prompt) and up to 56.1 points (model switch) while accuracy barely moved (Vaugrante). Yes/no bias splits into order and word parts, family-specific (Huang). A polarity-inverted template explained most of the factuality inconsistency (0.63→0.9 or higher; JudgeSense). Check that your two wordings are not polarity-inverted before reading 4-vs-20 refusals or 18 side-changes. (D; JudgeSense single-author PP)\n6. **Do not show the checker the draft or the claimed answer; ask open questions, not yes/no.** Factored vs joint: FActScore 63.7 vs 60.8; MultiSpanQA F1 0.48 vs 0.46; Wiki-Category precision 0.22 vs 0.15. Open vs yes/no 0.22 vs 0.19. Quoting others' judgments anchors by 19–26 points (PP, VLM judges). Dhuliawala 2024; Shu 2026. (D)\n7. **Expect only small extra gain from masked-condition or backward checks over a good forward baseline.** +0.32 on GSM8K over self-consistency (Weng); FOBAR +2.1 / +1.1 / +0.3 points over self-consistency for davinci / GPT-3.5 / GPT-4. They need a maskable condition. (D)\n8. **Do not let the generator decide whether optional content is needed.** Make \"needed?\" a separate gate settled by independent agreement. Soylent kept a patch only if at least 20% of workers flagged it. Lazy workers take the easiest fix; combining find and fix gave poor coverage; Eager workers over-edit. (D for people; I for your 27 of 45.) Soylent 2010.\n9. **Separate generating from verifying, never self-verify, and budget for noisy verification.** Soylent barred Fix workers from Verify, yet Verify caught only 19 of 37 errors and removed 15 good items. GPT-4 self-critique collapsed while a sound external verifier helped (Stechly). LLMs validate less reliably than crowd workers (Grunde-McLaughlin). (D)\n10. **Make the checked thing code-checkable, and ask for a verifiable sub-answer before the judgment.** Verifiable questions cut invalid responses 48.6%→2.5% (Kittur 2008). Cited quotes: +6.0% accuracy and 25.9% faster (Worledge). Programmatic checks were the most reliable in LLM chains (Grunde-McLaughlin). Tool-grounded checks agreed 72% with crowd labels (SAFE). (D)\n11. **Keep each task's context minimal and free of earlier model output.** Models self-condition on their own earlier errors, and clearing history helps (Sinha). Extra prior suggestions lowered validation quality (Grunde-McLaughlin). Answers over about 700 tokens went from 1-in-1000 to about 10% error (MAKER). (D)\n12. **Voting over repeated samples works only with independent errors and exact-match outputs.** Formula: worst-case selection probability 1/(1+((1−p)/p)^k). At p=0.99 over 10^6 steps, k=3 gives about 36% overall success and k=4 about 99% (my arithmetic). Nine judges gave about 2 independent votes (Kohli). Correlated gates decay polynomially and cap below 1 (Han). Open-ended premises need a semantic-equivalence matcher, which MAKER lists as future work. Your \"told a link exists\" lean is systematic, so extra identical samples will not remove it (I). Red-flagging over-long or malformed output cut collisions (D, MAKER).\n13. **Do not cut below the unit that carries its own meaning.** With a strong verifier, atomic decomposition lowered F1 from 72.32 to 59.90 (Hu). Context-free atomic facts can look supported when they are not (Gunjal). Naive decomposition cost up to 27.6 points (Warrant Gap, PP). Separate workers' cuts did not match (Soylent). A single bad upstream partition cascaded (CrowdForge). (D)\n14. **Check each step when the reasoning is the product.** Reasoning error among answer-correct solutions: 14.0%→3.4% with process supervision, while final-answer error was similar either way (Uesato). Step reward model 78.2 vs outcome 72.4 vs majority vote 69.6 (Lightman). LLM step checkers over-flag: best location accuracy 52.87% (Tyen). Budget for false alarms. (D; applying it to your case is I)\n15. **Collect independent answers first and show them to each other later.** About 4 non-expert labels matched an expert (Snow). A merge step beat single outlines 61% to 39% (CrowdForge). Showing earlier guesses misled later workers (Little). Checking against blind votes blocked 84.9% of forgeries (Shu). Aggregating judge runs improves agreement with humans (Rating Roulette). (D)\n16. **Treat agreement as triage, not proof.** Paraphrase consistency predicts correctness for short factual answers (Rabinovich), but consistent-but-wrong cases exist (Rabinovich; Coherence NAACL 2025). A high benchmark score on warrant selection came from artifacts (Niven and Kao: 77%, chance on adversarial data). Unanimity among nine judges is weak evidence (Kohli). Route disagreement to a person. (D)\n17. **Report the wording and judge variance, calibrated on a small human-labelled set.** Naive standard errors were 40–60% too small (Messing). Routing 20% to humans restored human-level accuracy (Wang). (D)\n18. **For the human reader, make them act on an assumption and attach the quoted sentence; do not just display a list.** The \"Assumptions\" forcing task cut overreliance most and combining prompts did not help (n=214). Forcing designs worked but were least liked (N=199). Explanations help only when they lower verification cost (N=731). General warnings are the weakest intervention. Readers preferred the format that did worse (Ghosh; Buçinca; Vasconcelos; Blanchard; Graphionale). (D; Ghosh and Graphionale are PP)\n\n---\n\n## What I looked for and did not find\n\n- A measured fix for premise-order effects by asking each premise separately, or by canonical ordering, on 2025–26 models. Not found. Chen leaves mitigation to future work; its few-shot attempt mostly failed. No follow-up measuring a mitigation turned up in my searches.\n- Any study of LLM-generated implicit premises (enthymeme completion) that measures repeatability under rewording or reordering. Not found.\n  - Closest: Feng and Hunter (COMMA 2026, arXiv 2608.18821, abstract only, no numbers) and Feng and Bikakis (arXiv 2608.18820, abstract only). The latter selects between two supplied candidates and raises strict accuracy by 2.95–30.86 points and cuts tie rates by 4.57–58.00 points. It tests selection, not generation stability.\n- A matched test of \"told the link holds\" vs \"not told\" for argument links. Not found. Closest: Sharma (user belief cues), the premise-critique benchmark you hold, Shu 2026 (anchoring, vision judges), CoVe (yes/no anecdotes), and Grading the Graders (conceptual).\n- An empirical curve for how an LLM step-checker's detection and false-alarm rates combine over chain length. Not found. I have only the p^n formulas, the MAKER voting formula, Han's synthetic theory, and Tyen's per-task table.\n- A measured link between \"the same generated premise recurring across many items\" and validity. Not found. Closest: Artificial Hivemind (repetition and convergence in open-ended generation), Niven and Kao (cue exploitation).\n- Evidence that people reading a machine-written list of assumptions judge better. Not found. What exists is forcing functions where the reader supplies the assumption (Ghosh), cited quotes (Worledge), and gated feedback (Thakkar).\n- Any study of assumption-surfacing in policy analysis. Not found. Education: only general guardrail results (Bastani, Confidence Without Competence).\n- Numbers for Farquhar et al.'s AUROC (login redirect), Verga PoLL (kappa), AMA (details), Oleson (gold-question effect sizes) and Snow (details). I have summaries only.\n- Not opened beyond abstract or extract: Thakkar and its Nature Machine Intelligence version, Worledge, Schwarcz, Vasconcelos (truncated), Cascade, Kuhn, Uesato, Stechly, ProcessBench, JudgeSense, Han, Huang, Shu, Dey and Collomosse, Variation in Verification.\n- Venues not checked: Plan-and-Solve, Press, Stechly, Uesato, Kuhn, Verga, ProcessBench, Drosos.\n- The \"Odds Law\" paper that Han cites. Not located.\n\n---\n\n## Cross-cutting reading (my inference, not a source claim)\n\n- Microtask-plus-voting is shown to work where each step has one exact answer, a strong per-step success rate and independent errors (MAKER). The one test of extending it to open-ended steps is a preliminary appendix on multiplication.\n- Your steady tasks (closed question, refusal gate first, pointing at a sentence, no degrees) match what holds up in the literature: checkable output, minimal context, separate gate and generator.\n- Your unsteady ones sit on the weakest ground: optional slots decided by the generator, existence-style questions (a verifier can always find some candidate), and two-premise order. Measured fixes exist for the last two, not for existence-style framing beyond hiding the target.\n- The crowd literature also says a decomposition is only as good as its seams. The measured failures are lost context at cuts, one bad upstream piece cascading, and a verify step that is itself noisy.\n"}