{"path":"research/one-inference-per-link/report-what-is-hard-per-kind.md","content":"# Research report: what is hard per kind of inference, and which methods measurably help\n\n*A research helper's report (Sonnet), 2026-10-24, saved verbatim for `../one-inference-per-link.md` § 8. Its brief: for deductive, normic, statistical, inductive, abductive and analogical inference, what is measurably hard for current models and untrained people, and what method measurably helps. Read-status tags are the helper's own.*\n\n---\n\n## Per-kind evidence on stringency: what is hard for LLMs and untrained humans, and which methods measurably help\n\nRead-status tags: [FULL] full text read; [PART] substantial excerpts of the paper or its results read; [ABS] abstract only; [SEC] secondary source. Almost nothing here is [FULL]. My tools returned excerpts, and tables that exist only as figures were not retrievable, so I say so where it matters. Preprints are flagged.\n\n### Bottom line on the founder's view\n\nThe evidence for \"minds need education and method\" is strong for statistical and methodological reasoning in humans, mixed for deduction, and indirect for the other four kinds. Three qualifiers apply:\n\n1. **The method has to be rule-plus-worked-example, in a field that practises the rule.** Abstract formal-logic training alone transfers poorly. Chemistry graduate training, which never exercises probabilistic rules, produced no change in statistical, methodological or conditional reasoning.\n2. **General instruction in \"critical thinking\" has only a small average effect (g = 0.30), and effect sizes are larger for narrow, rule-specific training.**\n3. **For LLMs the methods that measurably help mostly work by adding an external checker** (an interpreter, a solver, a computed Bayes step). The model still writes the candidate. A prose procedure the model has to follow on its own helps much less.\n\n### 1. Human training evidence\n\n**Statistical reasoning (law of large numbers).**\n- Fong, Krantz & Nisbett 1986, *Cognitive Psychology* 18:253–292 [PART; excerpts of the paper and the Nisbett et al. 1987 *Science* summary].\n  - Experiment 1: control and demand groups used statistical principles on 42% and 44% of 15 everyday problems. Both rule training and example training raised this. Training on both did best. The rule-versus-examples difference was not reliable.\n  - Experiment 2: 53% statistical answers for controls against 72, 81 and 79% for the three example-training groups. Quality of statistical answers was .47 for controls against .70, .70 and .66.\n  - Nisbett et al. 1987 summarise rule training as raising statistical answers from 42 to 56% and high-quality answers from 54 to 67%.\n  - Training in one domain raised performance on untrained domains as much as on the trained one when tested immediately. Experiment 3 found a large advantage for people with several years of statistics. A fourth experiment tested outside the training context.\n- Fong & Nisbett 1991, *J Exp Psychol: General* 120:34–45 [PART]. A 15-minute example booklet.\n  - Trained-domain performance did not decline over 2 weeks.\n  - Untrained-domain performance declined significantly but stayed above controls, with an effect size d = 0.44.\n  - Retention tracked the ability to state the abstract rule, not memory of the examples.\n- I could not retrieve the exact sample sizes per condition, or the later replications (Kosonen & Winne 1995; Beattie 1995, both listed in [SEC] search results only).\n\n**Graduate training by field.** Lehman, Lempert & Nisbett 1988, *American Psychologist* 43:431–442 [PART].\n- Design: first- versus third-year students at one university (213 of 241 law, 127 of 133 medicine, 25 of 27 psychology and 31 of 32 chemistry students at baseline). Cross-sectional plus longitudinal.\n- Psychology and medicine produced \"large effects\" on statistical and methodological reasoning. Psychology cross-sectional t(56) = 5.00; medicine t(173) = 2.73. Law and chemistry showed no significant change.\n- Conditional-logic reasoning improved in law, medicine and psychology in the longitudinal study (p < .05 to .002) but not in chemistry. The cross-sectional differences were not significant.\n- A second sample of psychologists showed a 59% improvement over two years for social-science psychologists, and none for natural-science psychologists or chemists. The authors warn this was one institution with a small n.\n- Their explanation: training helps where the field teaches and practises the rules. Law drills permission and obligation logic. Chemistry's early years deal with deterministic, necessary-and-sufficient causes and do not teach statistics.\n\n**Undergraduate training.** Lehman & Nisbett 1990, *Developmental Psychology* 26:952–960 [PART].\n- Social-science training produced large gains in statistical and methodological reasoning. Natural-science and humanities majors improved less, only marginally significantly.\n- Natural-science and humanities training produced large gains in conditional logic. Social-science training did not.\n- The natural-science gain tracked the number of mathematics courses taken. The authors read this as proof by contradiction acting as a pragmatic schema.\n- The authors report 60 to 70% improvement on average in the favoured domains, against verbal-reasoning gains of about 4 to 17%.\n- A secondary source on Nisbett's work [SEC] reports that formal logic courses \"do nothing\" for conditional reasoning. I could not see the primary figure.\n\n**Deduction and schemas.**\n- Cheng & Holyoak 1985 [PART]. On abstract Wason selection tasks, under 10% of college students choose the correct cards. A postal-rule thematic version gave 81% against 15% on the card version in an earlier study (Johnson-Laird et al.), but that facilitation was inconsistent across populations. Their permission-schema versions raised performance.\n- Cheng, Holyoak, Nisbett & Oliver 1986, *Cognitive Psychology* 18:293–328 [ABS and SEC only].\n  - Abstract logic principles alone did little.\n  - Instruction coupled with selection-problem examples helped, in lab and classroom.\n  - Brief training on a pragmatic permission or obligation schema had a substantial effect on schema-interpretable problems.\n  - A secondary summary says formal-rule teaching gave no gain even immediately.\n- Noveck & O'Brien (Q J Exp Psychol, no year given in my source) [ABS] argue that the scope of pragmatic-schema theory needs narrowing. Permission rules help, and obligation rules did not in their studies.\n\n**Teaching Bayesian reasoning.**\n- Sedlmeier & Gigerenzer 2001, *J Exp Psychol: General* 130:380–400 [ABS plus SEC]. Representation training (build a frequency tree) beat rule training (plug into Bayes's rule) on immediate learning and on retention. Transfer was roughly equal.\n- A secondary source (Martignon & Wassner, ICOTS 2002) reports over 90% long-term performance with frequency trees against about 20% for formula training. I did not see this in the original, so treat it as [SEC].\n- The often-quoted Gigerenzer & Hoffrage 1995 jump from probability to natural-frequency format: I could not verify specific percentages and do not quote any.\n\n**Analogy and induction (human method).** Loewenstein, Thompson & Gentner 1999, *Psychon Bull Rev* 6:586–597 [PART].\n- Management graduate students who compared two cases (analogical encoding) were nearly three times as likely to use the strategy in a live negotiation as students who received the same two cases separately.\n- Across both experiments, comparison gave at least double the transfer rate.\n- Comparing cases beat receiving a principle plus an example in the authors' related work.\n- Even experienced, motivated participants showed low structural transfer without explicit comparison.\n\n**General critical-thinking instruction (meta-analyses).**\n- Abrami et al. 2015, *Rev Educ Res* 85(2):275–314: 341 effect sizes, weighted mean g = 0.30 [SEC; via a Stanford Encyclopedia supplement and a TUM summary]. A combination of application-oriented instruction, dialogue and mentoring gave g = 0.57 [SEC].\n- Abrami et al. 2008, *Rev Educ Res* 78(4): g+ = 0.341 over 161 effects [SEC].\n- I did not open either paper.\n\n### 2. LLMs per kind, 2023–2026\n\n**Deductive.**\n- Dasgupta, Lampinen et al., PNAS Nexus 2024 [PART]. Humans and models both say a syllogism is valid when the conclusion is believable, regardless of actual validity. Exact accuracy cells are in figures I could not retrieve. Natural-language inference showed no significant content effect. Few-shot examples gave \"mild improvements\" and did not remove content effects. The excerpt I read did not test chain-of-thought.\n- NeuBAROCO (arXiv 2408.04403) [ABS, preprint]. The authors' conclusion is that LLM limitations lie in the reasoning process itself rather than in interpreting the syllogisms.\n- SymbCoT (Xu et al., arXiv 2405.18357) [PART, results table read; the one retrieved page did not give the venue]. GPT-4 accuracy:\n\n| Method | ProntoQA | ProofWriter | FOLIO | LogicalDeduction | AR-LSAT |\n|---|---|---|---|---|---|\n| Naive | 77.4 | 52.7 | 69.1 | 71.3 | 33.3 |\n| CoT | 98.8 | 68.1 | 70.6 | 75.3 | 35.1 |\n| Logic-LM | 83.2 | 79.7 | 78.9 | 87.6 | 43.0 |\n| SymbCoT | 99.6 | 82.5 | 83.3 | 93.0 | 43.9 |\n\n  SymbCoT beats CoT by +9.31 on the first-order-logic average and +13.32 on the constraint-optimisation average. With GPT-3.5 it did worse than Logic-LM on FOLIO (57.8 against 62.7). In the ablation, removing planner and solver together dropped GPT-4 by about 10.4 points and removing the translator about 6.3.\n\n**Defeasible and exception-bearing.**\n- Newer than the corpus already covered: Liu et al. 2025 (Princeton; a preprint, I did not confirm a venue) [PART]. Inference-time reasoning hurt on \"classifying with patterns containing exceptions\": up to 36.3 points absolute accuracy lost by o1-preview versus GPT-4o (zero-shot) on a 440-problem subset. CoT also cost GPT-4o 23.1 points and the other models 6 to 9. This is one of three settings where deliberation hurts humans too, so it is evidence that \"more explicit reasoning\" is not uniformly better for normic classification.\n\n**Statistical (base rates and posteriors).**\n- \"Dual Traits in Probabilistic Reasoning of LLMs\" (arXiv 2412.11009, Dec 2024) [PART, preprint]. On the Jason (lawyer–engineer) test, for a description that is uncharacteristic of the group, four of six models (gpt-4o, gpt-4, claude-3-opus, llama-3-70b) gave a normative posterior-odds ratio of about 9. For a representative description the ratio was about 1 to 1.7 for most models (gpt-4o 1.25, gpt-4 1.18), against a normative value of 9. The human column quotes about 2 for this condition. The authors also report that changing a single word in the prompt (\"indicate\" to \"compute\") could push models into step-by-step Bayes, but in an unstructured test GPT-4o still mostly reasoned by resemblance. GPT-4o recalled priors inconsistently (2–4% when asked separately, 10–30% when reasoning).\n- Qiu et al., \"Bayesian Teaching Enables Probabilistic Reasoning in LLMs,\" Nature Communications, Jan 2026 [PART]. Off-the-shelf LLMs update beliefs far less than a Bayesian assistant, and their accuracy \"often plateaus after a single interaction.\" Fine-tuning on the Bayesian assistant's outputs beats fine-tuning on an oracle's, with fine-tuned models matching the Bayesian assistant about 80% of the time and generalising to hotel and web-shopping tasks. Exact accuracy numbers are in figures I could not see. This is user-preference recommendation, not word problems.\n- \"Do Chains-of-Thought of LLMs Suffer from…Bayesian Reasoning?\" (arXiv 2503.15268, Mar 2025) [PART, preprint]. Three reasoning models used symbolic probability by default and avoided natural frequencies, and applied them inconsistently even when prompted. The conclusion is a qualitative claim from a few examples.\n- Macmillan-Scott & Musolesi, *R Soc Open Sci* 2024 [ABS and SEC]: seven models show irrationality unlike human patterns and inconsistent answers, and did not improve with extra context. GPT-4 did best.\n- Reference class: I found no measured LLM study of reference-class choice. See \"not found\" below.\n\n**Inductive.**\n- Qiu et al. ICLR 2024 [PART; Table 1 read]. GPT-4, raw / task accuracy:\n\n| Method | ACRE | MiniSCAN | List Fns | MiniARC |\n|---|---|---|---|---|\n| IO prompting | 64.0 / 28.0 | 61.7 / 0.0 | 65.1 / 39.6 | 33.1 / 13.8 |\n| Hypothesis refinement (T=3, N=5, symbolic interpreter) | 82.5 / 59.0 | 93.3 / 85.0 | 71.2 / 61.2 | 18.7 / 14.6 |\n\n  Using the LM itself as the interpreter to apply its own rules gave 77.8 / 47.0 on ACRE, 67.6 / 0.0 on MiniSCAN, 65.8 / 50.0 on List Functions and 10.8 / 5.4 on MiniARC. On MiniARC, hypothesis refinement without an interpreter was worse than IO (15.1 / 9.2 for self-refine with an LM interpreter, against 33.1 / 13.8 for IO). With noisy exemplars on List Functions (T=1, N=1), accuracy fell from 51.6 / 42.4 at 0% noise to 23.1 / 12.4 at 50%.\n- Wang et al., \"Hypothesis Search,\" ICLR 2024 [PART]. On 100 ARC problems: direct prompting 17%, program-only 23%, LLM-summarised hypotheses 30%, human-selected hypotheses 33%, human-written 45%. GPT-4 produced a correct hypothesis for only 49 of 100 tasks. (The ICLR abstract version reports a 40-problem subset with 12.5% against 27.5%.)\n\n**Abductive.**\n- RECV, ACL Findings 2025 [PART]. Average error rates across models and prompts: VitaminC 10.3% deductive against 32% abductive; CLIMATE-FEVER 15.6% against 48.6%; PHEMEPlus 20.1% against 44.7%. Adding rationale generation did not reliably help and cost about 7.5 points on VitaminC overall, with abductive cases suffering more.\n- InAbHyD / \"Language Models Do Not Follow Occam's Razor\" (arXiv 2509.03345, Sep 2025) [PART, preprint]. Above 80% accuracy on height-1 ontologies with a single hypothesis. With multiple hypotheses, accuracy fell from above 80% to below 50% when height went from 1 to 2 (GPT-4o the exception). Models often produce valid but poor-quality hypotheses. R1-distilled Llama-70B beat Llama3-70B by roughly 10 to 20% on average, apparently by checking its hypotheses deductively. 8-shot in-distribution demonstrations helped modestly; out-of-distribution demonstrations did not.\n- Elenchos (arXiv 2607.12733, July 2026) [ABS, preprint]. Frontier models detect that a system was altered but struggle to say which rules changed, and the gap widens when several mutations interact. More inference-time reasoning gave only modest gains. No numbers in the abstract.\n- \"Concept\" board-game benchmark (arXiv 2510.13271) [ABS, preprint]. Humans solve above 90%. No model exceeds 40%. Models fail to revise initial hypotheses when new clues arrive.\n- Black Swan (CVPR 2025) [PART]. Humans beat GPT-4o by 24.9 points on abductive multiple-choice video tasks and 26 points on defeasible yes/no.\n- Original αNLI (Bhagavatula et al.) [SEC]: baselines about 68.9% against humans at 91.4%. Older, but I found no newer αNLI-specific measurement.\n\n**Analogical.**\n- Lewis & Mitchell 2024 (arXiv 2402.08955) [PART]. Letter-string analogies; humans 0.753 (n = 1,876) against GPT-3 0.488, GPT-3.5 0.350 and GPT-4 0.452 across alphabets. Humans stayed roughly constant across permuted alphabets, while GPT models dropped on permuted alphabets and again on symbol sequences. Errors differed from human errors. They found GPT performance on the original problems lower than humans', unlike Webb et al.\n- Webb, Holyoak & Lu reply (arXiv 2404.13070) [PART]. GPT-4 with code execution was on par with humans on counterfactual letter strings (P = 0.496); without code it was significantly worse (P = 8.9×10⁻¹³). Human n = 99 and 97 (two interval sizes). 21 of 52 GPT-4+code errors (40%) were valid alternative rules against 39% for humans. They argue interval-size-2 difficulty reflects counting, not analogy.\n- This is consistent with the pattern in your other results: a code interpreter moved a model from clearly worse to human-level on one analogy task. It is a single study pair, and Lewis and Mitchell have not replicated it to my knowledge.\n\n### 3. Does an explicit method help beyond plain CoT?\n\n| Kind | Method | Measured effect | Source and status |\n|---|---|---|---|\n| Deductive | Translate to symbolic form, plan, solve, verify (SymbCoT) | +9.3 over CoT (first-order-logic average, GPT-4); +13.3 on constraint tasks | Xu et al. [PART] |\n| Statistical | One-word prompt change to \"compute posterior\"; Bayesian-assistant fine-tuning | Normative odds ratio about 9 restored only in structured tests; fine-tuning generalised to new tasks | Dual Traits [PART, preprint]; Qiu et al. 2026 [PART] |\n| Inductive | Propose, filter with an interpreter, refine | Task accuracy 28 to 59 (ACRE), 0 to 85 (MiniSCAN); ARC 17 to 30 | Qiu 2024; Wang 2024 [PART] |\n| Inductive | Human-chosen or human-written hypotheses | ARC 30 to 33 to 45 | Wang 2024 [PART] |\n| Abductive | In-distribution demonstrations; deductive self-checking | Modest gains; no gain out of distribution | InAbHyD [PART, preprint] |\n| Analogical | Code execution | Human-level on letter strings versus clearly worse without | Webb reply [PART] |\n\nNotes:\n- Methods that include an external verifier (interpreter, solver, a computed Bayes step) show the largest and best-documented effects. Methods that are only prose procedures show smaller and less consistent ones. This matches the induction results, where models propose well but cannot apply their own rules.\n- I did not find a clean test of \"list the rival hypotheses explicitly\" for abduction beyond the InAbHyD observation that models often fail to prefer the simplest valid explanation.\n- Whether CoT beats plain prompting depends on the kind: it helps deduction, does not reliably help abductive claim verification, and can hurt classification with exceptions.\n\n### 4. Summary table\n\n| Kind | What is hard (models) | What is hard (untrained humans) | Method that measurably helps | Source |\n|---|---|---|---|---|\n| Deductive (rule applied to a case) | Belief-biased validity judgments; weaker on neither-entails-nor-contradicts items | Abstract conditionals: under 10% correct on abstract Wason; belief bias in syllogisms | Humans: schema-linked instruction, math or law training. Models: symbolic translation plus planning plus verification | Dasgupta/Lampinen; Cheng & Holyoak; Lehman & Nisbett; Xu |\n| Defeasible / normic | Inference-time reasoning hurt classification with exceptions (up to 36.3 points) | I found no human-training study specific to normic rules | None measured here | Liu 2025 (preprint) |\n| Statistical (proportion to case) | Insensitive to base rates when description is representative (odds ratio about 1 against 9); inconsistent recall of priors | Base-rate neglect; weak statistical heuristics in everyday problems | Humans: brief law-of-large-numbers training; frequency-tree representation training. Models: \"compute posterior\" wording, Bayesian-assistant fine-tuning | Fong et al.; Sedlmeier & Gigerenzer; Dual Traits; Qiu 2026 |\n| Inductive (cases to rule) | Applying their own induced rules; brittle to noisy examples | Few-shot rule discovery; inert knowledge without comparison | Humans: compare cases. Models: propose plus interpreter-filter plus refine | Loewenstein et al.; Qiu 2024; Wang 2024 |\n| Abductive | Selecting the simplest explanation; error 32 to 49% on claim verification; revising hypotheses | I found no human-training measurement | Modest: in-distribution demonstrations, deductive self-checking | RECV; InAbHyD; Concept (preprints) |\n| Analogical | Fragile on counterfactual or symbolic variants; errors differ from human ones | Low transfer from single examples; failing to notice parallels | Humans: analogical encoding (about 2 to 3 times transfer). Models: code execution on one task | Loewenstein et al.; Lewis & Mitchell; Webb reply |\n\n### What I looked for and did not find\n\n- Exact effect sizes (d or g) for the Lehman, Lempert & Nisbett 1988 and Lehman & Nisbett 1990 results. I only got t-values, one 59% figure and \"60 to 70%\" in prose.\n- Primary text of Abrami et al. 2008 and 2015. The numbers above are from secondary pages.\n- A verified Gigerenzer & Hoffrage 1995 percentage jump, and the primary Sedlmeier & Gigerenzer numbers. The 20% versus over 90% figure is secondary.\n- Exact syllogism accuracy cells from Dasgupta/Lampinen (figures only) and any chain-of-thought test of content effects in that paper.\n- Any measured LLM study of reference-class choice or statistical syllogism specifically. The reference-class literature I found is philosophical (Roth & Tolbert 2025, Buchholz 2023).\n- Any human-training study for abduction or for normic (exception-bearing) rules specifically.\n- A controlled test of explicit rival-hypothesis listing for abduction in LLMs.\n- A newer measured result than the corpus already holds on defeasible reasoning in LLMs beyond Liu et al. 2025.\n- Replications of Lewis & Mitchell versus Webb et al. beyond the two papers above.\n- Verification of SymbCoT's publication venue and of the 2026 Nature Communications figure values.\n"}