{"path":"research/catching-a-proposal-made-unnecessary/report-llm-methods.md","content":"# Report: LLM methods for catching \"made unnecessary by\" findings (verbatim, 2026-10-26)\n\n*A research helper's final report (Explore, Claude Sonnet, web search; it wrote no files), saved unchanged\nbefore it was read further. Every number carries its source and the helper's own read-status tag; none is\nrecomputed here. Brief: methods that make an LLM reader catch a proposal that settled items made\nunnecessary, skipping what `corpus-graph/reading-for-insight/finding-inconsistencies-as-a-helper-read-it.md`\nalready holds.*\n\n---\n\nRESEARCH REPORT: LLM methods for catching \"made unnecessary by\" (obsolete-rationale) findings\n\nScope and honesty note. I found no paper that measures the exact task (does a new ruling set remove the reason for a proposed rule?). Closest analogues are premise/assumption extraction, requirement-change impact analysis, version-resolution benchmarks, and LLM-judge variance work. Where I infer, I say \"inference\". Tags: [FULL TEXT] means I read the paper's body, [PART] means I read parts of the body (tables or sections), [ABSTRACT] means abstract page only, [SECONDARY] means a search summary or third-party page only. I could not decode several PDFs (the fetch tool returned binary), so I used arXiv abs/html pages. None of the sources below reached [FULL TEXT]; the best are [PART]. Some arXiv IDs are 2026 (today is 2026-10-26).\n\n======================================================\n1. PREMISE / ASSUMPTION ELICITATION BEFORE JUDGEMENT\n======================================================\nDirect evidence that \"state the premises, then check each separately\" helps verification:\n\n- PARC (Premise-Augmented Reasoning Chains), ICML 2025. Each step of a reasoning chain is linked to the earlier steps it depends on. Models identify premises well (open-source models about 90% recall), and verifying each step with its premises shown raises error-identification accuracy by 6 to 16 points absolute. Also adds an \"accumulation error\" class: a step that is fine alone but rests on a flawed premise. https://arxiv.org/abs/2502.02362 [ABSTRACT]. Domain is math, so transfer to rulings is inference.\n- Ling et al., NeurIPS 2023, \"Deductive Verification of Chain-of-Thought\". Verifying a whole chain at once is unreliable; splitting into steps that each see only their premises makes verification reliable. The paper gives no number I could read. https://proceedings.neurips.cc/paper_files/paper/2023/hash/72393bd47a35f5b3bee4c609e7bba733-Abstract.html [SECONDARY, abstract text via search].\n- Chain-of-Verification (Dhuliawala 2023, Llama 65B). Plan checking questions, then answer them independently of the draft. List-question precision on Wikidata: baseline 0.17, joint 0.29, 2-step 0.36, factored 0.32. Wiki-Category: 0.12, 0.15, 0.21, 0.22. Closed-book MultiSpanQA F1: 0.39, joint 0.46, factored 0.48. Biographies FActScore: 55.9, joint 60.8, factored 63.7, factor+revise 71.4. So answering each check separately beat answering them together in the same context on 3 of 4 settings (not Wikidata precision, where 2-step was best). https://arxiv.org/html/2309.11495v2 [PART].\n- AssumptionMiner (July 2026). Extracts implicit assumptions in LLM code generation. 180 ambiguous tasks, 676 annotated assumptions; a confidence-weighted ensemble reaches F1 0.816, reported as 3.6x the strongest offline baseline. Its definition fits our case: an assumption is absent from the prompt, instantiated in the output, and has a plausible alternative that would change behaviour. The abstract shows no improvement in downstream correctness, only extraction quality and controllability. https://arxiv.org/abs/2607.22898 [ABSTRACT].\n- Feng and Hunter (COMMA 2026). An LLM generates intermediate implicit premises; they are translated to logic and used to decide entailment, contradiction or neutrality between two statements. Evaluated on the Microtext corpus; abstract gives no numbers. https://arxiv.org/abs/2608.18821 [ABSTRACT].\n- Older warrant work (Habernal 2018 ARCT) found models of that era inadequate at reconstructing warrants. A search summary says a GPT-4 warrant study had experts accept 61.7% of GPT-4 warrants versus 45.5% of human ones; I could not confirm this on the paper I fetched (WarrantScore, https://arxiv.org/abs/2601.17377, whose abstract has no such number). Treat that figure as unverified [SECONDARY].\n- Gap: I found no paper that tests premise-first prompting against a plain \"judge directly\" baseline on a redundancy or obsolescence task.\n\nRelated caution: Tam et al. (EMNLP 2024 Industry) report that strict structured-output formats reduce reasoning accuracy; loosening the schema improved scores and lowered standard deviation across prompt perturbations on GSM8K for three models. They also note strict formats can help classification. https://arxiv.org/abs/2408.02442 [SECONDARY]. Your required \"view\" and \"does today's graph hold it\" fields are forced-choice schema fields; inference: the model may fill them as a classification step without having reasoned first.\n\nWhat this suggests for the reader: have it write each proposal restriction as \"rule R holds because condition C\" in free text first, then check each C alone against a small set of rulings, and only then fill the structured fields.\n\n======================================================\n2. SUBSUMPTION / REDUNDANCY / ENTAILMENT WITH LLMs\n======================================================\nThin and mostly indirect.\n\n- LEREDD (2026). LLM plus retrieval plus in-context examples for requirement-pair dependencies (GPT-4.1, temperature 0). Dependency types include \"Is similar\", which covers redundancy. Overall accuracy 0.93, F1 0.84; zero-shot GPT-4.1 macro F1 about 0.39 averaged over three systems. Few-shot lifted \"Requires\" F1 from 0.52 to 0.63, and adding retrieval brought it to 0.70. Redundancy cannot be judged from it: only 7 annotated \"Is similar\" instances, zero-shot F1 0.00 on two datasets and 0.40 on one, and no per-type result for the full method. The authors conclude retrieval precision matters more than volume. https://arxiv.org/html/2602.22456v1 [PART]. Note that pairwise judgements are the unit here, not a set entailing a statement.\n- Hofmann et al., REFSQ 2025, redundancy between user stories, graph vs LLM. Claim: the graph method catches strict redundancy, the LLM catches most of those plus many more semantic redundancies, with less determinism. I could not read the numbers. https://www.uni-marburg.de/de/fb12/arbeitsgruppen/swt/publikationen/dateien/2025/hofmannetal.pdf [SECONDARY].\n- Ontology work: DL-Lite study (https://arxiv.org/pdf/2406.17532) finds LLMs reasonable at TBox subsumption, degrading with transitivity or large data [SECONDARY]. OntoLAMA (ACL Findings 2023) finds subsumption inference harder for LMs than ordinary NLI [SECONDARY]. I found no work on LLM detection of redundant axioms, and no number for it.\n- Multi-premise effects: premise order alone cut deductive accuracy by over 30% (https://arxiv.org/abs/2402.08939 [ABSTRACT]). A syllogism study reports accuracy falling at 3 and 4 premises versus 2 (https://arxiv.org/pdf/2406.11341 [SECONDARY]). A default-reasoning study found no consistent length penalty for strong models above 90% (https://arxiv.org/pdf/2508.13718 [SECONDARY]). RuleArena: LLMs \"struggle to identify and apply the appropriate rules\" and are confused by similar-but-distinct regulations; no numbers in the abstract. https://arxiv.org/abs/2412.08972 [ABSTRACT]. The distractor-rule ablation (irrelevant forms hurt more than padding of the same length) is only in a search summary [SECONDARY].\n- Obsolescence-specific, closest analogue: TIDE (2026), 3,050 QA pairs over 644 amended customs instruments, 9 LLMs. Best macro accuracy 68.5%; resolving the governing version from an implicit date 59.7%; detecting that a supplied version does not govern the query only 26.7%. Models also preferred confident memory over the authoritative text. https://arxiv.org/abs/2608.08512 [ABSTRACT]. This is the nearest measured number for \"the model fails to notice that something no longer applies\".\n- HoH (2025): outdated passages lower RAG accuracy and mislead even when current text is present; no numbers in abstract. https://arxiv.org/abs/2503.04800 [ABSTRACT]. VersionRAG reports 58% for plain RAG versus 90% with a version graph on its 100-question set (https://arxiv.org/abs/2510.08109 [SECONDARY]), a retrieval-side fix.\n\nWhat this suggests for the reader: do not ask \"does anything make this redundant\" over everything. Ask a short, concrete yes/no question per premise and per candidate ruling, and expect models to be weakest at saying \"this no longer applies\".\n\n======================================================\n3. PER-PREMISE RETRIEVAL AND LONG-CONTEXT DEGRADATION\n======================================================\nLong-context numbers:\n\n- NoLiMa (ICML 2025): when the needle shares few words with the question, 13 models claiming at least 128K context show strong short-context scores, but at 32K, 11 of them fall below 50% of their short-context baseline; GPT-4o drops from 99.3% to 69.7%. CoT and reasoning models still struggle. https://arxiv.org/abs/2502.05167 [ABSTRACT]. This matches your case, because a proposal's wording and the ruling that undercuts it are unlikely to share vocabulary.\n- Du et al. (Findings EMNLP 2025): even with perfect retrieval, accuracy dropped 13.9% to 85% as input grew, across 5 models and math, QA, and code, and the drop held when distractor tokens were whitespace or masked. Fix: make the model recite the relevant evidence first, which gave up to 4% on RULER for GPT-4o. https://arxiv.org/abs/2510.05381 [ABSTRACT].\n- LongBench v2: 503 questions, 8k to 2M words. Humans under a 15-minute limit 53.7%; best model answering directly 50.1%; o1-preview 57.7%. https://arxiv.org/abs/2412.15204 [ABSTRACT].\n- Lost in the Middle: performance is highest when relevant text is at the start or end and falls when it is in the middle. No numbers in the abstract. https://arxiv.org/abs/2307.03172 [ABSTRACT].\n- Retrieval vs long context (Li et al., EMNLP 2024 Industry): long context beat RAG on average (Gemini-1.5-Pro: 49.70 long context, 37.33 RAG, 46.41 for Self-Route, which used about 38% of the tokens). Older models. https://arxiv.org/abs/2407.16833 [SECONDARY].\n- I did not find clean 2026 frontier-model numbers at 60k words. The aggregator and blog figures I saw were for older models or lacked sources, so I report none.\n\nPer-item retrieval or splitting in change-impact work:\n\n- ProReFiCIA (Etezadi et al., 2025), requirement change impact analysis. The abstract reports recall 85.7% at 3.0% of requirements reviewed (best LLM and prompt, unseen industrial dataset) and 95.8% at 3.4% with retrieval added. The abstract page for the earlier version and the body I read differ: the body I read says about 93.3% recall at 8.5% review on the WASP benchmark (72 requirements, 22 impacted) and about 95.8% at 2.1% on the industrial set (192 requirements, 35 impacted). I could not reconcile the two versions, so cite them separately. Design points that matter to you: the whole requirement list goes to the LLM in one pass; requirements not selected are re-sent as a shorter list with the same prompt and the picks merged (recall refinement by shrinking the context); a fine-tuned NLI model then filters each pick pairwise against the change. Across 64 zero-shot prompt variants, average recall/precision with GPT-4o was 82.7/65.0 and with DeepSeek-R1 90.6/31.4, so prompt wording moves recall far. No repeated-run variance reported. https://arxiv.org/abs/2511.00262 [ABSTRACT] and https://arxiv.org/html/2511.00262v1 [PART].\n- No redundancy- or obsolescence-specific per-claim retrieval study found beyond the version-aware RAG work above.\n\nWhat this suggests for the reader: it should not hold 60k words and a proposal at once. Give it each premise with a handful of retrieved rulings, ask it to recite the relevant words first, then judge; and re-run on whatever the first pass did not select.\n\n======================================================\n4. COUNTERFACTUAL / PERSPECTIVE PROMPTING AND TEMPORAL SCOPE\n======================================================\nEvidence is weak and mostly qualitative.\n\n- Pre-mortem / prospective hindsight: I found no controlled LLM evaluation. The human-study figure of about 30% improvement (Mitchell, Russo and Pennington 1989) is widely repeated but I could not verify it [SECONDARY]. A 2025 \"Could you be wrong\" metacognitive prompt (https://arxiv.org/pdf/2507.10124) only demonstrates that the model surfaces extra errors and alternatives, with no benchmark [SECONDARY].\n- Devil's Advocate: anticipatory reflection (Findings EMNLP 2024) reports a 23.5% WebArena success rate in a zero-shot setting, 3.5 points above prior zero-shot methods, and 45% fewer trials and plan revisions. It is agent planning, not rule review. https://arxiv.org/abs/2405.16334 [SECONDARY].\n- Self-correction caution: Huang et al. find that without external feedback LLMs struggle to self-correct reasoning, and performance sometimes gets worse. https://arxiv.org/abs/2310.01798 [ABSTRACT]. Inference: \"reconsider the opposite\" with no new information may not help; the prompt needs new facts to react to (for example, the text of the rulings that built the missing capability).\n- Persona framing does not buy accuracy: 162 personas, 4 model families, 2,410 questions, no improvement over a no-persona control; picking the best persona per question was no better than random. https://arxiv.org/abs/2311.10054 [ABSTRACT]. So \"imagine you are a skeptical maintainer\" is probably not the lever.\n- Temporal scope: TimeStress, 18 LMs, the best perfectly separates correct from incorrect time contexts for only 11% of facts, and its errors are of a kind humans would not make. https://arxiv.org/abs/2502.01220 [ABSTRACT]. (A search snippet gave 6% for another version; I use the abstract I read.) Temporal Validity Change Prediction (Wenzel and Jatowt, Findings ACL 2024) tests spotting context that shortens or lengthens a statement's validity window; I did not obtain its numbers. https://preview.aclanthology.org/dois-2013-emnlp/2024.findings-acl.84 [SECONDARY]. evolveQA: up to 31% lower performance than static questions across 12 LLMs. https://arxiv.org/pdf/2510.19172v1 [SECONDARY].\n- Practice analogue: architecture decision records use Amends/Supersedes status fields, linked in both directions, and an \"Accepted, with Amended\" state when a premise expired but the reasoning stands. https://docs.typo3.org/p/netresearch/nr-llm/0.34/en-us/Adr/Index.html [SECONDARY, practice not research]. An LLM study of 980 ADRs from 109 repos found accuracy was good for explicit decisions and fell for implicit ones. https://arxiv.org/html/2602.07609v1 [SECONDARY].\n- Truth-maintenance systems (dependency-recording belief revision) are the classical frame for your failure: store the reason a belief holds and retract when it fails. I found no peer-reviewed LLM evaluation of this idea, only practitioner posts.\n\nWhat this suggests for the reader: skip persona and \"be skeptical\" wording; instead give it the concrete present-day facts (\"these three rulings now exist\") and ask whether condition C is still true.\n\n======================================================\n5. RUN-TO-RUN VARIANCE AND HOW TO REDUCE IT\n======================================================\n- Rating Roulette (EMNLP Findings 2025): low intra-rater reliability across 3 runs, Krippendorff's alpha 0.3263 (Llama 3.1), 0.6278 (DeepSeek-R1), 0.7883 (Qwen 3) on SummaC; on MT-Bench 0.265, 0.507, 0.563. Qwen 3 gave identical judgements on all three runs for only 61.3% of MT-Bench cases. Majority of 3 beats single-run mean balanced accuracy: 61.4 vs 59.1, 72.3 vs 69.8, 80.6 vs 79.4. Turning sampling off cost accuracy (58.4, 69.3, 79.2). https://arxiv.org/html/2510.27106 [PART]. Gains are real but small (1 to 2.5 points).\n- Coin Flip Judge (2026): GPT-4o-mini and GPT-4.1-mini, 29 tasks, 50 trials per question. Pairwise preferences flip 13.6% on average; 28% of questions exceed a 20% flip rate. Single-trial consensus fidelity 86.6%; 3 trials about 90%; 11 trials about 95%; hard questions need 15 trials for 90% and even 50 were not enough for 95% on the hardest. Temperature 0 cut flips from 13.3% to 2.8% (GPT-4o-mini) and from 13.9% to 7.9% (GPT-4.1-mini), not to zero. Equivalent prompt templates changed majority outcomes in 25% of cases. Two models from one provider only. https://arxiv.org/abs/2606.13685 [ABSTRACT] and https://arxiv.org/html/2606.13685 [PART]. Caveat: these are verdict flips on a pair choice, not instability in which items are found.\n- Recall instability (closest to your problem): 300 repeated agentic security scans of the same code. 134 of 158 unique findings matched to a reference appeared in all 5 repetitions; of 161 unique unmatched findings, 80 appeared in only one of five runs and only 22 in all five. https://arxiv.org/abs/2606.15762 [ABSTRACT]. Reading: obvious items are stable, marginal items are the unstable ones. Inference: your \"different set of rulings each run\" is the marginal tail, and a single run samples that tail at random.\n- Union vs majority: repeated sampling raises coverage (fraction of problems solved by any sample) log-linearly over four orders of magnitude; on SWE-bench Lite, 15.9% at 1 sample to 56% at 250. Without an automatic verifier, majority voting and reward models plateau beyond several hundred samples. https://arxiv.org/abs/2407.21787 [ABSTRACT]. SciLitBench (systematic review screening, 22 open-weight LLMs): strong pool at 3 of 5 votes F2 0.808, precision 0.551, recall 0.915; an \"either pool\" union rule reaches recall 1.000 at precision 0.275 (F2 0.655); ensembles did not beat the best single model on full-text screening in that corpus. https://arxiv.org/html/2609.05505 [PART]. So union buys recall and costs precision; you need a second stage to prune.\n- Order shuffling: permutation self-consistency (shuffle list, aggregate) improves on single-order inference by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA-2 70B across five ranking datasets. https://arxiv.org/abs/2310.07712 [ABSTRACT]. Ranking tasks, not finding tasks. Order sensitivity persists across tasks (https://arxiv.org/abs/2502.04134 [ABSTRACT], no numbers), and premise order alone costs over 30% in deduction.\n- A blog-level report, not a study, argues repeated local-model review means \"majority voting picks persistence, not correctness\" [SECONDARY]; consistent with a shared bias such as always saying \"as designed\".\n- Number of samples for a stable set of items: I found no figure. The only anchors are 3 runs (Rating Roulette), 11 trials (95% fidelity, verdict), 5 repetitions (security scan).\n\nWhat this suggests for the reader: use many cheap runs with shuffled order, keep the union of what any run finds, and put a separate verifier after it; use majority only on the final yes/no for each candidate.\n\n======================================================\n6. THE OTHER DIRECTION: WHAT DOES THIS NEW RULING MAKE UNNECESSARY?\n======================================================\n- ProReFiCIA (above) is exactly this shape: given a change, find which existing items it affects. Recall 85.7% to 95.8% with 2.1% to 8.5% of items to review, depending on version and dataset. It reads all items per change, with the change as the short input. [ABSTRACT]/[PART].\n- Related change-impact papers (Nejati et al., Arora et al., Rusdianto et al. 2024 for GPT-4 impact prediction) were named in search results; I did not read them [SECONDARY]. https://arxiv.org/pdf/2405.01695.\n- Code-comment inconsistency after a code edit is a trigger-time check in practice: 2512.19883 reports up to 13.54% F1 over recent state of the art and 4.18% to 10.94% over fine-tuned LLMs, with a small CodeT5+ model on structured edit actions. https://arxiv.org/abs/2512.19883v2 [SECONDARY].\n- SATD (TODO) repayment: LLMs fixed only about 2% to 8% of instances in the best 2023 setting (https://arxiv.org/pdf/2308.08943 [SECONDARY]); I found no study of detecting a TODO that became unnecessary without being edited.\n- Duplicate PR/issue detection: DupLLM P@1 0.929 (APSEC 2024) and 93.9% accuracy / 90.5% recall in another paper [SECONDARY]; no work on \"does this merged change resolve an open issue\".\n- Regression of decisions: nothing found.\n\nWhat this suggests for the reader: also run the check the other way at ruling time, with the new ruling as the short input and the open proposals (or their extracted premises) as the list.\n\n======================================================\n7. CHECKLISTS / NAMING THE FAILURE CLASS\n======================================================\n- JuICE (2026), one model (gemma-4-31B-it), cultural-error span detection: baseline P 0.6133, R 0.3532, F1 0.4457; plus taxonomy P 0.5662, R 0.3829, F1 0.4546; plus few-shot P 0.5392, R 0.3963; both P 0.5393, R 0.4031, F1 0.4591. So naming categories raised recall by about 3 points and lost about 5 points of precision; examples helped a bit more. Single-run table. https://arxiv.org/html/2605.26955 [PART].\n- ETGPO (2026): building guidance from a taxonomy of the model's own most frequent errors beat CoT on average (GPT-4.1-mini: 69.08 vs 66.12 across six benchmarks, FOLIO 81.34 vs 75.09). https://arxiv.org/html/2602.00997v1 [PART]. This is accuracy on tasks, not recall of flaws, but the class list was drawn from real failures, which you can do from your own missed case.\n- CWE-specific prompting for vulnerabilities: authors report improved performance and F1 gains up to 0.25 for dataflow prompts on larger models; recall not reported. https://arxiv.org/html/2311.16169v2 [SECONDARY].\n- LEREDD (above): few-shot examples per type gave the biggest early gains, peaking around 3 to 5 examples per type. [PART].\n- Gap: no study isolates \"names the failure class\" versus \"does not\" for recall of redundancy-type defects.\n\nWhat this suggests for the reader: define \"made unnecessary by\" with the stated class plus 3 to 5 worked examples (including the \"starts no wider than the links\" case), and expect more flags with more false ones.\n\n======================================================\nTOP 5 CHANGES, RANKED BY EXPECTED EFFECT ON THE FAILURE\n======================================================\n1. Split the reading: extract the proposal's premises (\"restriction R because C\"), then check each C alone against a few retrieved rulings, with the model reciting the ruling words before judging.\n Rests on: NoLiMa (11 of 13 models under 50% of baseline at 32K, low word overlap), Du et al. (13.9% to 85% loss even with perfect retrieval; recitation helps up to 4%), PARC (6 to 16 points from verifying with premises shown, premise recall about 90%), CoVe factored beats joint on 3 of 4 settings, Ling 2023. Weakness: none measured on obsolescence; retrieval must handle low word overlap (the premise should be matched by meaning).\n2. Add the forward, triggered check: when a ruling is made, give the model that ruling plus the premises of each open proposal and ask which stated reasons it removes.\n Rests on: ProReFiCIA (85.7% to 95.8% recall at 2.1% to 8.5% review), TIDE (only 26.7% at noticing something does not govern, so the backward read is the weak direction), and the short-input shape from change 1. Weakness: depends on proposals' premises being stored.\n3. Run the finder stage many times with shuffled order and keep the union, then verify each candidate once more with majority.\n Rests on: security scan (80 of 161 unmatched findings seen in one run only; 134 of 158 matched ones in all five), Large Language Monkeys (coverage rises log-linearly with samples), SciLitBench (union recall 1.000 at precision 0.275), Coin Flip (about 11 trials for 95% verdict fidelity), permutation self-consistency (7 to 18%). Caveat: no paper gives the number of runs for a stable item set; measure it on your three existing runs. Union will not help if the model never says \"made unnecessary by\" in any run, which is why this ranks below 1 and 2.\n4. Record the reason at ruling time: every ruling and rule gets a written \"holds because\" and \"stops being needed when\" line, kept in the ledger.\n Rests on: AssumptionMiner (extraction F1 0.816, but can be wrong without a human), ADR practice (Amends/Supersedes links, \"premise expired\" status), TMS idea. This is an inference, not a measured result, but it turns changes 1 and 2 from guessing premises into lookups.\n5. Name the failure class and unbundle the output: give a definition and 3 to 5 worked examples of \"made unnecessary by\", and have the model write free-text reasoning before the \"view\" and \"does today's graph hold it\" fields, with today's graph facts supplied as text.\n Rests on: JuICE (+3 points recall, about -5 points precision), LEREDD (3 to 5 examples per type), Tam et al. (strict formats hurt reasoning), Huang et al. (self-critique without new information does not help). The \"today's graph\" field can only be answered if the graph state is in the prompt (inference).\n\nNot worth trying first: persona or \"skeptical reviewer\" wording (https://arxiv.org/abs/2311.10054), temperature 0 alone (flips drop to 2.8% and 7.9% but sampling off lowered accuracy in Rating Roulette), and plain majority voting over whole-document runs (it keeps the shared \"as designed\" bias).\n"}