{"path":"research/how-independent-reasons-add/polarity-eval/README.md","content":"# Reading two attacks on one claim\n\nNovember 13, 2026. The standing-work query selects both supports and attacks. Its reader receives the same target and two reason texts for either. Our question is whether this changes a judgment a person would rely on.\n\n## Before the first call\n\nTen newly written, fictional cases have expected answer sets and a separate criterion for checking the explanation. Four are for development and six are reserved for the later comparison. They are local judgments by Codex. They are not founder rulings or ground truth about natural arguments. Only the target, reasons and method go to the reader. The claimed answer and grading criterion stay local. The router is fixed to direct Claude Sonnet 5.5 with one recorded attempt per question, short answers, and no fallback. Paid calls evaluate the prompt in isolation. No graph reads or writes are needed.\n\nThe baseline is the November 13 pre-fix prompt with an attack pair. The first candidate tells the same prompt that the reasons bear on the falsity of the quoted target. This gives an explicit rival world while preserving the original words, including an already negative target. A separate held-out test challenges evidence for the conclusion without itself asserting the opposite. An error on such a case has a different remedy from a sign error.\n\nScore the classification and the counterfactual separately. For a joint inference the premise gate suffices. For other cases read whether each proposed rival could produce the observation while the original attacked claim is true. A correct class with an explanation that assumes the attacked claim false cannot justify a score. Record failures of the rubric as well. This small targeted evaluation measures these examples only.\n\n## Iteration 1 · original prompt\n\nOf four development cases the model returned three answers, two with an accepted label. In the cold-box case both rivals assumed the box warmed up. That defeats the very claim that must hold in the rival world, and it made two distinct error sources count as one. The clock answer had an accepted label but attributed both observations to an inaccurate source display. The distinction between timekeeping and display makes its rival ambiguous under this fixture. The repeated sensor example retained its correct label and identified a possible sensor misreading while the key remained in the drawer. The provider declined the voucher example, which counts as unanswered. Decision: try explicit negation of the attacked claim with the same schema, model and four inputs. An abstention is part of coverage. Logs retain it.\n\n## Iteration 2 · explicit direction\n\nFour of four development labels matched. The cold-box answer now distinguishes battery error from tray exchange. The clock still leaves open whether the source display or the mechanism that kept time was wrong. The voucher reading reached the joint-premise gate. The two runs together cost about 2.8 cents at the router's computed rate. Decision: freeze this wording for six fresh cases, compare against the old prompt, and reverse reason order for the corrected wording. The held-out cases include a negative original statement and a challenge to the evidence itself.\n\n## Iteration 3 · direction on reserved cases\n\nOn six reserved cases the baseline matched four classes and refused one. The signed prompt matched four and refused one in ordinary order, then matched five with no refusals in reverse order. The two independent-failure cases already passed on baseline: the model sometimes corrects a mistaken instruction on its own. Signed wording fixed the negative-target case relative to the baseline refusal and identified the shared source on the parcel in reverse order. There is no overall gain in ordinary-order class matches on these six. The troublesome substantive case was the uncalibrated gauge. The same defect undermines both copies of a trace; it gives no direct reason that the valve failed. The signed model called the reasons shared in one order and partly shared in the other. Selecting the more conservative label still treats the wrong inference as a candidate for weighing.\n\nDecision: retain signed inputs for direct attacks and separately test an explicit evidence-scope instruction. The gauge example is now a development example for that revision. Freeze six **new** challenge cases before running either prompt on them. Three challenge evidence alone, two directly bear on the negated target and one directly bears on a target about the evidence itself. The last clarifies that mentioning measurement reliability can be exactly on target. Also test that the answer and the explanation written to storage agree when reason order changes.\n\n## Iteration 4 · adding a sentence about scope\n\nOn the six new cases, signing alone matched two labels in the original order. Appending the evidence-scope instruction matched three in ordinary order and two in reverse order. Neither order correctly abstained on any of the three challenges to evidence alone. A refusal and cannot_tell have different meanings. On the copied photograph in reverse order the model even called the two objections independent, potentially licensing an addition in the wrong place. The appended sentence lost to the longer question and schema that presuppose direct reasons. Decision: try a separate, short task whose sole job is identifying what the two reasons bear on. Predeclare fresh examples before asking it. Treat this as a separate pilot; a successful label is no proof its input should be automatically rerouted or rewritten.\n\n## Iteration 5 · ask what the objection bears on first\n\nWith a short schema for the object of the objection, Claude recovered all three challenges to evidence on the earlier challenge set. It also mistook a fallible observation for an objection to that observation. The barrier alarm signalled a fall and the observer saw an apparently horizontal barrier. The possibility of an equipment error limits their force, but both still give reasons to doubt the barrier stayed upright. On the six new synthetic scope cases, ordinary and reversed reason order both matched all six expected categories. Some explanations remain too strong even when their labels match. Detecting oxygen or nitrogen bears against the chamber having no air. Those gases can also be present in mixtures other than air, so the model’s categorical contradiction overstates the case.\n\nDecision: keep this short scope check experimental. Test whether it follows a change in the *claim being attacked* when the two reasons and source topic stay fixed. Use public, source-based paraphrases from Narayanan and Kapoor on measurement and from RAND on coordination. The quoted test propositions are counterpositions constructed for this experiment. They are not statements attributed to the authors, and the pairs were selected to exercise the distinction. Test both reason orders. The model receives only the case texts and scope instructions. This still leaves ordinary graph extractions for future evaluation.\n\n## Iteration 6 · change the proposition while holding its two reasons fixed\n\nOn six public-source cases the separate scope check matched four in each order. The coding pair worked as intended. The benchmark’s limited coverage weakens the evidence for success on real projects. The same limitation bears directly on the stronger proposition that benchmark success suffices to establish real-project success. The model treated both bar-exam targets as direct. Its own explanation described a weak inference from the exam. The first proposition can be read as about one successful system, or generically as a rule about systems that pass the exam. This is both a reader error under the prespecified interpretation and an ambiguity in the test sentence. Do not silently relabel it. In the race case, the model called verification a direct reason against actual mutual restraint. Verification is a condition on that outcome, but neither supplied statement says it is missing in these governments. The rubric allowed mixed or unclear. A better test would vary the presence of that case-specific premise. The source establishes a conditional and articulates a policy judgment. [Narayanan and Kapoor on benchmarks](https://www.normaltech.ai/p/ai-as-normal-technology) · [Abraham, Kavner and Moon on verification](https://www.rand.org/pubs/research_reports/RRA4245-1.html)\n\nDecision: keep the separate scope check as an evaluation instrument. It has earned a comparison against humans on source-attested arguments. It has not earned unattended authority over argument weights. Swapping order produced identical mistakes on the public cases. An order check tests stability, while changing the proposition tests whether the judgment tracks its meaning.\n\n## Changes in the working implementation\n\nThe standing independence reader now presents the denial of the quoted target for a pair of attacks. The original support prompt is preserved. The ledger tracks coverage against the conclusion actually shown. An earlier unsigned record cannot satisfy a newly directed attack. Graph storage and the calculation keep opposite directions distinct, including when the pair has the same two IDs. The legacy reading format retains its original supporting interpretation. A refused call supplies no successful answer for coverage. When the reversed-order answer wins, the stored explanation and the two rivals now come from that answer, with each rival restored to its original claim. The revised scope classifier is confined to this evaluation. The sign repair does not resolve an objection attached to the wrong target.\n\nAll calls used the project's direct Claude Sonnet 5.5 credit. **74 requests, nine refusals, $0.2065343 in computed charges**, including the refusals. Each recorded request took one attempt. No OpenRouter top-up, local inference, live scoring or full graph sweep was needed. The 22 synthetic cases and six public-source cases include deliberate repeats across prompts and orderings. The original four development cases and subsequent six-case sets are reported separately. Repeated readings from one model are not independent replications. A read-only count found 20 eligible attack pairs across ten targets, and 870 support pairs. A local manual review of those 20 exposed attribution, quantifier, missing-premise and shared-survey cases. That is where I would take the next small iteration.\n\n## Reproduction\n\nFrom the project root, use the repository Python environment with PYTHONPATH=. and run the adjacent run.py with a variant, split, optional --swap and a new --output path. It spends direct Claude credits. The committed JSONL files already contain the exact messages, structured replies and router receipts. The cases file holds the interpretations chosen before the corresponding batch. The valid combinations used here are baseline or signed on development and held-out, signed or scope on challenge, and bearing on challenge, scope-holdout and public-pairs. The sign variant uses the same formulation as the production reader. The note records Codex’s grading and decisions. No scores in these tables estimate general graph-wide accuracy.\n\nVerification: 297 affected tests passed across nine suites using the temporary embedded graph. All 74 saved prompts were reconstructed byte-for-byte from the committed cases and replay code.\n"}