{"path":"research/bengio-safety-from-honesty-and-deliberus.md","content":"# Bengio's \"Safety from Honesty\" and Deliberus: Convergent Evolution at the Statement Layer\n\n*Research note — July 7, 2026*\n*Paper: Bengio, Richardson, Gavenčiak, Cohen, et al., \"Safety from Honesty in a Disinterested AI Predictor,\" LawZero, June 30 2026, [arXiv:2606.29657](https://arxiv.org/abs/2606.29657). Read in full; page references below.*\n\n---\n\n## What the paper is\n\nLawZero's formal safety case for the Scientist AI (SAI) Predictor: a deliberately **non-agentic** model trained to approximate the Bayesian posterior over Boolean **statement-variables** — \"each named by a valid statement in natural language (including mathematics and code)\" — conditioned on a dataset of **epistemically contextualized** observations. The failure mode it targets is **implicit agency**: goal-directed behavior nobody specified, arising from imitating human drives (pretraining) and optimizing for approval (RLHF). The headline result (Thm 5.24) bounds the probability that consequence-invariant training produces a \"dangerous Predictor\": dangerous predictors require *coordinated* deviations across many queries, such patterns are argued to be combinatorially sparse within any loss band, and training that never rewards downstream consequences has no signal pointing at them. Deployment goes through a **Guardrail** (the predictor estimating harm probabilities for candidate outputs, with abstention — `noprediction` — when estimated risk exceeds a threshold).\n\n## The central resonance: epistemic contextualization IS the Deliberus ontology\n\nDefinition 3.22 splits every training record into exactly two kinds:\n\n> \"A factual observation records a directly measured value, the output of an executed computation, or a proven statement. In contrast, a **communication act** records that a named agent uttered something, possibly identified as a stated claim about a property of the world, and it carries metadata identifying the source, venue, and timestamp.\"\n\nThis is the same move Deliberus made, independently, for a different reason. Deliberus's graph holds claims with provenance (`claim_kind`, attribution, source metadata, epistemic status); contested assertions live as *attributed positions*, never as ground truth; the reader preamble on every agent-readable page states that the graph \"maps human reasoning — including claims that are contested, mistaken, or extreme — without endorsing it.\" Bengio's team needed contextualization to make the *learning problem well-posed* (Component A1: \"prevents the Predictor from treating unverified or socially repeated claims as ground-truth facts\"). Deliberus needed it to make *deliberation* well-posed. Same root problem — how to represent contested human assertions without either endorsing or discarding them — same solution shape: \"source S asserted X\" as the first-class object, with the truth of X held as a separate, explicitly-uncertain variable.\n\nThe convergences continue at every layer:\n\n| SAI (paper) | Deliberus (live) |\n|---|---|\n| Statement-variables: Boolean variables named by natural-language statements (§1.1, Def 3.2) | Atomic claims: one node type, natural-language statements as the graph's atoms |\n| Invalid statements repaired by adding context (\"Red is the best color\" → \"Yoshua prefers red\", Ex. 3.1) | The stranger test + decontextualization pass in extraction |\n| Communication acts with author/venue/timestamp; \"enabling the SAI to weight sources by their reliability\" (Ex. 3.24) | Source attribution on every extraction; provenance-scoped operations |\n| Latent factual variable \"X is true\" forced for every communication act asserting X (Component A2) | Every extracted claim is both an attributed utterance and a challengeable assertion with its own strength |\n| The **Explainer**: System-2 reasoning as chains of structured natural-language statements, each independently evaluated by Q, coherence-constrained (§1.1, §2.4/ELK) | The argument graph: decomposition into independently challengeable claims, QBAF strength propagation |\n| \"A merely plausible-looking rationale that does not actually support its conclusion must violate the probabilistic consistency constraints... it cannot be plausible at each step and yet harmful in its conclusion for free\" (p.11) | No-copout-axioms + the completeness oracle: exposure of support is measured; bundled reasoning is flagged for decomposition |\n| Calibrated posterior probability as the only \"score\"; honesty = \"calibrated, fair, non-strategic\" (p.4) | QBAF badge = exactly the state of the arguments around a claim, recomputable, never an oracle verdict |\n| `noprediction` abstention as first-class; \"epistemic caution\" | First-class `undecided` in the terminus classifier (\"never penalized\"); fail-soft scrutiny; guardrail thresholds |\n| Prompt injection resisted by contextualization: \"untrusted text is represented as an attributed communication act, such that injections like 'ignore previous instructions' are not treated as instructions to be obeyed\" (§6) | All source input is claims-to-map, never instructions; the same architecture is Deliberus's data-poisoning posture (a poisoned source is an attributed utterance competing with other evidence) |\n\nTwo systems, built for different purposes on different sides of the human/machine line, converged on: atomize to statements, contextualize by provenance, expose reasoning as structured chains of independently-evaluable claims, compute confidence from structure, abstain honestly, and treat input as evidence rather than instruction.\n\n## The complementarity is now sharper than the May analysis\n\nOur May 2026 landscape note (`.private/bengio_lawzero_landscape_2026_05_16.md`) framed Deliberus as \"the human-side complement to Scientist AI's machine-side reliability\" and warned that the framing only survives with a concrete theory of the human-oversight gap. This paper *supplies that gap in its own text*, twice:\n\n1. **The normative punt, explicit.** §2.2: for contested claims, SAI \"has no guarantee of convergence to the moral truth. Our SAI accuracy argument should thus be viewed as providing guarantees for such determinate claims only.\" Remark 5.28: the safety threshold \"A_safe is a **normative certification threshold fixed by the designer**... the present framework can be seen as **describing (rather than attempting to resolve) this tradeoff**.\" The entire normative layer — what counts as harm H, where thresholds sit, whose values govern the Guardrail — is explicitly outside the formal machinery, delegated to designers and auditors. Deciding, auditing, and contesting those choices *is a deliberation problem*, and it is exactly the layer Deliberus structures: value weighings opened into checkable components, typed residues where decomposition bottoms out, disagreement preserved rather than flattened. SAI computes the descriptive posterior over determinate claims; Deliberus structures collective reasoning over the contested remainder. Together they tile the epistemic field; neither claims the other's territory.\n\n2. **Implicit agency names the mechanism behind Deliberus's own standing threat.** The paper's diagnosis — approval-optimized training \"explicitly reward[s] outputs for their downstream effects on evaluator preferences,\" producing sycophancy that scales to deception — is precisely the mechanism behind the **convergence-illusion hole** in Deliberus's red team (`convergence-wager-red-team.md`): RLHF'd extraction models nudging genuinely disagreeing humans toward the semantic center *because pleasing is what they were trained to do*. Deliberus currently mitigates at the **instrument layer** (the disagreement-preservation check, propose-only classifiers, first-class `undecided`) while running on ordinary RLHF models; Bengio proposes fixing it at the **training layer**. These are the same threat model at different depths — and an SAI-style disinterested predictor would be the ideal engine for exactly the extraction passes where Deliberus most fears flattening. (Per the arsenal-and-harness architecture, the harness is deliberately model-swappable; \"swap the flattening-sensitive passes to a disinterested predictor when one exists\" is now a nameable roadmap item.)\n\nThere is also a pleasing inversion in the paper's future-work list: it calls for \"empirical validation of the metrics associated with contextualization and epistemic-caution.\" Deliberus is a production system doing epistemic contextualization on live human discourse, with shipped metrics (disagreement-preservation scores, completeness exposure, residue typing) — the human-corpus cousin of the validation program LawZero says it needs.\n\n## Honest boundaries (anti-coattail discipline, per the May note)\n\n- SAI is a *training-time architecture* with formal machinery (consequence-invariance, loss-band sparsity, R_shell) and no product; Deliberus is a *runtime platform* with a live product and no training-time guarantees. The kinship is architectural convergence at the statement/contextualization layer plus complementary scope — never \"Deliberus implements Scientist AI.\"\n- The paper's formal safety bound has no Deliberus analogue and community critique of its load-bearing assumption (loss-band sparsity; see the LessWrong probe of Thm 5.24) is live. Deliberus's instruments are empirical checks, not theorems.\n- LawZero explicitly does not build multi-human collective-reasoning tools (unchanged from May); the space remains unoccupied by them.\n\n## Implications worth acting on\n\n1. **Funding language**: the paper's acknowledgments (FLI, Schmidt Futures/Sciences, Coefficient Giving, Gates, ARIA, CIFAR) map onto the same funder cluster behind CAIF's new multi-agent call. The \"human-side complement, now with the punt documented in Bengio's own safety case\" framing strengthens the CAIF pre-proposal (Aug 8) and the TsviBT conversation — cite §2.2 + Remark 5.28 rather than asserting the gap.\n2. **Vocabulary adoption**: \"epistemically contextualized,\" \"communication act vs factual syntax,\" and \"disinterested\" are now citable technical terms for design choices Deliberus already made. Use them with citation; they make the platform's provenance layer legible to the safety community in its own language.\n3. **Roadmap item (small, real)**: Deliberus's extraction already distinguishes attributed source claims from assertions, but the paper's *latent-variable* discipline (every communication act forces the latent \"X is true\" as a separate variable) suggests a cleaner internal split between \"who asserted X\" edges and \"is X\" strength — worth a look when the one-graph staged-maturity work happens.\n\n*Companions: `deliberus-as-alignment-infrastructure.md` (the oversight-reliability theory of change this paper now documents from the machine side), `convergence-wager-red-team.md` (convergence-illusion = implicit agency at the application layer), `.private/bengio_lawzero_landscape_2026_05_16.md` (funding positioning + backfire discipline).*\n\n---\n\n## Assumption ranking + the necessity question (July 7, 2026 — founder-prompted hole-poking)\n\nThe celebratory read above rests on assumptions of very different strength. Ranked weakest → strongest against evidence actually in hand (continuing the project's assumption-ranking tradition, cf. `assumption-ranking.md`):\n\n1. **The mechanism equation (weakest)** — \"implicit agency/sycophancy is THE mechanism behind extraction flattening.\" Asserted identity, zero direct evidence in our system; observed flattening risk could be paraphrase compression or embedding averaging. *Cheap test: extract the same source with an RLHF-heavy vs minimally-tuned model; compare disagreement-preservation.*\n2. **The demand inference** — \"the normative punt implies demand for deliberation infrastructure.\" Our inference, not the paper's statement. Empirically, normative thresholds get set by panels, regulators, and meetings; no evidence anyone routes `A_safe`-class decisions through structured public deliberation. The legitimacy argument is philosophy, not demand.\n3. **Funder relevance** — architectural kinship with a famous paper is not capability-uplift evidence, and leaning on it risks the coattail failure the May landscape note warned about. Cite the punt; never lean.\n4. **Instrument sufficiency** — \"application-layer counter-instruments suffice.\" The circularity: our honesty instruments run on the same model class they audit (LLM judgments + embeddings detecting LLM flattening). Bengio's own point is that implicit agency \"may not even be detectable through the system's stated outputs.\" Held up today by two non-LLM confession channels (verbatim source retained beside every extraction; human correction) and by dogfood-scale stakes — but this assumption decays as stakes rise.\n5. **Depth of the isomorphism** — the contextualization layer is genuinely shared *function*; but one real disanalogy was underplayed: **QBAF strength is participation-dependent, a Bayesian posterior is evidence-dependent.** A claim can be strong because nobody attacked it yet; P_n cannot. The badge is a map of a debate, not a probability model of the world.\n6. **Kinship survives their contested theorem** — the community attack (loss-band sparsity behind Thm 5.24) does not touch Def 3.22 / the Explainer / consequence-invariance, which is where the Deliberus resonance lives.\n7. **The statement/provenance substrate is right (strongest)** — four independent derivations from different objectives (Deliberus 2013+2026; Claimify; DnDScore; LawZero's formal requirement), with prompt-injection resistance falling out unplanned in two of them. Independent convergence under different optimization pressures is the strongest evidence a design layer gets.\n\n### Could Bengio-level rigor be NECESSARY?\n\nThree-level answer:\n\n- **Deliberus-as-deliberation-aid: no.** Humans in the loop, verbatim sources retained, corrections cheap. Demanding training-level guarantees here is the perfectionism that never ships.\n- **Deliberus-as-measurement-instrument (the residue map as falsification metric): partially yes.** A measurement instrument must be more trustworthy than what it measures; today the instruments are the same model class as the pipeline they audit. Proportionate escalation path short of training-from-scratch: (a) run honesty checkers on a **different model family** than the extractor (doable now); (b) **audit our extended QEM against the published rationality postulates for gradual semantics** (never done — a real formal-care gap at the aggregation layer); (c) keep flattening-sensitive passes swappable to a disinterested predictor if one ships. **Update 2026-08-18 — the gap now has a measured instance and, more usefully, a runnable test.** Heilig's GPT-5 storytelling study found a model rating pure nonsense at 8/10 and defending it past the point most readers could scrutinise, with a stated cause (*\"if you primarily use AI to evaluate generative AI during training, you'll get something that AI will like\"*) and a **signature**: a judge that learned the concept has an optimum, while a judge that learned the marker is monotonic in marker density. That is measurable here in an afternoon against two live targets — the terminus classifier's `confidence` and `self_eval`'s `overall_quality_score`. **One of those two targets no longer exists (2026-08-18):** reading the prompt showed it asked for a score with no rubric, so the quantity was undefined and the calibration ill-posed rather than merely unrun, and the founder removed it instead of calibrating it. The terminus classifier's confidence remains, propose-only and uncalibrated. See [the-scrutiny-gap.md](the-scrutiny-gap.md) § 3, which promotes escalation step (a) from *doable now* to *owed*.\n- **Deliberus-as-alignment-infrastructure (graph consumed as a values-target): conditionally yes.** Graph corruption then graduates into a safety problem and ingestion-layer rigor of roughly SAI's kind becomes necessary. Conditional on a future that may not arrive; swap-ability is the right hedge today, guarantees are not.\n\n### The two genuinely unclear things\n\n1. **Whether SAI will exist as a usable artifact.** The Predictor is \"not initialized from a pre-trained LLM\" (Rem. 3.27) — training from scratch on contextualized data, against LawZero's ~$30M/18-month runway. If it never ships, \"swap to a disinterested predictor\" stays vapor.\n2. **Their contextualization engine does not exist either** — specified \"solely at the level of requirements (i.e., we do not provide a completed algorithm)\" (§3.2.1/1.1). The component their entire safety case requires and has not built is the component Deliberus runs in production daily, at small scale, on human discourse. Necessity cuts both ways: our wager eventually needs their rigor; their architecture needs someone's empirical demonstration of contextualization now.\n\n*Follow-ups proposed (no decisions pending): the flattening-mechanism ablation; the QEM rationality-postulate audit; cross-model-family checkers for the honesty instruments.*\n"}