{"path":"research/grounded-critics-and-quarantined-imagination.md","content":"# Grounded Critics and Quarantined Imagination\n\n**Date**: 2026-08-25\n**Type**: Founder-brought paper read and researched, then related to Deliberus at the architecture, threat-model, and open-founder-question layers. Third piece of an accidental trilogy with [fragile-checkers-and-the-verification-bottleneck.md](fragile-checkers-and-the-verification-bottleneck.md).\n**Source**: *Q-Learning With World Models* (QWM), [arXiv:2608.17163](https://arxiv.org/abs/2608.17163) (Aug 2026) — robotic reinforcement learning on the Robomimic and LIBERO manipulation benchmarks.\n\n---\n\n## 0. The plain-language version\n\nA robotics team asked: how do you use a model's *imagination* without letting its errors poison what the system actually learns? Their field had already discovered the failure the hard way. A \"world model\" — an AI that predicts what happens next — can dream up unlimited cheap training experience, and earlier methods trained their robots on that dreamed data. The result is **compounding model bias**: every small error in the dream becomes part of what the robot believes, the robot then acts on the belief, and the errors stack until the learned values are confidently wrong.\n\nQWM's fix is a strict separation of powers. The thing that *learns* — the policy and the value function (the \"critic\") — is trained **exclusively on real transitions**: things that actually happened in the environment. The world model's imagination is used only at **decision time**: before acting, the system imagines a handful of candidate futures, scores them, picks the best action — and then *throws the imagination away*. Dreams may inform the next move; they may never become the record. And there is a subtle second benefit: because imagination picks better actions during data collection too, the *real* experiences the system gathers are higher-value — imagination steers which reality gets sampled, without ever substituting for it. The hybrid beat both pure alternatives on sample efficiency and final performance.\n\nDeliberus is this architecture, built for reasoning instead of robotics — mostly. The graph is the critic; the LLM passes are the world model; propose-then-ratify and the draft tier are the quarantine; and the hinge and worth-asking layers are, quite literally, test-time search over imagined futures. But the mapping also exposes the honest gap: today, with near-zero real human transitions in the graph, our critic is still trained almost entirely on imagined data — and QWM names exactly why that must remain a temporary, labeled condition rather than a settling one.\n\n---\n\n## 1. What the paper establishes\n\nQWM sits on top of standard Q-learning (off-policy RL: learn a value function from a replay buffer of real experience). Contributions relevant here:\n\n1. **The failure class it avoids is named and structural.** \"Prior model-based RL methods often optimize the policy or value function directly on imagined rollouts, which is prone to compounding bias and struggles to scale… a problem that worsens with task horizon and visual complexity.\" Longer chains and richer observation spaces make imagination-trained values *worse*, not better.\n2. **The mechanism**: an action-conditioned world model performs **test-time tree search** — propose candidate actions, imagine short-horizon futures for each, score them with the learned Q-function, execute only the single best action. The policy and critic never see the imagined trajectories as training data.\n3. **Imagination is applied at two stages, and the ablation matters**: search at *evaluation* time improves action choice directly; search at *data-collection* time improves **which real transitions enter the replay buffer** — better questions to reality yield a better-grounded critic. Both together beat either alone.\n4. **Results**: significant gains in sample efficiency and success rate over both model-free and imagination-trained baselines, from low-dimensional state up to pixel observations (a fine-tuned video model as the world model).\n5. **Owned limitations**: test-time search costs real latency per decision, and world models remain expensive and hard to train — grounding prevents their bias from *compounding*, not from *existing*.\n\n---\n\n## 2. The mapping — Deliberus already has this architecture in three places\n\n### 2a. The graph is the critic; the LLM passes are the world model\n\n**Founder correction (same day, and it improves the result): the mapping has two axes, and the first draft stated only one.** The founder's intuitive mapping — *the graph is the world model* — is correct on the outward axis and is the corpus's own prior framing ([world-models-and-the-ontology-revival.md](world-models-and-the-ontology-revival.md): the graph as the coherent street map versus the LLM's incoherent internal one; the hinge computation literally runs simulations *on* the graph). The critic mapping below is the inward axis: not what the artifact *is* to its users, but *what may be trained on what* — in QWM's bias question, the graph sits where the accumulating, must-stay-grounded component sits, and the LLM passes where the fallible generator sits. The reconciliation is the sharpest available formulation of the architecture bet: **the graph is a world model built with the critic's grounding discipline** — an explicit model of the discourse assembled from attributable, challengeable pieces, in contrast to a neural simulator's implicit and error-compounding one. QWM's quarantine rule then says which ingredient may become part of that model ungated: reality's transitions yes, imagination's proposals only through ratification.\n\nThe persistent record — claims, typed edges, propagated strengths — is Deliberus's learned value structure. The LLM passes — extraction, scheme classification, terminus proposals, auto-enrichment, tentative pre-answers — are its world model: fast, cheap, generative, and biased in ways that are invisible from inside. QWM's separation of powers is precisely the shape this project has been converging on from other directions:\n\n- **Propose, never assert** is \"never train the critic on imagined rollouts\" — a machine judgment on contested ground lands as a proposal a human ratifies; wrong proposals are corrected at the boundary instead of compounding through the record.\n- **The draft tier / eager-into-draft** (the kartpaus auto-enrichment design) is the imagination buffer: the machine explores everything — decomposes, pre-answers weighing questions, fetches research — but into a tier that is *displayed as imagination* and surfaced on contest, not written as ground truth.\n- **Capture immediately, publish deliberately** is the same quarantine at the human input boundary.\n\n### 2b. The hinge and worth-asking layers ARE test-time tree search — verified, not analogized\n\nThis is the most satisfying part of the mapping, because it is literal. `hinge.py` computes exactly a counterfactual rollout: clamp a descendant to granted (1.0), re-propagate, clamp to denied (0.0), re-propagate, measure how far the root moves (verified in code this session: `propagate(..., overrides={nid: 1.0})` vs `{nid: 0.0}`). `question_ordering.py` does the same per question — `strength_if_yes` / `strength_if_no` — and `worth_asking` offers the question only if a decisive answer could cross a display band. That is QWM's loop, step for step: *propose actions (candidate questions), imagine future states (clamped re-propagations), aggregate values, pick the single best path* — and then **discard the imagination**. The counterfactual strengths rank the question; they are never written into the graph. The one structural difference is in our favor: QWM's imaginer is a learned, fallible dynamics model, while ours is the deterministic QBAF semantics itself — the fallibility lives one layer down, in the LLM-supplied edge parameters the rollout reads.\n\n### 2c. The data-collection finding is the live-session architecture, pre-validated\n\nQWM's subtlest result — search during *sampling* improves the replay buffer, because imagination chooses better questions to ask reality — is the design of the facilitated live modality and of the kartpaus loop in one sentence. In a live session, every participant answer is a **real environment transition**; the machine's whole role is QWM's role: plan over imagined futures (which gap, if closed, moves the map most?), ask, and let only the human's actual answer enter the record. Hinge-ordered questioning is sampling-time search: it does not make the graph's values better directly — it makes the *incoming real experience* higher-value. The paper's ablation (\"both stages beat either alone\") is the argument for keeping both halves: machine-planned question order *and* machine-assisted reading of the answers, with neither writing values.\n\n---\n\n## 3. The honest inversion: today, our critic is trained on imagined data\n\nHere the mirror stops flattering. QWM's rule is \"policy and value function trained *only* on real transitions\" — and Deliberus's strength layer currently violates it wholesale, by necessity and with labels, but wholesale:\n\n- The badge computes over **machine-extracted** SUPPORTS/ATTACKS edges (verified: `compute_claim_badge` reads incoming typed edges; extraction writes those edges at store time; no ratification gate stands between the pipeline and the strength layer).\n- All 25 sources are machine- or operator-extracted; real human transitions — votes, challenges, CQ answers from people with stakes — are near zero. The corpus already knows this in the adversary-class register (\"no instrument has met an interested party\"); the QWM lens adds the *learning-theoretic* name for the risk: **compounding model bias**. Extraction errors become embeddings, embeddings become auto-connect candidates, candidates become edges, edges become strengths, strengths rank feeds and feed synthesis and will be read by agents — a multi-hop chain where each pass consumes the prior pass's output as if it were data. The disagreement-preservation instrument measures *one* hop (extraction). Nobody measures the drift across the chain.\n- And the paper's scaling warning transfers uncomfortably well: compounding \"worsens with task horizon and visual complexity\" — for us, with decomposition depth and register diversity. The deeper the descent and the wilder the register, the more hops machine output feeds machine input.\n\nThe right conclusion is not that the machine-filled graph is illegitimate — the cold-start has no alternative, and QWM itself *pretrains its world model offline* before reality arrives. The conclusion is that the pre-launch job is **keeping the two data classes distinguishable so that reality can take over the moment it arrives**: provenance labels (asserted / minted / reported), `claim_kind`, the gray band, the propose tier. Those are not display niceties in this frame; they are the separation of training data from imagination, maintained in advance of there being any training data.\n\n## 4. What this contributes to an open founder question\n\nConviction-and-critique's first open founder question — **the standing of untouched machine structure: certification vs earning** — gets an outside answer with a track record. QWM's field tried \"machine structure is authoritative until corrected\" (train on rollouts, patch the errors later) and abandoned it, because errors compound faster than correction arrives. The architecture that won treats machine output as **planning-tier by default**: legitimate, valuable, load-bearing for *deciding what to do next* — which claim to surface, which question to ask, which gap to flag — and never authoritative *value* until grounded by a real transition. Translated: untouched machine structure would *earn* standing through human engagement rather than hold it by certification, and the display layer would say which kind of standing a strength has. A concrete buildable follows (proposal only, founder-paced): a **provenance-split strength readout** — the badge's inputs partitioned into machine-derived and human-engaged contributions, reported separately the way the stance instrument already reports its two signals. The corpus's own prediction of what it would show today: confident-looking structure that is 100% world-model rollout, which is exactly what the gray-band family exists to keep honest.\n\nThis is input to the founder's question, not its resolution — the question also carries dimensions (what certification *means* for a commons; whether earning gates discourage the contribution funnel) that no robotics result touches.\n\n## 5. The trilogy: authority routing\n\nThree arrivals in two weeks, each validating a different edge of the same architecture:\n\n| Arrival | What it measured | The edge it validates |\n|---|---|---|\n| **Accelerated Understanding** | Verification is the bottleneck; a world model with an oracle can attack it | Build the model — where physical law exists to ground it |\n| **Jagged Judges** | Conversational checkers fold under pressure, net-corruptingly | Never let pressure reach a verdict through a channel; route challenges through the record |\n| **QWM** | Imagination-trained values compound bias; grounded values + imagined planning wins | Never let model output write to the learned record; use it to plan, and to choose what to ask reality |\n\nThe common principle deserves its name: **authority routing** — the explicit assignment of which component may write to which layer. Deliberus's tier vocabulary (committable / propose-only / never; draft / candidate / canonical; asserted / minted / reported; record vs channel) is an authority-routing architecture, arrived at from confession-channel and ratification reasoning before any of these three results existed. The world-models doc's stable form — *coherence must be imposed, and the question is where* — gains its companion: **authority must be routed, and the question is who may write where.** That the same routing keeps being rediscovered under different failure pressures (persuasion, bias compounding, steering) is the kind of convergent external warrant the convictions-across-scales method looks for — with the standing caveat that three sympathetic arrivals do not test the design; only interested parties will.\n\n## 6. Limits of the analogy — owned, so the mapping cannot be quietly overclaimed\n\n1. **QWM has real rewards; we have human judgment.** A robot's terminal reward is ground truth. Our \"real transitions\" are human assertions and ratifications — which are themselves fallible, conformity-prone (Asch; the wiggle paper's fabricated-consensus result), and sparse. Grounding in reality means something weaker here: grounding in *attributable, challengeable* human moves, not in an oracle. The two-floors result says this is permanent, not a maturity gap.\n2. **No single policy is being optimized.** RL optimizes one agent's behavior; Deliberus hosts many agents with conflicting objectives. \"Compounding bias\" therefore has a second, adversarial mode QWM never faces: bias *injected* by strategy-class contributors, not merely accumulated by an innocent model.\n3. **Search overhead is real for us too.** QWM's owned latency cost maps to hinge computation, worth-asking cadence, and — for LLM-assisted planning — quota. The display-band pricing (offer a question only when the answer could cross a band) is our version of their cost discipline, and it exists for the same reason.\n4. **Their world model is pretrained on demonstrations; so is ours.** The extraction pipeline learned its habits from human text wholesale, with the biases the threat model already tracks (flattening, register ceilings). Grounding prevents compounding; it does not make the imaginer good. Instrument quality remains its own axis.\n\n## 7. Buildable consequences (proposals — nothing assumed)\n\n1. **Provenance-split strength readout** (§ 4): partition badge inputs into machine-derived vs human-engaged, report both. The certification-vs-earning question's measurable half.\n2. **Multi-hop flattening check**: the disagreement-preservation instrument measures extraction only; a round-trip probe across the chain (extraction → decontextualization → auto-connect → synthesis) would measure whether drift compounds hop-to-hop, which is the compounding-bias signature in our domain. Cheap to prototype on the existing adversarial pairs.\n3. **Name the quarantine in the ontology constraint layer**: one sentence alongside the judge-isolation rule — *machine-generated structure informs planning surfaces by default and moves authoritative strength only through ratified transitions* — so the rule the architecture already mostly follows is stated where future passes will be gated against it. (The extraction-path exception — pipeline edges feeding badges pre-ratification — would then be a *documented, labeled* cold-start exception rather than an unstated one.)\n\n## 8. Sources\n\n- [arXiv:2608.17163](https://arxiv.org/abs/2608.17163) — *Q-Learning With World Models*, read via HTML full text: mechanism, the sampling-vs-evaluation ablation, world-model instantiations (residual MLP dynamics for state; an action-conditioned fine-tune of a video diffusion model for pixels), stated limitations.\n- Founder-shared podcast summary screenshot (Best AI Papers Explained) that surfaced the paper.\n\n**Cross-references**: [fragile-checkers-and-the-verification-bottleneck.md](fragile-checkers-and-the-verification-bottleneck.md) (trilogy; the judge edge) · [world-models-and-the-ontology-revival.md](world-models-and-the-ontology-revival.md) (coherence-imposed-where gains authority-routed-who) · [conviction-and-critique.md](conviction-and-critique.md) (the open founder question § 4 feeds) · [graph-daemons-design-space.md](graph-daemons-design-space.md) (enzyme rule = the planning/equilibrium quarantine, same shape) · [active-inference-context-acquisition-and-deliberus.md](active-inference-context-acquisition-and-deliberus.md) (worth-asking as priced question policy — QWM's sampling-time search is its RL twin) · [islands-of-coherence.md](islands-of-coherence.md) + [live-election-test-design.md](live-election-test-design.md) (live modality: humans as the environment, machine as planner) · [the-scrutiny-gap.md](the-scrutiny-gap.md) (why our \"real transitions\" are softer than robotics reality) · [engagement-gradient-priors.md](engagement-gradient-priors.md) (how sparse the real transitions will be).\n"}