{"path":"research/soft-canonical-clustering-and-reversible-merge-semantics.md","content":"# Soft Canonical Clustering and Reversible Merge Semantics\n\n**Date**: April 3, 2026\n**Type**: Design synthesis / scaling strategy\n**Status**: Research conclusion — not yet implemented\n\n---\n\n## The Prompting Question\n\nAs Deliberus's graph becomes denser, does the original fear of recursive auto-decomposition still hold in the same way?\n\nThe earlier concern was that recursive decomposition would explode:\n\n- one claim becomes several premises\n- each premise becomes several further premises\n- generated premises become less grounded as recursion proceeds\n- the system ends up doing the user's thinking for them\n\nThis is the problem articulated in [scheme-bounded-decomposition-and-evidence-as-subgraph.md](scheme-bounded-decomposition-and-evidence-as-subgraph.md): full recursive auto-generation is vulnerable to combinatorial explosion, hallucinated premise chains, interpassivity, and formal distortion of non-deductive arguments.\n\nThat is why **scheme-bounded decomposition** originally looked like the wise compromise. Argument schemes and their critical questions offer a finite, interpretable, grounded decomposition frontier.\n\nBut the graph is no longer empty.\n\nAs more people process articles, PDFs, meta-analyses, opinions, voice notes, and their own reasoning, Deliberus may accumulate a growing layer of reusable empirical and normative mid/high-level subtrees. If so, the problem changes.\n\n---\n\n## The Core Distinction: Generation vs Retrieval\n\nThe older fear remains correct for **recursive generation**.\n\nIt becomes much less compelling for **recursive retrieval and reuse**.\n\nThese are not the same operation:\n\n- **Recursive generation**: invent new subclaims at each level\n- **Recursive retrieval**: search the existing graph for already-vetted subclaims that can serve as the next decomposition layer\n\nThe graph-aware pruning insight from [scheme-bounded-decomposition-and-evidence-as-subgraph.md](scheme-bounded-decomposition-and-evidence-as-subgraph.md) already pointed toward this:\n\n> do not recursively decompose everything; only keep or create structure where it meaningfully connects to the existing graph\n\nAs the graph saturates, a growing share of new decompositions may terminate not in fresh generation but in **reuse of already-existing subclaims**. In that regime, recursion becomes less computationally and epistemically dangerous, because the later layers are increasingly discovered rather than invented.\n\nThat suggests a refined thesis:\n\n**Recursive auto-decomposition is still dangerous as unconstrained generation, but increasingly plausible as retrieval-first reuse of existing subtrees under strong pruning.**\n\n---\n\n## What Is Already True in the Current System\n\nThe current system does **not** perform true identity merging.\n\nWhat it does today:\n\n- creates `SIMILAR_TO` edges automatically at conservative embedding thresholds\n- performs cross-extraction auto-connect into stronger logical relations when justified\n- keeps claims as separate graph nodes with separate provenance\n\nWhat it does **not** yet do:\n\n- canonicalize near-duplicate claims into one operative identity\n- cluster claims into shared equivalence classes that the product treats as one by default\n- merge concept senses automatically into reversible canonical groups\n\nThis matters because the next scaling move should not be \"hard merge everything that looks similar.\"\n\n---\n\n## The Proposed Direction: Soft Canonical Clustering\n\nThe right next step is not hard merge. It is **soft canonical clustering**.\n\nThe idea:\n\n1. Keep the underlying claims as separate graph nodes.\n2. Preserve full provenance, authorship, source span, timestamps, extraction path, and audit trail.\n3. Introduce a higher layer of **canonical equivalence / cluster identity**.\n4. Let the product treat members of a strong cluster as one thing **by default** for practical purposes.\n5. Make that clustering reversible.\n6. Let humans intervene primarily by **splitting** false equivalences rather than manually merging obvious duplicates one by one.\n\nThis preserves two truths at once:\n\n- pragmatically, users do not want to manually merge near-identical claims forever\n- epistemically, false merges are more dangerous than missed merges\n\nSo the system should be allowed to say:\n\n> these claims are probably instances of the same operative claim; show them together unless someone separates them\n\nwithout saying:\n\n> these claims are literally one node now and the distinction is gone\n\n---\n\n## Why Split-First Human Oversight May Be Wiser Than Merge-First Oversight\n\nThe labor asymmetry likely favors automated clustering with human splitting rather than the inverse.\n\nWhy:\n\n- the graph will accumulate many near-duplicates\n- manual merging scales badly\n- users are more likely to tolerate \"these were grouped, but I can separate them\" than \"these remain cluttered until I manually unify them all\"\n\nBut this is only wise if clustering is:\n\n- high precision\n- reversible\n- provenance-preserving\n- visible as an interpretation layer, not ontological erasure\n\nIn other words:\n\n**automatic clustering is acceptable only if false sameness can be discovered, explained, and undone without loss.**\n\nThis is the same underlying principle already present in Thread 4 of [conceptual-threads.md](../conceptual-threads.md): false merges are worse than missed merges.\n\nThe proposed solution is simply a more pragmatic operationalization of that principle.\n\n---\n\n## Claims: Where Soft Clustering Is More or Less Safe\n\nNot all claim types should be treated equally.\n\n### Empirical near-duplicates\n\nThis is the safest domain for aggressive clustering.\n\nExample:\n\n- \"Global temperatures have risen about 1.1°C since 1900\"\n- \"The planet has warmed roughly 1.1 degrees Celsius since 1900\"\n\nThese are often just alternate formulations of the same operative claim.\n\n### Definitional claims\n\nModerately safe, but only with stronger structural checks.\n\nExample:\n\n- \"By freedom here I mean absence of coercion\"\n- \"Here, freedom means non-coercion\"\n\nLikely clusterable, but only if downstream use patterns do not diverge.\n\n### Normative and value-laden claims\n\nMuch less safe.\n\nExample:\n\n- \"Freedom matters\"\n- \"Freedom matters\"\n\nThese may be textually identical while resting on very different operative structures and leading to very different downstream implications. The discussion in [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md) makes this explicit: same words often hide different meaning, and even semantic agreement can mask deeper discursive divergence.\n\nSo normative clustering should require a higher bar:\n\n- semantic similarity\n- structural similarity\n- downstream-use similarity\n- possibly evaluation-pattern similarity\n\nNot just textual proximity.\n\n---\n\n## Concept Senses: The Same Idea, But More Dangerous\n\nThe same design pattern probably applies to concept senses too:\n\n- not hard merge\n- soft cluster\n- preserve lineage\n- allow splitting\n\nBut the danger is even sharper here.\n\nIf claims are over-merged, one can still sometimes recover the difference from downstream structure.\n\nIf **concept senses** are over-merged, the system may collapse the very distinction Deliberus exists to reveal.\n\nSo for concept senses the right pattern is probably:\n\n- **merge candidate**\n- **candidate canonical sense**\n- **locally stable cluster**\n\nrather than immediate full merge.\n\nThis fits the current lifecycle direction in [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md): concepts can be underdefined, emerging, bifurcated, fragmented, or locally stable. Soft clustering would be a natural governance layer on top of that, not a replacement for it.\n\n---\n\n## The Computational Implication\n\nThe original fear was:\n\n> recursion gets more expensive at each layer\n\nThe revised possibility is:\n\n> once the graph is rich enough, recursion may get cheaper per marginal claim because more decomposition steps terminate in already-known reusable substructure\n\nThis does **not** mean infinite regress disappears.\n\nIt means the practical frontier changes from:\n\n- \"how many layers can we generate?\"\n\nto:\n\n- \"how quickly can we recognize that a new decomposition path has entered already-inhabited graph territory?\"\n\nThis is a very different optimization problem.\n\nThe limiting factor becomes:\n\n- identity confidence\n- cluster precision\n- reversible canonicalization\n- reuse selection\n\nnot brute-force recursive generation depth.\n\n---\n\n## The Lean Analogy\n\nThis fits the `@[simp]` and `exact?` flywheel in [lean-deliberus-analogies.md](lean-deliberus-analogies.md).\n\nThe point of a mature library is not that every new proof must be generated from scratch. It is that the existing library increasingly lets new work terminate in retrieval and reuse.\n\nDeliberus may be heading toward an analogous regime:\n\n- the graph accumulates canonical argumentative substructure\n- new claims increasingly map onto existing subtrees\n- the system gets better at saying \"this part already exists\"\n- human labor shifts from building everything de novo to curating identity, distinction, and exceptions\n\nThat is the deeper significance of soft canonical clustering. It is not just a dedup trick. It is part of the route by which Deliberus becomes an accumulated reasoning substrate.\n\n---\n\n## Recommended Design Principles\n\n### 1. Never hard-merge by default\n\nKeep the underlying graph nodes separate unless there is an explicit, audited reason to collapse them.\n\n### 2. Prefer canonical clusters over node collapse\n\nProduct surfaces, feeds, and retrieval should default to the cluster view; the storage layer should preserve plurality.\n\n### 3. Preserve lineage\n\nEvery clustered claim should retain:\n\n- source provenance\n- author / attribution\n- source span\n- timestamps\n- extraction path\n- cluster-entry reason\n\nThis is the audit trail for why the system treated these claims as equivalent.\n\n### 4. Make splitting easier than manual merging\n\nIf clustering is automatic, human oversight should primarily focus on separating false sameness, not on endlessly performing obvious merges by hand.\n\n### 5. Use stricter thresholds for normative and definitional material\n\nEmpirical near-duplicates are safer to cluster than value claims or contested definitions.\n\n### 6. Treat concept senses as even more delicate than claims\n\nDefault to merge-candidate and locally-stable clustering states before anything like full canonical sense identity.\n\n### 7. Let downstream structure constrain identity\n\nTwo claims that look similar in text but diverge in downstream decomposition, supporting evidence, or vote/evaluation patterns should resist canonical clustering.\n\n---\n\n## The Cleanest Formulation\n\nThe right future is probably not:\n\n> full recursive auto-decomposition\n\nbut rather:\n\n> recursive decomposition under strong pruning, via retrieval-first reuse of existing subtrees, with soft canonical clustering and reversible provenance-preserving identity semantics\n\nThat preserves the wisdom of the original scheme-bounded caution while acknowledging the newer possibility opened by a denser graph.\n\n---\n\n## Addendum (Jul 6, 2026): The Mint-Time Carve — One Population Below the Fork\n\nThe first dogfood run surfaced a population this doc's caution does not need to protect: **system-minted templated text**. CQ polarity claims are instantiated from scheme templates with known slot bindings, so their identity is decidable *by construction* — `(template_id, slot_bindings)` equality at mint time, no embeddings, no philosophy at risk. Every objection in [claim-sameness-philosophical-readings.md](claim-sameness-philosophical-readings.md) (the *différend*, standpoint-flattening, sense/reference collapse) targets merging *human* expressions; none applies to the generator recognizing its own duplicate instantiations. This is candidate policy Option 4 (cluster by scheme + CQ-slot, not text similarity) applied to the one population where it is provably safe — and it unblocks the `@[simp]` flywheel's question-basis decay without waiting for the human-claim policy fork to be settled. Concept-term normalization (`Human Dignity` vs `human dignity`, observed live) is the same class: mint-time hygiene, below the waterline. The four-policy fork for HUMAN claims remains open and deliberately so. Evidence and context: [dogfood-run-1-friction-log.md §I3](dogfood-run-1-friction-log.md).\n\n## Cross-References\n\n- [scheme-bounded-decomposition-and-evidence-as-subgraph.md](scheme-bounded-decomposition-and-evidence-as-subgraph.md) — original anti-regress caution, pruning insight, and graph-aware retrieval direction\n- [self-similar-decomposition-and-claim-ontology.md](self-similar-decomposition-and-claim-ontology.md) — everything is currently undecomposed, not permanently atomic\n- [semantic-disambiguation-and-concept-tracking.md](semantic-disambiguation-and-concept-tracking.md) — why false sameness is especially dangerous for concepts and definitions\n- [lean-deliberus-analogies.md](lean-deliberus-analogies.md) — `@[simp]` / `exact?` flywheel and accumulated reusable structure\n- [conceptual-threads.md](../conceptual-threads.md) §Thread 4 — false merges worse than missed merges; concept tracking as the antidote\n- [execution-plan-clarification-first-roadmap.md](execution-plan-clarification-first-roadmap.md) — where concept governance and deeper recursive decomposition should mature in product form\n"}