{"path":"research/interaction-modalities-and-the-pointing-test.md","content":"# Interaction Modalities and the Pointing Test\n\n*Aug 17, 2026. The founder read the claim that \"whether people point at claims\" is the one signal that would redirect the whole build, and answered with an objection that weakens it and a design space that widens it. Both are recorded here, together, because they are the same move: **the interface is a variable, and a null result under one interface is not a verdict on the substrate.***\n\n> *\"But it just inspires me to enable alternative UX modes/paths or whatever. We can and should adapt to whatever use Deliberus could conceivably transmogrify into. Considering dev velocity nowadays, this is the least of our worries… UI flows/views can be designed to adapt perfectly to different use cases and contexts, if we convince ourselves the effort is warranted.\"*\n\n---\n\n## 1. What this does to the falsifier, precisely\n\n[structure-versus-scale.md](structure-versus-scale.md) § The signal that would redirect the build states it as:\n\n> *\"If people in the friends round do not point at claims — if they keep arguing at the document level with claim-level tools in hand — then addressability is wrong, and everything above it is wrong with it.\"*\n\nThe phrase carrying the weight is **\"with claim-level tools in hand\"**, and it is doing less work than it appears to. *In hand* is not *well afforded*. A clumsy affordance satisfies the condition and still produces non-pointing, so a null is under-determined across **three** explanations rather than one:\n\n1. **Claim granularity is not how people work.** The signal firing. The substrate is wrong and everything above it goes.\n2. **The structure was someone else's.** Already handled: this is why the first session is live and from-scratch rather than a tour of a pre-built graph ([islands-of-coherence.md](islands-of-coherence.md) § 5c, [ux-principles.md](../ux-principles.md) P20).\n3. **This interface never made pointing easy.** The founder's objection, previously unhandled.\n\nBranch 3 is real. Pointing at a claim in the current UI means finding it in a vertical list of extraction results and clicking its text. That is one affordance among many possible ones, and it is not obviously the best one for two people mid-argument.\n\n## 2. The danger in the same move, which has a precedent in this corpus\n\nIf the modality set is **unbounded and expandable after each null**, the signal can never fire. Every negative result gets answered with *we had not tried the right interface yet*, and a falsifier that cannot fire is precisely what got an earlier formulation of the architecture bet **retracted** on the founder's own challenge (2026-08-13: *\"its falsifier cannot fire… it argues for a transcript rather than a claim graph\"*). Conceding branch 3 without a stopping rule would rebuild that defect in a new place. The corpus's own warning about five independent reasons being unfalsifiable in aggregate applies to modalities: *N* interfaces means *N* chances to explain away a null.\n\n**The resolution is already in the project's vocabulary, and it is the terminus.** A classified terminus is *a fallible fixed point under the current move-set, never proven exhaustion* — and what distinguishes bedrock from arrested inquiry is nothing about the claim itself, it is **the moves-tried record**. Apply the same structure here:\n\n- The verdict is never *\"addressability is wrong\"*. It is **\"addressability did not pay under the modalities tried\"**, with the modalities named.\n- The modality set for a given round is **pre-registered before the session**, the way run 6 and run 7 pre-registered predictions before any source was read.\n- Expanding the set after a null is **allowed and logged as a new pre-registration**, so the number of attempts is visible rather than absorbed. That is the same discipline as *no silent caps*: an escape hatch that is counted is an honest one.\n\nThat preserves what the signal was for while conceding what the objection earned. A null under one interface is data about that interface. A null across a pre-registered set of good-faith interfaces is the signal.\n\n## 3. The modality space, from the founder's message\n\nFour candidates, in descending order of what they would buy against what they cost.\n\n### 3a. The diarized conversation — the strongest, and mostly already built elsewhere\n\n*\"There is voice input already on the landing page I believe. This could record a conversation between two people and we could use something like the meeting-pipeline from Sarpetorp (based on KBWhisper IIRC) to transcribe and scaffold/extract from the diarized contents.\"*\n\n**This is the strongest candidate and it dissolves two of P21's three hardest blockers rather than mitigating them.** Nobody types, so *latency is social* stops applying — there is no dead air because there is no submit step to wait on. And *two writers, one graph* stops applying, because there are no concurrent writers: there is one recording and one extraction.\n\n**What exists, verified rather than recalled** (`~/dotfiles/docs/2026-06-11-pyannote-profile-aware-research.md`, `~/Projects/swhisper-work`): `pyannote/speaker-diarization-3.1` for diarization, `pyannote/wespeaker-voxceleb-resnet34-LM` as a 256-dimension embedder, **KB-Whisper-Large** for Swedish ASR, and **eight enrolled speaker profiles** already captured from prior clean recordings. Profile-aware assignment went live in that pipeline on 2026-06-26, opt-in behind a flag, with two production calibrations worth inheriting: a naive per-segment override regressed muffled audio until a **margin gate** was added (override the cluster dominant only when the per-segment match clears a threshold *and* beats the dominant by a margin), and **a weak enrolled profile inverts the signal**, so the fix for a bad match is re-enrollment rather than threshold tuning.\n\n**What Deliberus has: none of it.** Measured — `diariz` and `speaker` return zero hits across `deliberus/*.py`, and zero across the whole `docs/` corpus. `POST /transcribe` is a flat single-speaker Gemini call. **So the gap is attribution, not transcription.** Transcription has shipped since March.\n\n**Why attribution specifically is worth more here than it looks.** The graph already carries `attributed_to` on claims, populated on 1,082 of 1,642 substantive claims — and extraction sets it to *the source's own author*, which is the finding that killed the sacredness brake's `attributed_to` guard. A diarized two-person session would produce the corpus's **first per-claim attribution to a live human participant** rather than to a document's byline. That is new material for the reported-speech work, the stance instrument, and the straw-man detector always-mint was chosen to buy: two people in a room routinely characterise each other's positions, and now both the characterisation and the original would be in the graph with different speakers on them.\n\n**Liveness is feasible, and the first version of this section said the opposite.** It read *\"post-hoc or incremental, never live\"*, inferred from throughput alone. The inference runs backwards: **every figure here is above unity, and throughput above real time is the condition streaming needs rather than a bar to it.** Corrected the same day on the founder's challenge — *\"3x real-time but never live? How come? It's all a matter of how you build this.\"*\n\nThe verified numbers (`~/dotfiles/docs/2026-06-27-whisper-a4000-vs-m2-research.md`, `2026-06-27-pyannote-a4000-vs-m2-research.md`): transcription **≈3× real time** on the Mac Mini M2 (whisper.cpp, KB-Whisper-Large Q5_0, Metal) and **7–12×** on the borrowed A4000 (faster-whisper/CT2, FP16); diarization **≈2.4×** on M2 MPS and **~30–60×** on the A4000. Run in sequence on the Mac, that composes to **≈1.3× real time** — above unity, and thin. On the A4000 it is **≈7×**, which is comfortable.\n\n**Enrollment widens that margin structurally rather than incrementally.** pyannote's cascade is slower than end-to-end diarizers *because of the clustering stage* (arXiv:2509.14737), and clustering is a global operation over the whole recording, which is the thing that actually favors batch. With two known speakers there is nothing to cluster: it is a per-segment embedding matched by cosine against two profiles, which is streaming-native and does not grow with conversation length. The production margin gate applies unchanged, per segment. **So the expensive stage is the one enrollment deletes** — and the eight profiles already exist.\n\nThree tiers follow, and which one ships is a design choice rather than a hardware verdict:\n\n1. **Live ambient.** Streaming ASR plus per-segment enrolled-speaker matching. The transcript appears continuously and the graph grows on a rolling boundary behind it.\n2. **Turn-boundary.** Extraction fires at a detected pause or topic shift, so structure arrives in visible increments instead of a trickle.\n3. **Post-hoc.** The whole conversation, then the graph. Now the fallback rather than the ceiling.\n\n**The real costs, once throughput stops being one.** *Window latency*: Whisper needs seconds of context, so partial hypotheses lag 2–10 seconds however fast the machine is. That is the number the interaction design owns, and it sits inside a conversation's own rhythm. *Extraction cadence*: the 8-pass pipeline takes ~2–3 minutes per source, so claims land tens of seconds to minutes behind the utterance — which makes tier 2 the honest default and tier 1 a display layer on top of it. *Money*: repeated extraction calls across one session are the hard wall while the Gemini key sits on free-tier quota (20 calls per day per model, fleet-shared). **That, not silicon, is what would cancel a live session.** *Integration*: swhisper-work runs on the fleet and Deliberus's extraction runs in a container on Darwin, so where diarization executes and how the artifact reaches the pipeline is unresolved. *And one cost that is not engineering at all*: structure displayed mid-conversation is a facilitator intervention, which is where the DeepMind study measured steering (§ 4). Tier 1 buys immediacy and takes on that risk; tier 3 avoids it and loses the immediacy. **That trade is what to pre-register — not the hardware.**\n\n### 3b. A fullscreen graph with keyboard and gesture navigation\n\n*\"We could enable a fullscreen snappy graph view with hotkeys or multitouch trackpad adaptations/use.\"*\n\nThe existing graph is an ego-network on a claim page, sized to sit inside a document. A fullscreen view navigated by keys or trackpad gestures is a different instrument: it makes the *structure* the object of attention rather than an illustration beside prose. For the pointing test this is the most direct probe available, because in that view there is nothing to point at *except* claims — the document level is not on screen. **That property is what makes it worth pre-registering**: if people still do not point there, branch 3 has been closed for the strongest affordance anyone has proposed.\n\nPrior modality work to build on rather than restart: [mobile-argument-ux.md](mobile-argument-ux.md) § 1.4 already worked out a map/explore mode, and § 2 has the microinteraction inventory.\n\n### 3c. Realtime presence between simultaneously logged-in users\n\nNamed in the founder's message and the least specified. It is the only one of the four that requires new backend behaviour rather than new frontend surface, and it inherits the untested-concurrency problem that 3a dissolves. Worth holding until a session has shown that two people want to act on the graph at the same time rather than in turns.\n\nOne consequence of the liveness correction above: **tier 1 delivers most of what this modality was wanted for without the concurrency problem.** Two people watching one graph grow from their own conversation are already co-present in the graph; what is still missing is two people *writing* to it at once, which is the part nothing has yet shown anyone wants.\n\n### 3c-bis. The modality with no human in it\n\nWorth naming here because it belongs to the same space and settles a question the other four cannot. **An agent client has no affordance to blame**, so it is the one reader for whom branch 3 does not exist. That makes it a companion instrument rather than a fifth candidate: it tests whether the substrate pays, while the four above test whether a *person* can reach it. Design and the accompanying authority risk: [agents-as-a-consumer-class.md](agents-as-a-consumer-class.md) § 3, § 5.\n\n### 3d. Claim-card clicking — the shipped baseline\n\nWorth naming as a modality rather than as *the* interface, which is the whole point of this document. It is the control arm for any comparison, and its affordance quality has never been assessed.\n\n## 4. What this changes about the first session\n\nIf modality is a variable, then **the first side-by-side rehearsal is not primarily a test of addressability — it is modality discovery.** That is a better goal for a first session and it lowers the stakes usefully: a stumble is information about an interface, not a verdict on the architecture. It also changes what to instrument. Watching *which* affordance a person reaches for when they want to disagree with a specific thing is the observation; whether pointing happened at all is the aggregate that only becomes meaningful across a pre-registered set.\n\nThe corpus already holds the caution that makes this honest: **preference is not a proxy for deliberative quality** (the DeepMind facilitation study, read in full and half-retracted in [synthesis-build-plan.md](synthesis-build-plan.md) § 9 — participants preferred facilitation that steered them and did not improve consensus). So *which modality people liked* is not the finding. *Which modality produced claim-level engagement* is.\n\n## 5. Registered caution\n\nThe founder's framing — *\"the least of our worries… the possibilities are endless\"* — is right about cost and is the thing to watch. Cheap modalities mean the temptation is to add rather than to decide, and an unbounded set is how the signal dies. The stopping rule in § 2 exists for that reason, and this document's own dependent is small and specific: **the modality set for session one is pre-registered before the session, in writing, with the affordance being tested named for each.** If that pre-registration does not happen, this document has joined the four instances of [the uncollected idea](conviction-and-critique.md).\n\n---\n\n## Cross-references\n\n[structure-versus-scale.md](structure-versus-scale.md) § The signal that would redirect the build (the falsifier this qualifies) · [islands-of-coherence.md](islands-of-coherence.md) § 5c (the session this feeds) · [../ux-principles.md](../ux-principles.md) P20, P21, P23 (the gradient, the venue, the substrate) · [mobile-argument-ux.md](mobile-argument-ux.md) (prior modality inventory, not to be restarted) · [dogfood-run-7-swedish-election-structured-vs-unstructured.md](dogfood-run-7-swedish-election-structured-vs-unstructured.md) (the language decision a Swedish spoken session inherits)\n"}