{"path":"research/session13-notification-snr-and-extraction-outage-rca.md","content":"# Session 13: Notification SNR Rebuild and the Apr 1 Extraction Outage RCA\n\n**Date**: April 15, 2026\n**Type**: Architectural refactor + production incident root-cause analysis\n**Status**: Deployed — notifications rebuilt, forensics table live, Apr 1–15 outage root-caused and fixed end-to-end\n**Commits**: `1e6a3b5` (notifications), `7bfb32b` (forensics), `26f6969` (outage fix)\n\n---\n\n## Executive Summary\n\nThree commits, one theme: **make invisible things visible**.\n\n1. The phone buzzed all day with low-signal noise while the one event that mattered — extraction failure — was either silent or indistinguishable from page-view chatter.\n2. Failed extractions left zero trace in any persistent store. The only write path (`save_extraction`) ran only on the success branch.\n3. Those two blind spots hid a 15-day, 80% first-use failure rate caused by a single regression: `AtomicClaim.claim_kind: ClaimKind` — an enum type annotation on a Pydantic response model — introduced by Session 9's ClaimBase refactor one day after the project's day-earlier Enum-to-Literal migration. Every focused-extraction LLM response failed validation. Every validation error was swallowed by `focused._extract_one`'s silent `except Exception: return empty` block. Every affected workflow returned `n_claims=0` and completed \"successfully.\" Every affected user got a 404 and left.\n\nThe three problems were discovered in that order; each fix surfaced the next.\n\n---\n\n## Problem 1: Ntfy SNR Collapse\n\n### Symptom\n\n\"The meaningful ones are buried in a sea of meaningless ones. SNR must become way higher.\" Every extraction pipeline stage fired its own ntfy push (8 per extraction). Every page view from Fredrik's own browsing fired a push. Every minute of any traffic fired an activity-summary push. Truth-graph queries double-fired (`query_submitted` + `query_completed`). Scanner probes were correctly silenced, but search/AI bots (Googlebot, GPTBot, ClaudeBot, PerplexityBot, Applebot, AhrefsBot, facebookexternalhit) hitting real routes like `/claim/*` and `/extraction/*` were not. The ntfy message cache showed **105 messages in 48h**, 95 of them noise, zero of them the things Fredrik wanted to see.\n\nMeanwhile, the actually-important events (new user, intellectual contribution, extraction failure, system error) were at `default` priority (same as the noise) or — in the case of `extraction_failed` — defined but never called. The loudest event in the system was silent.\n\n### Design\n\n**Interrupts only.** Every notification path was audited and reclassified into three tiers:\n\n- **Interrupt-worthy** (ntfy, `default`/`high`/`urgent`): `new_user` (high), `user_contribution` (high), `extraction_failed` (high), `system_error` (urgent), `client_error` (default, deduped), `extraction_started` / `extraction_completed` (default, only for external users).\n- **Aggregated** (one `low`-priority push per day): `page_view`, `doc_read`, `query_completed`. A daily digest accumulator runs in an asyncio task under FastAPI's lifespan, flushes once 24h has elapsed, summarizes top pages/docs/queries.\n- **Silent** (logs + Prometheus only): pipeline intermediate stages, scanner probes, any event attributed to the owner or a detected bot.\n\n### The owner-vs-housemate question\n\nThe home WAN is shared with housemates, guests, and the Deco mesh. A naive \"block my home IP\" filter would silence their genuine traffic too. The fix is layered:\n\n1. **Session email** matches `DELIBERUS_OWNER_EMAIL` (primary signal when logged in).\n2. **HMAC-signed `deliberus_owner` cookie** set automatically during the Google OAuth callback when the authenticated email is the owner. 365-day max-age, HttpOnly, SameSite=lax. Covers browsing across devices and survives session expiry.\n3. **Narrow IP allowlist** (`DELIBERUS_OWNER_IPS`) — empty by default, intended only for Fredrik-only infrastructure egress (Darwin WAN, WireGuard exit), **never the home WAN**.\n\nLog in once per browser → that browser is silent forever. Housemates and guests are counted normally. Bots are filtered independently.\n\n### Temporal migration via `workflow.patched`\n\nNew executions skip the old per-stage ping calls (`_notify_stage`) and fire a single lifecycle notification at completion / empty / failure. Legacy in-flight replays take the old path — the activity body is a log-only no-op, so nothing reaches ntfy either way, but replay correctness is preserved. The patch id is `v2-notifications-lifecycle-only`.\n\n### What shipped in `1e6a3b5`\n\n- `deliberus/notifications.py` (73% rewrite): `is_owner`, `is_bot_ua`, HMAC owner cookie, `notify` + 8 event functions + `record_*` digest accumulators + `flush_digest_if_due`.\n- `deliberus/api.py`: middleware computes `request.state.is_owner` and `request.state.is_bot` once, all downstream handlers respect them; `/query` calls `record_query` instead of double-notifying; `/api/docs/*` uses `record_doc_read`; lifespan spawns the daily digest task.\n- `deliberus/auth.py`: Google callback calls `set_owner_cookie(response)` when the authenticated email is the owner.\n- `deliberus/temporal/workflows.py`: `workflow.patched(\"v2-notifications-lifecycle-only\")` guard; new `_notify_lifecycle` helper; old `_notify_stage` calls gated to legacy-only.\n- `deliberus/temporal/activities.py`: new `workflow_lifecycle_activity(WorkflowLifecycleInput)` activity; old `notification_activity` demoted to log-only no-op for replay correctness.\n- `deliberus/temporal/models.py`: new `WorkflowLifecycleInput` dataclass (grows url + text_preview + n_arguments fields in commit 2).\n- `deliberus/temporal/worker.py`: register the new activity.\n- `.env` / `.kamal/secrets` / `config/deploy.yml`: `DELIBERUS_OWNER_EMAIL`, `DELIBERUS_OWNER_IPS`, `DELIBERUS_OWNER_COOKIE_SECRET`.\n\nTests: 10 new in `tests/test_infrastructure.py` (owner self-filter, bot UA patterns, digest accumulation and flush, client-error dedup, enriched failure body, HMAC cookie). 711 total pass.\n\n---\n\n## Problem 2: Failed Extractions Left No Trace\n\n### Symptom\n\nWith the ntfy noise gone, Fredrik asked: *\"Any failed extractions we should know about?\"* and *\"I know for a fact that carl.bror@gmail.com tried to submit an extraction the other day and got an error back about that it found no arguments in the url or something.\"*\n\nPostgres showed **11 successful extractions, all dated March 31** — nothing since. The `extractions` table had no row for any of them. The users table showed **8 of 10 users with zero successful extractions**, including a cognitive scientist, an EA-Sweden-affiliated researcher, and several early adopters from the rationalist/AI-safety community.\n\nTemporal's workflow retention is 72 hours. Only one workflow survived (User G's, started Apr 12 09:39:53 UTC, just inside the cliff). kamal-proxy's `json-file` log driver had 3 days at 10 MB rotation — enough to see Apr 12 browser sessions by IP + UA but not Apr 10 (User G's signup day) or earlier. The ntfy message cache has 48h retention. The previous container's logs were pruned on the notification-refactor deploy. Every diagnostic vault had just emptied out.\n\n### Why no URL gets stored on failure\n\nThe Temporal workflow was written for the success path only. The one and only write to Postgres happens inside `store_activity`, which runs AFTER the `implicit_out.n_claims == 0` check passes. When that check fails (empty branch) or when any earlier activity raises (failed branch), the workflow returns/re-raises — nothing calls `save_extraction`, so **no row exists to store the URL in**. The input (URL, text, title, user email) sits on `ExtractionInput` the entire time; the code just never writes it anywhere on failure.\n\nThis is the exact \"make failures visible before fixable\" principle from CLAUDE.md, violated at the persistence layer. The impulse was \"add a retry\" or \"improve the error message\"; the real fix is decomposing the aggregate failure into components and persisting them.\n\n### Fix: the `extraction_attempts` forensics table\n\n```sql\nCREATE TABLE extraction_attempts (\n    workflow_id   VARCHAR(256) PRIMARY KEY,\n    user_email    VARCHAR(500),\n    input_type    VARCHAR(20),\n    url           VARCHAR(2048),\n    title         VARCHAR(1000),\n    pdf_filename  VARCHAR(500),\n    text_preview  TEXT,            -- first ~500 chars\n    status        VARCHAR(20),     -- started | completed | failed | empty\n    source_id     VARCHAR(256),    -- resolved by fetch_url_activity\n    n_arguments   INTEGER,\n    n_claims      INTEGER,\n    error         TEXT,            -- exception message, truncated to 2000\n    created_at    TIMESTAMP DEFAULT NOW(),\n    updated_at    TIMESTAMP DEFAULT NOW()\n);\n-- indexed on (user_email), (status), (created_at DESC)\n```\n\nWrite path:\n\n1. `_start_temporal_extraction` in `api.py` calls `save_attempt_started(workflow_id, user_email, input_type, url, title, pdf_filename, text_preview)` **before** `client.start_workflow`. Even a fetch-activity crash leaves a forensic row.\n2. `fetch_url_activity` calls `mark_attempt_source(workflow_id, source_id)` once the slug is generated, correlating attempt rows to successful extraction rows.\n3. `workflow_lifecycle_activity` calls `mark_attempt_status(workflow_id, status, n_arguments, n_claims, error)` with the terminal state. Self-filtering on `user_email` still applies to the ntfy push, but the row is always written.\n\nQuery:\n\n```sql\nSELECT workflow_id, user_email, status, url, text_preview, n_arguments, error, created_at\n  FROM extraction_attempts\n WHERE status IN ('failed', 'empty')\n ORDER BY created_at DESC\n LIMIT 20;\n```\n\n### Enriched failure notification\n\n`notifications.extraction_failed(source_id, error, user_email, url, text_preview)` now builds a message body containing user, URL, and a 200-char preview. The ntfy push itself is enough forensic evidence to diagnose a failure from the phone — no archaeology required.\n\n### The `quality_score` universal NULL\n\nSeparate bug surfaced during the audit: **every one of the 11 Postgres extractions had `quality_score = NULL`** despite `self_eval_activity` successfully computing scores of 0.75–0.98 and writing them into `combined_json[\"self_eval\"][\"overall_quality_score\"]`. Root cause: `store_activity` in `deliberus/temporal/activities.py:387` hardcoded `quality_score=None` in the `save_extraction` call because `StoreInput` didn't carry the field. The inline non-Temporal fallback path (`/extract/stream`) read it correctly; the Temporal path regressed. Fixed by reading from `combined[\"self_eval\"][\"overall_quality_score\"]` inside the activity — no schema change, no dataclass migration.\n\n### What shipped in `7bfb32b`\n\n- `deliberus/database.py`: `ExtractionAttempt` SQLAlchemy model, `save_attempt_started`, `mark_attempt_source`, `mark_attempt_status`; `init_db` adds the table and three indexes via `CREATE TABLE IF NOT EXISTS` + `CREATE INDEX IF NOT EXISTS`.\n- `deliberus/api.py`: writes the `started` row before `client.start_workflow`.\n- `deliberus/temporal/models.py`: `ExtractionInput.workflow_id` field; `WorkflowLifecycleInput.url` + `text_preview` + `n_arguments`.\n- `deliberus/temporal/activities.py`: `fetch_url_activity` calls `mark_attempt_source`; `workflow_lifecycle_activity` calls `mark_attempt_status`; `store_activity` reads quality score from `combined[\"self_eval\"]`.\n- `deliberus/temporal/workflows.py`: lifecycle calls pass `url`, `text_preview`, `n_arguments` through.\n- `deliberus/notifications.py`: `extraction_failed` signature grows `url` + `text_preview`, body includes them.\n- `tests/test_database.py`: 10 new tests for the attempt model and helpers, including never-raises-on-DB-error.\n- `tests/test_infrastructure.py`: regression test for the enriched failure body.\n\n721 tests pass. Schema live in production via `init_db` idempotent DDL.\n\n---\n\n## Problem 3: The Apr 1 Extraction Outage\n\n### Symptom\n\nEven with the forensics layer shipped, the harder question remained: *what was actually breaking for these users?* The forensics table would capture the next attempt, but the backlog was silent.\n\nRunning the reproducer user's exact URL through a local reproduction (MERIAN-hosted Python against the same Darwin-hosted Gemini that production uses) gave 0 claims and a `print` statement swallowed by a silent except block. Removing the silent except and re-running revealed the real exception:\n\n```\n8 validation errors for FocusedResult\nclaims.0.claim_kind\n  Input should be an instance of ClaimKind [type=is_instance_of, input_value='extracted', input_type=str]\nclaims.1.claim_kind   ... (same error, 7 more times)\n```\n\nGemini returned 8 valid-looking atomic claims per argument. Every one had `claim_kind: 'extracted'` as a plain string. Pydantic v2 strict validation rejected all 8 because the field was typed as the `ClaimKind` Enum class — *even though `ClaimKind` inherits from `str`*, instructor's `from_provider` + `GENAI_TOOLS` mode passes the plain string through and Pydantic's `is_instance_of` check on the enum class returns False for raw strings.\n\n### How the regression entered the codebase\n\nExact git archaeology:\n\n| Commit | Date | What |\n|---|---|---|\n| `ae8a2f7` | Mar 30 01:32 | Scheme-bounded decomposition: added the `ClaimKind` enum class as a value holder (not yet as a response-model field annotation). |\n| `5456f8e` | Mar 31 07:57 | **\"Migrate to instructor from_provider + Literal types (fixes enum bug)\"** — converted every response-model enum field to Literal. `claim_kind` did not yet exist as a field annotation, so the migration passed it by. |\n| Mar 31 | all day | 11 successful production extractions (Fredrik + one external user). |\n| `0d8a8e3` | Apr 1 12:25 | **\"Session 9: atomicity improvements, DRY refactor, ClaimBase ontology, Frontier keyword\"** — introduced `AtomicClaim` as a `ClaimBase` subclass with the line: `claim_kind: ClaimKind = Field(default=ClaimKind.extracted, ...)`. The Literal migration had already shipped; the new field was typed as the Enum class; no test in the suite exercised the LLM response-validation path for this field. |\n| Apr 2–15 | 14 days | 7 new users signed up. **Zero successful extractions**. |\n\nCLAUDE.md explicitly documents this gotcha: *\"Enum → Literal in all Pydantic response models (GENAI_TOOLS returns strings that strict validation rejects as non-enum-instances).\"* Session 9 missed one field, and the \"silent capability degradation is the deadliest bug class\" rule was violated simultaneously in `focused._extract_one`:\n\n```python\ndef _extract_one(arg, source_text, model):\n    try:\n        return extract_claims(arg, source_text, model)\n    except Exception as e:\n        print(f\"    ⚠ Failed for '{arg.label}': {e}\")   # stdout in dev, nothing in prod\n        return FocusedResult(claims=[], argument_label=arg.label)\n```\n\n`print` (not `logger`), then an empty return. Every argument failed identically and the workflow returned `ExtractionResult(n_claims=0)` as a \"successful\" completion. Temporal marked it `Completed`. The frontend navigated to `/extractions/<slug>` which 404'd because nothing had been saved. The user saw a broken experience and left.\n\n### Blast radius\n\nUser signup vs. successful extractions, queried just before the fix:\n\n| User | Signup | Extractions | Status |\n|---|---|---|---|\n| Fredrik Bränström | Mar 31 | 10 | pre-regression (seed content) |\n| Olle Bjerkås | Mar 31 | 0 | probably never tried |\n| (one external user) | Mar 31 | 1 | pre-regression (before Session 9) |\n| User A | Apr 2 | 0 | **regression active** |\n| User B | Apr 2 | 0 | **regression active** |\n| User C | Apr 4 | 0 | **regression active** + Kamal-secrets-incident overlap |\n| User D (cognitive scientist) | Apr 4 | 0 | **regression active** — exact target persona |\n| User E | Apr 4 | 0 | **regression active** |\n| User F (EA-Sweden-affiliated) | Apr 8 | 0 | **regression active** — the Apr 8 case documented in [truth-graph-evidence-system.md](truth-graph-evidence-system.md) |\n| User G | Apr 10 | 0 | **regression active** — the reproducer case |\n\n15 days. Seven affected users. 80% first-use failure rate. Zero diagnostic signal in production until this session.\n\n### One user's exact incident (reconstructed)\n\nTemporal's 72h retention preserved workflow `extraction-d2cb7074cae1` — the only surviving workflow in the entire namespace, about to expire. The full session was recovered by correlating that workflow's activity payloads with kamal-proxy's JSON access logs filtered by the user's source IP and browser fingerprint.\n\n- **URL**: `https://ordningochanarki.blogspot.com/2014/04/ett-egoistiskt-skal-att-agera.html`\n- **Title**: \"Ordning och anarki: Ett egoistiskt skäl att agera altruistiskt\" — a 2014 Swedish Effective-Altruism-adjacent blog post (6492 chars). Matches the user's profile: a Swedish-speaking new user browsing Swedish altruism content.\n\nTimeline (Apr 12, 2026):\n\n```\n09:33:50  GET 200 /                                    landing page\n09:33:50  GET 401 /auth/me                             not yet authed\n09:33:54  GET 302 /auth/google/login                   clicked sign-in\n09:33:55  GET 307 /auth/google/callback                back from Google\n09:33:55  GET 200 /auth/me                             authenticated\n09:35:39  GET 200 /extractions/basic-ai-drives-omohundro   browsed a seed extraction to see what success looks like\n09:38:05  POST 200 /extract/preflight                  pasted the URL\n09:39:53  POST 200 /extract/start                      clicked Extract → workflow d2cb7074cae1\n09:41:29  GET 200 /extract/status/extraction-d2cb7074cae1/stream   SSE closed (workflow done)\n09:41:30  GET 404 /extractions/ordning-och-anarki-ett-egoistiskt-sk-l-att-agera-altruistisk\n09:41:30  POST 200 /api/client-error                   frontend reported the 404\n09:42:37  GET 200 /extractions                         gave up, browsed seed content\n09:49:46  GET 200 /extractions/language-and-thought-the-view-from-llms   checked one more, then left\n```\n\nActivity payloads from the Temporal workflow history:\n\n```\nfetch_url_activity:        text fetched, 6492 chars Swedish, title decoded\nscout_activity:            6 argument structures (translated to English):\n                              - Self-interest and public utility\n                              - Market Altruism (Helping as Self-Fulfillment)\n                              - Superiority of World B\n                              - Egoist Construction of Altruistic Society\n                              - Mistaken Self-Interest in Violence\n                              - Future Rational Cooperation\nfocused_extraction_activity:  claims_json: \"[]\", n_claims: 0   ← the silent swallow\nimplicit_premises_activity:   n_claims: 0, n_implicit: 0\nconcepts_activity:           0 contested concepts\nrelationships_activity:      0 relationships\nself_eval_activity:          overall_quality_score=0.0, summary:\n  \"The extraction output is completely empty. The source text contains\n   numerous complex arguments regarding self-interest, utilitarianism,\n   and social cooperation, yet no claims or relationships were extracted.\"\n```\n\nSelf-eval *knew* the result was broken. Nothing in the pipeline acted on that signal. The workflow returned `Completed`.\n\n### The fix\n\nA `ClaimKindLit = Literal[\"extracted\", \"cq_positive\", \"cq_negative\", \"implicit\", \"human\"]` alias was added, and `AtomicClaim.claim_kind` was retyped from `ClaimKind` to `ClaimKindLit`. The `ClaimKind` enum class itself was preserved for backward compatibility — its members still equal their string values via the `(str, Enum)` inheritance, so all the existing `ClaimKind.extracted` references in `graph/store.py`, `feed.py`, `api.py`, `decomposition.py`, and elsewhere continue to work.\n\nDefense in depth:\n\n- `focused.py::_extract_one`: `print + return empty` anti-pattern upgraded to `logger.exception(...)` with full argument context. The fallback to empty remains so one bad argument doesn't abort a six-argument extraction, but the real failure is now visible in the container logs.\n- `focused.py::extract_claims`: the `if not decompose_result.atoms: return empty` fast path now emits `logger.warning` before returning.\n- `tests/test_models.py::test_claim_kind_accepts_plain_string`: new regression test that asserts every `ClaimKindLit` value round-trips as a plain `str` through `AtomicClaim`. This exact test would have caught the Session 9 regression within 2 seconds of running the suite.\n\n### Verification\n\nRe-running the same URL against the production Gemini backend, after the fix:\n\n```\nscout found 6 argument structures\n  [0] 'Market Altruism':                        3 claims\n  [1] 'Helping as Self-Fulfillment':            5 claims\n  [2] 'Superiority of World B':                 6 claims\n  [3] 'Egoist Construction of Altruistic...':  32 claims\n  [4] 'Mistaken Self-Interest in Violence':     6 claims\n  [5] 'Future Rational Cooperation':            5 claims\nTOTAL CLAIMS EXTRACTED: 57\n```\n\nBefore fix: 0 claims. After fix: 57. 722 tests pass. Deployed on `26f6969`.\n\n---\n\n## Recovery Hunt: What Was Retrievable\n\nEvery diagnostic vault for Apr 1–15 was either already empty or about to be:\n\n| Source | Retention | Result |\n|---|---|---|\n| Temporal workflow history | 72h | Only one user's Apr 12 workflow survived — hours from expiring. Full input + activity payloads recovered. |\n| kamal-proxy JSON access log | `json-file` + 10 MB rotation → ~3 days | Apr 12 session fully reconstructable by IP + UA. Apr 10 (the user's signup) just past the cliff. |\n| ntfy server cache | 48h (server.yml) | Empty of anything older than Apr 13. |\n| Previous container logs | `docker rm` during deploy prune | Pruned. |\n| Postgres `extractions` table | Infinite | Nothing, because the failure path never called `save_extraction`. |\n| FalkorDB `deliberus_extraction` graph | Infinite | Clean — no orphan sources, no stray claims. |\n| Prometheus counters | Unconfirmed | Would show count by `status` label but no URL/user attribution. Not pursued. |\n| Temporal namespace archival | Disabled | Nothing. |\n\n**The lesson here is real-time**: retention cliffs are adversarial. When diagnosing a stale incident, assume the clock is running on every source and harvest in parallel before any of them tick over. The workflow was literally minutes from expiring when I pulled it. The kamal-proxy log rotation would have dropped another day of context if I had waited. Had this session started four hours later, even the reproduction would have been impossible without re-running the broken pipeline against fresh traffic.\n\nFor everyone other than the reproducer user, the specific URLs and error traces are gone. The affected users are worth a personal ping — they're high-value target users who never saw Deliberus working. One user's Apr 8 case is preserved in [truth-graph-evidence-system.md](truth-graph-evidence-system.md); the URL (a capitalism-incentives prompt) is documented there.\n\n---\n\n## Prevention (shipped, not promised)\n\n1. **Regression test** `test_claim_kind_accepts_plain_string` asserts every `ClaimKindLit` value round-trips as a plain `str` through `AtomicClaim`. Trivial to write, would have caught the Session 9 regression in 2 seconds.\n2. **Forensics table** `extraction_attempts` guarantees that every future failure — fetch error, scout empty, focused validation, storage crash, timeout — leaves a row with `user_email`, `url`, `text_preview`, `error`, timestamps. `SELECT * FROM extraction_attempts WHERE status IN ('failed','empty') ORDER BY created_at DESC LIMIT 20;` is the new forensics starting point.\n3. **Enriched failure notifications** carry url + user + preview in the ntfy push body. No more \"meaningful ones buried in a sea of meaningless ones\" — failures now self-diagnose from the phone.\n4. **Loud exception logging** in `focused._extract_one` ensures the next bug in this class surfaces a real traceback in the container logs, not a swallowed `print`.\n5. **Retention cliff awareness**: any future stale-incident investigation should start by cataloguing what's about to expire — Temporal 72h, kamal-proxy 3d at current size, ntfy 48h — and harvesting in parallel.\n\n---\n\n## Lessons\n\n### 1. Silent capability degradation is the deadliest bug class\n\nCLAUDE.md has warned about this for months. This session provided the textbook example: one missed enum field + one `except Exception: return empty` = 15 days of production outage with zero signal. The pipeline was \"running fine\" in every monitoring dashboard, self-eval was generating quality scores of `0.0` with damning summaries, Temporal was marking workflows `Completed`, and 80% of first-time users were getting 404s. The rule applies everywhere — **never write `except Exception: return empty` without a `logger.exception` above it**, and **never annotate a response-model field as an Enum class when using instructor + GENAI_TOOLS**.\n\n### 2. Failures must leave a forensic trace in persistent storage\n\nNot logs (pruned), not ntfy (48h), not Temporal (72h), not browser console (gone on navigate). Persistent storage — the same tier where success rows live. The cost of one extra DB row per submission is nothing compared to the cost of 15 days of blind failure. The `extraction_attempts` table is the architectural fix: every submission creates a row at the API boundary, before any activity runs, updated by lifecycle events. The success path remains unchanged; the failure path now writes the same shape.\n\n### 3. Adoption depends on first-impression reliability\n\nUX Principle 1 of the project — \"Stellar user experience is key to adoption\" — has a corollary that wasn't explicit: **the first extraction has to work**. Every first user except Fredrik had a broken experience during the outage. One external user succeeded only because they signed up and submitted on Mar 31, before the regression. User D (cognitive scientist, exact target persona) signed up Apr 4 and never saw Deliberus working. For an adoption-fragile product, this is the expensive class of bug: you can't un-break someone's first impression, and the cost of each lost user is not just one session but potentially the entire relationship.\n\nThe forensics table + enriched failure notifications won't bring those users back, but they will make the next outage visible before it burns seven more.\n\n### 4. Retention cliffs are adversarial to investigation\n\nDiagnostic data has a half-life. Temporal's 72h retention nearly deleted the user's workflow before it could be examined. The `docker rm` during deploy prune destroyed container logs that were two deploys old. The ntfy 48h cache held exactly none of the relevant period. **When investigating a stale incident, harvest in parallel and assume every source is on a clock.** A 1-hour delay can erase the reproducer.\n\n### 5. The \"make failures visible before fixable\" principle is recursive\n\nCLAUDE.md says: *\"STOP when the impulse is 'add a retry', 'increase timeout', 'build a guard.' That impulse means you can't see the failure clearly yet. Add DIAGNOSTICS first — decompose aggregates ('score=bad') into components ('A=good, B=bad, B2=specifically wrong').\"* This session applied the principle at three layers in sequence:\n\n1. Ntfy was aggregating \"all events\" into one noisy stream — decomposed into interrupt/digest/silent tiers.\n2. Failures were aggregating into \"nothing visible in Postgres\" — decomposed into a per-attempt forensics row with status/error fields.\n3. focused._extract_one was aggregating \"all exception types\" into \"return empty\" — decomposed into `logger.exception` with full argument context.\n\nEach decomposition exposed the next layer's invisibility. The ClaimKind enum regression was only diagnosable once all three layers were fixed.\n\n---\n\n## Cross-References\n\n- **Commits**: `1e6a3b5` (notifications), `7bfb32b` (forensics), `26f6969` (outage fix) — all deployed Apr 15, 2026\n- **Prior context**: [session12-production-recovery-and-secrets-fix.md](session12-production-recovery-and-secrets-fix.md), [session11-merian-deploy-recovery-and-secrets-drift.md](session11-merian-deploy-recovery-and-secrets-drift.md), [truth-graph-evidence-system.md](truth-graph-evidence-system.md) (Apr 8 short-input case)\n- **Global CLAUDE.md**: \"Silent capability degradation is the deadliest bug class\", \"Make failures visible before fixable\", \"Enum → Literal in all Pydantic response models\"\n- **Project CLAUDE.md**: extraction pipeline architecture, Temporal gotchas, notification SNR directive (new)\n- **Related insights**: [ux-principles.md](../ux-principles.md) on first-impression fragility, [technical-direction.md](../technical-direction.md) on forensics as first-class architecture\n"}