{"path":"research/session12-production-recovery-and-secrets-fix.md","content":"# Session 12: Production Recovery — Secrets Mangling Diagnosis and Fix\n\n**Date**: April 4, 2026\n**Type**: Production incident / deployment recovery\n**Status**: Resolved — deliberus.com fully restored\n\n---\n\n## The Problem\n\ndeliberus.com landing page showed zero extractions despite 11 extractions (792 claims) existing in Postgres. The site appeared functional but was silently degraded: `/health` returned `{\"falkordb\": \"error\", \"status\": \"degraded\"}` and `/extractions` returned `[]`.\n\n## Root Cause\n\nThe Codex session on April 3 (Session 11) deployed from MERIAN with a broken `.kamal/secrets` file. The file had been \"improved\" from simple grep:\n\n```bash\n# CORRECT (commit 0d8a8e3, last good deploy Apr 1)\nDATABASE_PASSWORD=$(grep '^DATABASE_PASSWORD=' .env | cut -d'=' -f2- | tr -d \"'\")\n```\n\nto an env-first fallback pattern:\n\n```bash\n# BROKEN (commit e6c5f7a, Session 11)\nDATABASE_PASSWORD=$(printf '%s' \"${DATABASE_PASSWORD:-$(grep '^DATABASE_PASSWORD=' .env | cut -d'=' -f2- | tr -d \"'\")}\")\n```\n\nThis hit the **known Kamal `${VAR:-default}` stripping bug** (documented in global CLAUDE.md since January 2026): Kamal's dotenv parser strips `${`, leaving `:-default}` as a literal string suffix. Every secret in the running container was corrupted:\n\n| Secret | Container value | Expected |\n|--------|----------------|----------|\n| `DATABASE_PASSWORD` | `deliberus:-deliberus}` | `6b73407126f54e49668e7297e1ccfbbb` |\n| `FALKORDB_PASSWORD` | `falkordb:-falkordb}` | `falkordb` |\n| `GOOGLE_CLIENT_ID` | `:-}` | (real OAuth client ID) |\n\nThe irony: Session 11's doc (§9) describes this fix as \"conceptually correct, parser-compatible.\" It was parser-incompatible — the exact pattern the project's own documentation warns against.\n\n## Why Data Was Safe\n\nThe Postgres accessory container (`deliberus-postgres`) runs independently with its own env vars set at creation time. The password mismatch only affected the *web container's ability to connect* — the database itself was never touched. All 11 extractions remained intact throughout.\n\n## Additional Discovery: Missing Production Secrets\n\nThe production Postgres password (`6b73407126f54e49668e7297e1ccfbbb`) was not stored in any secrets tier — it existed only inside the running Postgres container's environment. This is a gap: if the Postgres accessory were recreated, the password would need to be known.\n\nThe Knot registry password was found in `~/Projects/knot/.env` on MERIAN.\n\nGoogle OAuth credentials were not on MERIAN and not on Darwin. Mac Mini (the original deploy host) was unreachable. The user retrieved them from Google Cloud Console during this session.\n\n## The Fix\n\n1. **Reverted `.kamal/secrets`** to the original simple grep pattern from commit `0d8a8e3` — the last known good deploy\n2. **Updated `.env`** with correct production secrets: real Postgres password, Knot registry password, Google OAuth credentials, Gemini API key, ntfy token\n3. **Deployed from MERIAN** — build on Darwin remote builder, image pushed to registry, container rolled out with correct env vars\n4. **Cleared stale deploy lock** — `rmdir` (not `rm -rf`, which Codex destructive guard blocks)\n\nPost-deploy verification: `/health` returns `{\"status\": \"ok\", \"falkordb\": \"ok\"}`, `/extractions` returns all 11 extractions.\n\n## Code Audit\n\nAn oracle-agent audit of all code changes since the last good deploy (10 commits, ~4,144 lines across 20 files) found no deployment risks: no new dependencies, backward-compatible Temporal dataclass changes, proper error handling on all new endpoints, no schema migrations needed. The deploy was safe from a code perspective — the only issue was secrets.\n\n## Lessons\n\n1. **The `.kamal/secrets` gotcha is load-bearing**: the global CLAUDE.md documents this pattern clearly, but it was violated anyway by an agent that didn't check. This suggests the warning needs to be in the *project-level* CLAUDE.md too, not just global.\n\n2. **Production secrets must be in a recoverable location**: a password that exists only inside a running container is one `docker rm` away from unrecoverable. The Deliberus production DB password should be added to `tier-servers.env`.\n\n3. **Session 11's assessment was optimistic**: the doc correctly identified that the deploy hit trouble from secret drift, but its §9 (\"Final fix: env-aware, parser-compatible\") described the broken pattern as the solution. This session proved it was the problem.\n\n4. **MERIAN is now a verified deploy host**: Ruby 3.3.6, Kamal 2.11.0, Docker CLI 24.0.9, buildx v0.11.2, correct `.env` with all production secrets. Deploy completes in ~54 seconds.\n\n---\n\n## Cross-References\n\n- [session11-merian-deploy-recovery-and-secrets-drift.md](session11-merian-deploy-recovery-and-secrets-drift.md) — the session that introduced the broken pattern\n- Global CLAUDE.md § \"Kamal Secrets Gotchas\" — the documented `${VAR:-default}` warning\n- Global CLAUDE.md § \"Kamal Deployment Patterns\" — self-contained secrets pattern\n"}