{"path":"research/practices-for-interconnected-research/practice-peers-as-a-helper-read-them.md","content":"# Matuschak, Litt and Ink & Switch, Karpathy, Willison, two failure accounts, Gwern: what they do and what stayed unsolved\n\n*A helper's report, kept word for word (2026-10-10; Sonnet, read-only web research, 39 tool uses). Asked for\nwhen the founder asked that the strongest matches in\n[practices-for-interconnected-research.md](../practices-for-interconnected-research.md) be read in more\ndepth. The helper was told to read primary sources in full and to say how deeply it read each. Nothing\nhere was checked by Claude. How it bears on Deliberus is in the parent note.*\n\n---\n\nDEPTH KEY: FULL / PARTIAL / NOT OPENED. I paraphrase throughout and use one short quote in total, to keep to my copyright limit. Pages were fetched through the mcp__912fe9e2 tools. No files were written.\n\n**1. MATUSCHAK (+NIELSEN)**\nDepth:\n- 2023 letter, \"Towards impact through intimacy\" (Dec 2022), the working note \"Collaborative knowledge management systems tend to decay\", and the 2026-03-03 MIT talk \"Apps and programming: two accidental tyrannies\" (the only 2024-26 item I could read): FULL.\n- TTFT: PARTIAL, about 85%. I skipped \"Two cheers for mnemonic techniques\", the start of the numerals section, and \"Mnemonic video\".\n- \"Five years of evergreen notes\" (Aug 2024) and \"Working with blips\" (Sep 2025): NOT OPENED. Both are Patreon-locked, and only the teasers loaded.\n\na. Practice\n- In 2023 he stopped surveys, telemetry and big experiments. He sat beside individuals for hours while they used one-off prototypes he built for each of them.\n- His rule: iterate one-on-one until a design works, and only then run quantitative tests.\n- The 2022 plan was a few demanding expert readers keeping diaries. He watches for moments when understanding feels \"shaky, brittle, or parrot-like\".\n- TTFT card rules: make cards atomic (splitting one failing card into two worked), then re-add the original to integrate them. Avoid orphan cards.\n- TTFT leaves a dated retraction in the text (9 Dec 2019: a claim rested on a data-analysis error).\n\nb. Evidence\n- In 2023 he got more insight in a few weeks than in all of 2022. One student showed that apparent forgetting was sometimes never having understood the material.\n- TTFT's informal test used 16 users who reviewed and 25 who were delayed. Accuracy went 89% to 96% with review, and 91% to 87% without. A second figure: 195 users showed a month of retention on 80% or more of cards.\n\nc. Went wrong and unsolved\n- The 2022 course experiments taught him little. He measured recall but knew little about depth of understanding.\n- TTFT says a user can answer when tested yet fail to use the knowledge: \"the user may not even recognize opportunities to use what they have learned\".\n- Users also answered from surface patterns, such as the one \"Who…?\" card.\n- In the 2026 talk, agent work left his attention fragmented, and he and his collaborators sometimes used agents to avoid thinking.\n- His note says team wikis become unusable. Notes stop being atomic, authors factor concepts differently, and terms drift.\n\nd. Copyable\n- Watch one real task on your corpus from start to finish and log where the assistant fails.\n- Test application on a novel case, not just retrieval.\n- Date any retraction in place and keep the old text.\n\ne. Not established\n- Single-user method versus statistical method was never compared. He says he is not worried about false negatives, which he does not check.\n- Nothing covers AI assistants applying notes.\n- The TTFT experiment is small, and he admits overlap between cards.\n\n**2. LITT / INK & SWITCH**\nDepth: Litt talk FULL. Patchwork notebook FULL (entries 01-11, last dated 2024-07-19).\n\na. Practice\n- Litt: after an agent finishes, his /explain-diff skill produces an explainer. It gives background first, then intuition, then a \"literate diff\" in prose order, then a five-question quiz.\n- He won't send code to others until he passes the quiz. He also has agents build \"micro-worlds\" (a Prolog debugger, a port \"command center\") so he can inspect the system himself.\n- Patchwork: an AI bot proposes edits on a branch and a human merges them fully or partly. The bot shows in history, its prompt is a versioned document, and it can leave a rationale comment per edit.\n\nb. Evidence\n- Litt reports daily use and coworker uptake, with no numbers.\n- Patchwork says its AI edit summaries beat its other diff views. That is based on weeks of the team's own writing.\n\nc. Unsolved\n- Litt: reading is hard work and easy to fake.\n- Patchwork: how history comments relate to document comments is unresolved. LLM summaries of visual edits were less convincing. Spreadsheet diffs could not tell manual edits from recalculated cells.\n\nd. Copyable\n- A per-change explainer plus quiz before accepting assistant work.\n- Assistants write on a branch and you merge.\n- Instruction files kept under version control.\n- A rationale note on each edit.\n\ne. Not established\n- No measure of understanding gained. Litt works at Notion and says he is biased. Patchwork evidence is internal and anecdotal.\n\n**3. KARPATHY LLM WIKI**\nDepth: gist FULL (April 2026). The page also carries many third-party comments, mostly tool promotion. I skimmed them and did not treat them as his.\n\na. Practice\n- Three layers: immutable raw sources, an LLM-owned wiki of markdown, and a schema file (CLAUDE.md or AGENTS.md) that the human and LLM co-evolve.\n- Ingest touches about 10-15 pages per source. He ingests one at a time and reads the summaries and updates.\n- Queries read index.md first, and good answers are filed back as pages.\n- log.md is append-only with a greppable prefix.\n- Periodic lint looks for contradictions, stale claims, orphan pages, missing cross-references and missing pages.\n- The wiki is a git repo.\n\nb. Evidence\n- None quantified. He says the index works \"surprisingly well\" at about 100 sources and hundreds of pages.\n\nc. Limits named\n- Beyond that scale, add real search.\n- Images are clunky.\n- He calls the document intentionally abstract.\n- Not addressed: verifying summaries against sources, re-ingesting a changed source, or an agent that reads a page and fails to apply it.\n\nd. Copyable\n- Index and log files, a lint checklist, immutable sources, a schema document.\n\ne. Not established\n- An idea file with no results, failure data or comparison, from one person.\n\n**4. WILLISON, \"HOARD THINGS\"**\nDepth: FULL.\n\na. Practice\n- He keeps runnable proofs of what works: over 1,000 GitHub repos, tools.simonwillison.net, and simonw/research (agent-written reports with code).\n- He tells an agent to build something new by combining two named examples, or to clone his repos and read them.\n- He tells the agent to use curl, because the default fetch tool summarizes pages.\n\nb. Evidence\n- His March 2024 OCR tool combined two code snippets in one prompt. It \"worked flawlessly\" and was done in minutes.\n\nc. Went wrong\n- Nothing is reported. Staleness of the hoard is not discussed.\n\nd. Copyable\n- Store verified how-to examples at known paths and point assistants at them.\n- Tell assistants to fetch raw text.\n\ne. Not established\n- One person, an anecdote, no miss rate. Sibling pages (walkthroughs, anti-patterns) not opened.\n\n**5. FAILURE ACCOUNTS**\nDepth: OGM page FULL. Arbital post body FULL. The Arbital comment bodies were not returned, so I read none.\n\nOGM (Oct 2021)\n- It is a diagnosis and proposal by Pete, not a postmortem. The group already had videos, transcripts, \"Jerry's Brain\" and wiki notes, and he asks who finds them usable.\n- Cleaning one recording takes about two hours per hour, plus more hours to map. After-the-fact harvesting loses nuance (\"knowledge archaeology\").\n- Proposal: build notes and maps live during the call, with talker, listener and note-writer roles, plus Miro boards.\n- Replies worry about sprawl and ground rules.\n- No outcome is recorded on the page, and I did not follow links.\n\nArbital (Alexei, Jan 2018)\n- It had a requisites system, where pages teach or require other pages, and later claim pages.\n- Only the designer used it, so features went untested. About 30 pages were being tuned for requisites that \"way too long\".\n- Few writers: about a dozen regulars, 50-200 readers a day. Almost no one would put in the effort to write explanations or claims.\n- His lesson is not to accept components he does not understand.\n- The account is from memory (his disclaimer), from one narrator, and about a commercial startup.\n\nd. Copyable\n- Capture relations while the thought is live.\n- Test a relation feature on dense real content before refining it.\n\n**6. GWERN, /about**\nDepth: FULL for content, maintenance, tags, checklist and feedback sections. I skipped the traffic regression, hosting and Benford sections.\n\na. Practice\n- Each page carries a last-meaningful-modification date and a status: notes, draft, in progress, finished.\n- Each page also carries a confidence tag from an 8-step probability-word scale, and an importance rank of 0-10.\n- Pages are kept as perpetual drafts. Predictions are added so time can falsify them. Follow-ups are scheduled, such as the 2-year review of the feedback form. Alerts watch for new results.\n- A checklist and a lint script catch predictable errors. Code is rerun, and everything is in git.\n\nb. Evidence\n- The feedback form drew 116 messages over 754 days, 2.884M page views. Of these, 34 were corrections. One author corrected figures Gwern had cited.\n\nc. Unsolved\n- No account of whether tags stay accurate. The page describes its own workflow as of late 2014.\n\nd. Copyable\n- A status, confidence and date header.\n- A lint script that flags known failure patterns.\n- Scheduled review dates.\n\ne. Not established\n- One person, with no check on tag calibration or use.\n\n(Search snippets also showed a third-party reading note quoting the locked \"Five years\" letter. I did not rely on it.)\n"}