{"path":"research/corpus-graph/graph-retrieval-measured-gotchas-as-a-helper-read-it.md","content":"# Graph indexes against plain retrieval: what has been measured, and the traps\n\n*A helper's report, kept word for word (2026-10-10; Sonnet, read-only web research, 26 tool uses). Asked\nfor before the corpus graph was built, on the founder's word: \"Research gotchas and insights a bit online\nand locally to keep in mind here first.\" The helper was told to read primary sources and to say how deeply\nit read each. Nothing here was checked by Claude. How it bears on the build is in\n[README.md](README.md).*\n\n---\n\nResearch on graph indexes versus plain retrieval, using only the two Exa tools. I did not touch local files.\n\n**Scale note (my inference):** the benchmark corpora were 4k to 23k passages (HippoRAG 2 Table 1). Your 1.4M words is probably around 7k chunks, so those effect sizes are the relevant ones, not the multi-million-token corpora in the GraphRAG and LightRAG papers.\n\n**1. Graph vs vector: wins, ties, losses**\n\n- **Han et al., \"RAG vs GraphRAG\".**\n  - Finding: under one protocol, vector RAG wins or ties on single-hop and detail questions. Graphs win on bridging and temporal questions.\n  - NQ F1: RAG 64.8, HippoRAG2 61.0, GraphRAG-Local 63.0, triplet-only KG 34.3.\n  - HotpotQA F1: RAG 60.0, HippoRAG2 63.0.\n  - MultiHop-RAG overall: RAG 67.0, HippoRAG2 70.3. Temporal: 30.7 vs about 50. Inference: tie (92.2 vs 91.5). Null queries: RAG 96.0 vs GraphRAG-Global 19.3.\n  - URL: https://arxiv.org/abs/2502.11371. Read: PARTIAL (intro, setup, Tables 1-2; not hybrids or efficiency).\n  - Implication: expect gains of roughly 0-3 points on passage-answerable questions and larger gains only on bridging. Keep plain retrieval as the base and the graph as an add-on.\n- **GraphRAG-Bench (Xiang et al., ICLR 2026).**\n  - Finding: HippoRAG2 vs reranked RAG on novels: fact retrieval 60.1 vs 60.9 (tie), complex reasoning 53.4 vs 42.9, summarize 64.1 vs 51.3. On medical it is +1.6, +3.3 and -2.7.\n  - MS-GraphRAG on medical fact retrieval: 38.6 vs 64.7. LazyGraphRAG on novel fact retrieval: 51.7 vs 60.9.\n  - Prompt size: MS-GraphRAG global search up to about 40k tokens, HippoRAG2 about 1k.\n  - URL: https://arxiv.org/abs/2506.05690. Read: PARTIAL (through Table 2, plus observations 1-9 via excerpts).\n  - Implication: \"what bears on this note\" is the complex-reasoning case. Return passages or notes, not long summaries.\n- **Debiased evaluation.**\n  - Finding: LLM-judge win rates are unreliable. LightRAG's self-comparison can win 90/10. A 25-token length gap on 200-token answers moves win rate by more than 50 points. LightRAG's reported 66.7% win vs naive RAG fell to 39.1% when debiased, and naive RAG slightly wins overall.\n  - URL: https://arxiv.org/abs/2506.06331. Read: SNIPPET (long excerpts).\n  - Implication: evaluate your index on gold-evidence recall, or counterbalance order, match length and repeat trials.\n\n**2. Microsoft GraphRAG and LazyGraphRAG**\n\n- **LazyGraphRAG.**\n  - Finding: it builds a noun-phrase co-occurrence graph with no LLM at index time. Index cost equals vector RAG and is 0.1% of full GraphRAG. Query cost is 4% of GraphRAG global search for better quality, and more than 700 times lower for comparable quality.\n  - All of this was Microsoft's own test: an LLM judge, 100 synthetic queries, 5,590 AP articles. Vector RAG still wins on relevance for DataLocal queries.\n  - URL: https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/. Read: FULL main text. BenchmarkQED post (https://www.microsoft.com/en-us/research/blog/benchmarkqed-automated-benchmarking-of-rag-systems/): SNIPPET.\n  - Implication: a cheap deterministic co-occurrence graph is Microsoft's own answer to index cost.\n- **Updates.**\n  - Finding: adding documents recomputes communities and re-summarises. Maintainers say they do not plan to invest further in incremental handling of deletes or edits, and recommend removing the documents and re-indexing from cache. The `update` command had a reported KeyError.\n  - URL: https://github.com/microsoft/graphrag/issues/741. Read: SNIPPET.\n  - Implication: rebuild-from-source, as you plan, is what upstream recommends too.\n- **Rebuild cost claims.** LightRAG says GraphRAG rebuilds cost about 1,399 communities x 2 x 5k tokens. EraRAG says GraphRAG re-clusters fully on each update. Both are competitors' claims. URL: https://arxiv.org/html/2410.05779v2. Read: SNIPPET.\n\n**3. Passage-graph methods with little or no LLM extraction**\n\n- **HippoRAG 2.**\n  - Finding: avg F1 59.8 vs 57.0 for NV-Embed-v2, a dense retriever. 2Wiki +9.5, MuSiQue 48.6 vs 45.7, HotpotQA 75.5 vs 75.3, NQ 63.3 vs 61.9.\n  - Recall@5: 2Wiki 90.4 vs 76.5, MuSiQue 74.7 vs 69.7.\n  - Earlier structure-augmented methods all fell below the best dense retriever. RAPTOR was hurt by noise from LLM summaries.\n  - Ablations: removing passage nodes cost 6 recall points. Mixing in passage similarity at a 0.05 weight is best (0.5 costs about 2.6 points).\n  - It still uses an LLM for triple extraction (OpenIE).\n  - URL: https://arxiv.org/abs/2502.14802. Read: PARTIAL (through Table 2; ablations via excerpt).\n  - Implication: gains concentrate where entities bridge notes. Dense similarity has to stay in the seed mix.\n- **HippoRAG v1 ablation.**\n  - Finding: adding 1-hop neighbours without PPR lowered average R@2 to 42.2, vs 50.7 for query nodes alone. Dropping node specificity (an IDF-like weighting by how few passages mention a node) cost 2.7 R@2 points. Most errors came from extraction and linking.\n  - URL: https://arxiv.org/pdf/2405.14831. Read: SNIPPET.\n  - Implication: raw neighbour expansion hurts and degree weighting helps. Your seed is a known note, so the linking error disappears (my inference).\n- **LinearRAG (ICLR 2026).**\n  - Finding: a spaCy-only entity-sentence-passage graph, no relation extraction. On 2Wiki, indexing took 250s and 0 tokens vs 1,147s and 6.2M tokens for HippoRAG2. Accuracy: 66.95 vs 58.85.\n  - E2GraphRAG (spaCy entity-chunk index, no graph propagation) scored only 46.2.\n  - Self-reported, from the same group as GraphRAG-Bench.\n  - URL: https://arxiv.org/abs/2510.10114. Read: SNIPPET (table).\n  - Implication: a cheap graph plus PPR propagation is the working recipe. A cheap graph alone is not.\n- **SPRIG.**\n  - Finding: an NER co-occurrence graph with PPR is competitive with BM25+dense fusion (RRF). It beats RRF on 2Wiki and loses on HotpotQA. It is sensitive to seed quality and graph noise.\n  - URL: https://arxiv.org/abs/2602.23372. Read: SNIPPET.\n  - Implication: always benchmark against a hybrid keyword-plus-vector baseline.\n- **HopRAG** (LLM-proposed passage edges).\n  - Finding: only +3.1% over HippoRAG and +1.1% over SiReRAG. The big headline gain is against plain dense retrieval, and it costs LLM calls at index and query time.\n  - URL: https://aclanthology.org/2025.findings-acl.97/. Read: SNIPPET.\n  - Implication: model-proposed edges are the least proven edge type.\n\n**4. Traps**\n\n- **Hubs.**\n  - Finding: in a 9,382-node graph, 75% of cases where text beat graph by more than 0.4 (n=60) were hub dilution. \"Horse\" had 800+ edges. The fix was ln(degree+2) normalisation.\n  - CatRAG puts 45.7% of PPR mass on the top 1% of nodes by weighted degree, cut to only 42.5% by the mitigation.\n  - URLs: https://aclanthology.org/2026.kallm-1.16.pdf and https://arxiv.org/abs/2602.01965. Read: SNIPPET.\n  - Implication: your ledger rulings are hubs by construction. Downweight by degree.\n- **Noisy or missing edges.**\n  - Finding: spurious edges cause retrieval drift, and missing bridges cause hallucinated continuation (CS-RAG).\n  - With nodes deleted, graph-only LightRAG degrades sharply, and the hybrid falls back toward text-only (A2RAG).\n  - Without dense backfill, graph-only retrieval fell (HELP).\n  - URLs: https://arxiv.org/pdf/2603.14828, https://arxiv.org/pdf/2601.21162v1, https://www.arxiv.org/pdf/2602.20926. Read: SNIPPET.\n  - Implication: always blend with plain search so bad edges cost little.\n- **Hop depth.**\n  - Finding: HELP's F1 peaked at 3 hops (76.2) and fell at 4, while retrieval time grew exponentially (577s at 3). Same source as above.\n  - A direct measurement of \"two hops reaches everything\" was not found.\n  - Implication: cap expansion at 2 hops.\n- **Staleness.**\n  - Finding: no measured harm from stale edges. Update-cost evidence is authors' self-reports (EraRAG: https://arxiv.org/html/2506.20963v2, SNIPPET).\n  - Implication: derive-and-rebuild sidesteps this, except for edges that are not derivable (drawn by a person, found by a reader, proposed by a model).\n\n**5. Existing hyperlink or citation structure**\n\n- **Referral augmentation (ACL Findings 2024).**\n  - Finding: appending the text of linking or citing passages to a document at index time, with no model inference, improved ACL paper retrieval by more than 90% (relative), and arXiv/RefSeer by more than 10%. It did not help multi-hop QA, and it helped only BM25 on open-domain QA.\n  - URL: https://aclanthology.org/2024.findings-acl.798.pdf. Read: SNIPPET (long excerpt).\n  - Implication: backlink context is a cheap signal for queries near a note's meaning, not for associative ones.\n- **CG-RAG.**\n  - Finding: a trained citation-graph retriever scored Hit@1 0.961 vs 0.913 for ColBERT on PubMed (7,849 papers). The excerpt showed no ablation removing the links.\n  - URL: https://arxiv.org/abs/2501.15067. Read: SNIPPET.\n- **LARAG.**\n  - Finding: following author-made hyperlinks from retrieved chunks gave the best answer quality with fewer chunks. The test was tiny: 20 queries on one corpus.\n  - URL: https://arxiv.org/abs/2605.07517. Read: SNIPPET.\n- **BRIDGE.**\n  - Finding: a training-free reference graph with PPR matched or beat HippoRAG on two of three benchmarks (Recall@2 79.8, 41.5, 65.7).\n  - URL: https://exa.ai/library/publication/clk0jhngm1b. Read: abstract only. How it builds the graph is unclear.\n- **Multi-hop WebRAG** (Nov 2025). I read only its abstract, which reports \"measurable improvements\" with no numbers.\n- **Overall:** existing-link evidence exists but is thin. No head-to-head against extracted graphs was found.\n\n**Disagreements**\n\n- Microsoft's judge-based wins for LazyGraphRAG conflict with GraphRAG-Bench, where gold-answer accuracy is below RAG on fact retrieval, and with the debiasing paper.\n- GraphRAG-Bench quotes older Han et al. figures (NQ accuracy -13.4%, time-sensitive -16.6%). They do not match the March 2026 version I read, where temporal queries favour graphs by about 19-23 points. I did not read v1.\n- GraphRAG-Bench links denser graphs (HippoRAG2 average degree 8.75-13.3 vs 1.5-3.2) to better recall. The hub papers say hubs hurt. A possible reconciliation is that the degree tail matters more than the mean. That is my inference only.\n\n**Not established**\n\n- Any measurement of note-to-related-notes retrieval, where the query is a whole document and not a question.\n- Anything on person-drawn or reader-found edges as edge types.\n- Measured entity-duplication or hallucinated-relation rates for Microsoft GraphRAG. The \"GraphRAG Costs Explained\" page returned no body text.\n- A crossover corpus size below which the graph does not pay.\n- Quantified harm from stale edges.\n- Independent replication of LinearRAG, SPRIG or BRIDGE.\n- Whether an agent that can follow links itself makes the graph redundant.\n"}