Sparse vs. Dense vs. Hybrid Retrieval | Vedanshu

Sparse vs. dense retrieval: Literal words vs. meaning

[0:00]Konnichiwa, I'm Vedanshu Daksesh Patel. Chapter 5 made search fast. Chapter 6 asks whether comparing embeddings is even the right way to match in the first place because there are two genuinely different ways to decide whether a chunk answers a question and each one fails exactly where the other works. Here is the problem and it is the chapter's own example. Two employees the same tuition

Exact code lookups (TR114) with BM25 keyword search

[0:20]Reimbursement policy, two completely different questions. The first quotes an exact code TR114. The second never says tuition or reimbursement or policy at all. Just, does the company help pay for grad school? Same answer, opposite questions. The ask sparse retrieval from scratch. BM25 term frequency, inverse document frequency, length normalization and saturation. So, a word repeated 50 times doesn't count 50 times. One detail

Paraphrase gaps: Why BM25 misses semantic questions

[0:46]Matters more than it looks. Tokenize so TR114 survives as a single term. Split it into TR and 114 and the exact match signal is gone before we start. Read the scoring function. Every query term contributes its inverse document frequency scaled by how often it appears in this passage and that contribution saturates so repetition stops paying. The normalizer divides by document length so a long passage can't win just

Reciprocal Rank Fusion (RRF) and how hybrid search can get outvoted

[1:10]By being long and TR114 appears in exactly one of 12 passages which gives it the highest IDF in the collection. Run it. The code query TR114 wins at 3.49 ahead of two other policies that also contain the word reimbursement. Sparse nails it. Now the paraphrase, help pay for grad school returns the certification exam policy at 3.82 because that passage literally contains the words company pays. The right answer

Prompting Claude to stress-test your retrieval fusion arithmetic

[1:35]Is third. That is the paraphrase gap measured. The revision, add dense retrieval real mini LM embeddings matching on meaning instead of words. Then fuse the two rankings with reciprocal rank fusion. Each candidate scores one over 60 plus its rank in every list it appears in summed. Ranks not scores because BM25 scores and cosine similarities are not on a comparable scale and report all three methods. Two pieces dense embed the query dotted against every passage embedding the vectors are normalized so that dot product is cosine similarity. Then fusion for every rank list a candidate appears in add one over 60 plus its rank. No weights, no tuning. That is the whole of reciprocal rank fusion and it is four lines long. And here is the result I did not expect. Dense got both. It found TR114 for

The code query two at 0.6. The entity-centric weakness the chapter warns about did not show up at 12 passages. In the hybrid, the combination that is supposed to be the safe default, missed the paraphrase. Only plain dense went two for two. Here is why. On the paraphrase, the certification policy was sparse as number one and dense is number two. The right answer was dense as number one but sparse had it third. 1/61 + 1/62 beats 1/63 + 1/61 by a quarter of 1,000. Fusion did not reason about either list. It counted ranks and a confidently wrong ranking outvoted a correct one. So, sparse matches words and misses paraphrases. Dense matches meaning and here missed nothing. Hybrid usually helps and rarely hurts but usually is doing real work in that sentence and this run is one of the

Times it hurt. Two honest limits. 12 passages is too small to reproduce dense retrieval's known weakness on rare entities and one fusion loss is not a verdict on fusion. The chapter says the same thing. The honest summary is not the hybrid always wins. Let your turn paste this. Take my own document set and build all three retrievers, sparse, dense, and a fused hybrid. Then find me the queries where the hybrid is worse than its best single retriever and show me the rank arithmetic that made it lose. That second sentence is the one that earns its keep. Anything will report an average that makes fusion look good. Asking for the cases where it loses plus the arithmetic is what turns a default into a decision you can actually defend. Watch the right answer get outvoted. Vedant Chudaxsh Patel.