You're Doing RAG Retrieval Wrong

The promise of RAG is simple: give a language model access to your data, and it can answer questions about it. No fine-tuning, no retraining. Just retrieve relevant chunks and let the model synthesize.

In practice, the retrieval step is where most RAG pipelines break. And it breaks silently. The model doesn't say "I got the wrong context." It reads whatever you gave it and generates a confident, well-structured, completely wrong answer.

The root cause is what I call semantic dissonance. You're comparing a question to a document chunk using vector similarity. But questions and document chunks are fundamentally different kinds of text. A question is short, interrogative, specific. A document chunk is declarative, dense, contextual. They don't land in the same part of the embedding space — even when one is the answer to the other.

This is like going to a library and instead of looking in the card catalog (which lists questions like "Where can I learn about X?"), you're trying to match your question directly against random paragraphs from random books. Sometimes you get lucky. Often you don't.

To understand why retrieval fails, you need to understand where things live in embedding space.

An embedding model converts text into a vector — a point in high-dimensional space. The idea is that semantically similar texts end up near each other. "How do I reset my password?" should be close to "What's the password reset process?" That works. Both are questions about the same thing.

But here's the problem: "How do I reset my password?" is not necessarily close to "To reset your password, navigate to Settings > Security > Reset Password and click the reset link." One is a question. The other is an answer. They're about the same topic, but they're different kinds of text. And embedding models often encode the kind of text as strongly as the topic.

This is the core insight. In embedding space, questions cluster with questions and documents cluster with documents. Your query — which is a question — will naturally be closer to other questions than to the document chunk that actually contains the answer.

The fix is elegant. Instead of searching your query against document chunks, search it against questions about document chunks.

Here's how: for each chunk in your corpus, use an LLM to generate 3-5 questions that the chunk answers. Index those questions. When a user query comes in, match it against the generated questions — not the raw documents. Then retrieve the document chunk associated with the best-matching question.

This works because you're now comparing apples to apples. A user question matches against synthetic questions. Both are interrogative, similar in structure, similar in length. The embedding model can actually do its job.

This technique is sometimes called HyDE (Hypothetical Document Embeddings) or query-to-question matching, depending on the variant.[1]

The difference is often dramatic. In benchmarks, question-to-question matching improves retrieval precision by 20-40% over naive chunk matching. And better retrieval means better answers downstream — the LLM can only work with what you give it.

Even after fixing what you compare, how you compare still matters. Generic embedding models — the ones you get out of the box — are trained on general web text. They know that "bank" relates to "money" and "river." They don't know that in your fintech codebase, "default" means "loan default," not "default parameter value."

This is the domain embedding problem. In a specialized corpus, words and phrases carry domain-specific meanings. A generic embedding model conflates them.

The solution: fine-tune your embedding model on your domain data, or use a model that's already been trained on similar text. This doesn't require massive datasets — even a few thousand domain-specific sentence pairs can significantly improve retrieval quality.

You've fixed what you compare and how you embed it. But you're still ranking by cosine similarity alone. And cosine similarity, while useful, only measures one thing: the angle between two vectors. It doesn't know about freshness, specificity, document quality, or user preference.

Two chunks can have nearly identical cosine similarity to your query. One is a three-sentence summary from an outdated FAQ. The other is a detailed, current technical explanation. Cosine similarity can't tell the difference.

This is where reranking comes in. After your initial vector search returns the top-N candidates, a second pass scores them on additional dimensions:

- Cosine similarity (semantic match)
- Recency (newer is often better)
- Specificity (longer, more detailed chunks score higher)
- Popularity (documents that users have found helpful before)

Cross-encoder rerankers — models specifically trained to score query-document pairs — are another powerful option. They're slower than cosine similarity but dramatically more accurate, because they see the query and document together rather than comparing pre-computed vectors.

Here's a question that most RAG tutorials skip: what cosine similarity score is "good enough"?

The honest answer is: it depends entirely on your data. A score of 0.82 might be excellent for one corpus and mediocre for another. The distribution of scores matters far more than any individual score.

If you set the threshold too high, you miss relevant results (low recall). Too low, and you flood the LLM with junk (low precision). The right threshold lives at the intersection of these tradeoffs — and you can only find it by looking at your actual score distribution.

In practice, the best approach is to evaluate a sample of queries with known-good answers, plot the score distribution, and pick the threshold that maximizes F1 — or whatever metric matters most for your use case. There is no universal magic number.

The complete pipeline looks like this:

1. Index time: For each document chunk, generate synthetic questions using an LLM. Embed both the questions and the chunks using a domain-specific embedding model.

2. Query time: Embed the user's query. Search against the synthetic questions (not the raw chunks). Retrieve the top-N associated chunks.

3. Rerank: Score the candidates using a combination of cosine similarity, recency, specificity, and optionally a cross-encoder. Reorder by combined score.

4. Threshold: Filter out any results below your empirically-determined quality threshold.

5. Generate: Pass the surviving chunks to the LLM as context.

Each step removes a different class of errors. Question matching fixes semantic dissonance. Domain embeddings fix vocabulary confusion. Reranking fixes the "close but useless" problem. Thresholding prevents garbage from reaching the model.

None of these steps is novel in isolation. But the combination — and understanding why each matters — is what separates RAG pipelines that work from RAG pipelines that confidently produce wrong answers.

◆