Hybrid Search: Combining Semantic and Keyword Retrieval for Better RAG Results

Anyone who has spent time building RAG systems knows this pretty quickly: your model’s answers are only as good as what you feed it. You can have the smartest language model in the world, but if it’s working with the wrong context, you’re going to get answers that sound confident but miss the mark. That’s the whole game with retrieval. It’s the step where the system goes and finds the right documents or text chunks before the model even starts writing a response.

Most people building these systems start with one of two approaches, either keyword search or semantic (vector) search. Both have their moments where they shine, and both have moments where they just don’t cut it. That gap is basically why hybrid search RAG has become such a hot topic lately, especially for companies like Impressico Business Solutions that work on building serious enterprise search and AI tools. Let’s get into what hybrid search actually means, how it functions, and why combining the two methods often beats picking just one.

Keyword search, or lexical search if you want the technical term, is the old school way of finding stuff. It matches the actual words in your search query against the words sitting in a document. This has been around for a long time, way before anyone was talking about AI-powered search.

The algorithm doing the heavy lifting here is usually BM25, or sometimes its older relative TF-IDF. The basic idea is this: BM25 checks how often a word shows up in a document, and also how rare that word is across your whole collection of documents. So if a word is uncommon overall but shows up a bunch in one particular document, that document jumps up in the rankings.

How it actually works: First, the text gets broken down into individual words or tokens. Then each word gets checked against an index that was built from all your documents ahead of time. Whichever documents contain the exact query words end up ranked higher, based on how often those words appear and how rare they are.

Semantic search takes a completely different approach, and you’ll often hear it called vector search instead. Rather than looking for matching words, it tries to figure out what the query actually means. It does this by turning text into something called an embedding, which is basically a long string of numbers that represents the meaning of a sentence or paragraph.

Here’s the process: The query and the documents both get run through an embedding model. That model spits out vectors, which you can think of as points scattered across a huge multi-dimensional space. From there, the system looks for documents whose vectors sit closest to the query’s vector, usually measured using something called cosine similarity. Documents that mean similar things end up clustered close together in this space, even if they don’t share many actual words.

Now here’s where it gets tricky. Neither of these approaches works perfectly on its own, and honestly, both have some pretty obvious blind spots.

Figure 1 — Where each method succeeds, where it fails, and why hybrid exists.

Pure semantic search tends to miss exact matches. Got a specific product code, a part number, an acronym, or some numeric value buried in your documents? Vector search might just skip right past it, because embeddings are designed to capture meaning, not exact precision. Sometimes two totally different numbers end up looking “similar” in vector space just because they show up in similarly structured sentences.

Then flip it around. Pure keyword search, whether that’s BM25 or old-school TF-IDF, completely ignores meaning and paraphrasing. If someone phrases their question differently than how it’s written in your source documents, keyword search might come back with nothing useful at all, even though the answer is sitting right there in your data.

This is basically the whole reason people keep bringing up hybrid search vs vector search in AI and RAG circles.

So what is hybrid search in RAG, really? Put simply, it’s a retrieval approach that uses both lexical (keyword) and semantic (vector) signals at the same time, then combines the results from both into one better-ranked list.

Instead of forcing you to pick a side, hybrid search grabs the strengths from each method. The keyword piece catches your exact terms, codes, and names. The semantic piece catches meaning and context. Put them together and you end up with retrieval that’s both accurate and flexible, which is honestly what most real-world use cases actually need.

RAG systems get used everywhere these days, legal, healthcare, e-commerce, enterprise support, you name it. And if you look at the actual documents these industries deal with, they’re usually a mix of technical jargon, product codes, dates, and plain conversational language, often all crammed into the same knowledge base.

Think about a customer support article that explains a return policy in plain English but also lists an exact SKU number somewhere in there. Or a legal contract that mixes formal legal language with specific section numbers. A retrieval system that can only understand meaning, or only match exact words, is going to fail on part of that document no matter what.

This is exactly the kind of mixed content that hybrid search RAG handles so much better. And when retrieval improves, the context going into the language model improves too, which means the final answer is more accurate and grounded. Fewer hallucinations, better responses, happier users.

Once you’ve got two separate sets of results, one from keyword search and one from semantic search, you need some way to merge them into a single ranked list. There are two methods that come up most often.

A. Weighted Score Fusion

With this approach, each system assigns a score to every document it returns. Keyword search hands out a BM25 score, semantic search hands out a similarity score. Then you combine both scores using a weighted formula, something along the lines of:

The tricky part is that BM25 scores and cosine similarity scores don’t live on the same scale at all. So before you can combine them fairly, you usually have to normalize both sets of scores first, otherwise one method ends up dominating the results just because its numbers happen to run higher. Teams usually spend a fair amount of time testing different weight combinations to see what actually works for their specific data.

B. Reciprocal Rank Fusion (RRF)

This brings up a question that comes up constantly: what is reciprocal rank fusion (RRF)? Introduced by Cormack, Clarke and Buettcher at SIGIR in 2009, RRF takes a different tack. Instead of messing with raw scores from two different systems, it only looks at where each document landed in each list, meaning its rank or position.

The formula looks like this:

“Rank” here just means the document’s position in a given results list, first place, second place, and so on. “k” is a small constant, usually set around 60, which softens the impact of very high rankings. Each document picks up an RRF score from the keyword list and another from the semantic list, and you just add those two numbers together to get its final combined score.

The nice thing about RRF is that it completely skips the normalization headache from the weighted approach, since it never touches the actual BM25 or similarity scores in the first place. That’s probably why so many RAG systems these days lean toward RRF over weighted fusion, particularly teams who want solid results without spending weeks fine-tuning weights. The trade-off is worth knowing: OpenSearch’s own benchmarking across six datasets found RRF scored roughly 3.9 percent below score-based fusion on nDCG@10. RRF buys you convenience and robustness; a well-tuned weighted setup buys you a slightly higher ceiling, if you have the evaluation data to get there.

Figure 2 — The two fusion methods side by side, and what each one costs.

Here’s roughly what a hybrid search setup looks like inside a typical RAG pipeline:

Figure 3 — The seven steps described below, as a pipeline.

Document ingestion: Your documents get split into chunks and stored in two separate places at once, a traditional search index like Elasticsearch or OpenSearch for keyword search, and a vector database like Pinecone, Weaviate, or Qdrant for the embeddings.

Query time: When someone asks a question, that same query gets fired off to both systems simultaneously.

Keyword retrieval: The keyword index sends back a list of top matches based on BM25 scoring.

Semantic retrieval: The vector database sends back its own list based on embedding similarity.

Fusion step: Both lists get merged together, either through weighted score fusion or RRF.

Re-ranking (optional): Some setups throw in an extra re-ranking model at this point to sharpen the final ordering before it goes anywhere near the language model.

Generation: The top chunks get handed off to the LLM as context, and that’s what generates the actual answer the user sees.

This setup basically lets you get the benefits of both retrieval methods without having to sacrifice one for the other.

If you’re asking whether hybrid search is better than vector search alone, the honest answer is “it depends,” but for most real business use cases, yes, it usually is.

Hybrid search really earns its keep when your documents contain a mix of precise terminology and natural, conversational language, and let’s be real, that describes almost every business knowledge base out there. Picture internal documentation that combines product SKUs with plain-English explanations, or medical records that mix exact drug codes with descriptive notes about how a patient is feeling. In situations like these, vector search alone is going to miss the exact codes, and keyword search alone is going to miss the natural language questions people actually type. Hybrid search catches both sides.

Now, if your data is extremely uniform, say it’s all technical codes or all free-flowing prose, a single method might do the job just fine. But most business content is messy and mixed together, and that messiness is exactly the problem hybrid search was designed to solve.

Hybrid search brings a lot to the table, but it’s not without its downsides. Worth keeping these in mind:

All of this sounds good on paper, but how do you actually prove it works? That comes down to measuring both the quality of what gets retrieved and the quality of the final answer the whole RAG system produces.

Retrieval metrics tell you how good your retrieved documents are before they even reach the language model:

RAG-level metrics go further and look at the quality of the actual generated answer:

When teams actually run these evaluations, hybrid search setups tend to consistently outperform semantic-only or keyword-only search on recall and nDCG, especially on datasets with mixed content types. The published numbers bear that out. On the WANDS e-commerce benchmark, a tuned hybrid setup reached about 7.4 percent higher nDCG than either BM25 or pure vector search on its own. On financial documents mixing text and tables, a hybrid pipeline with reranking hit Recall@5 of 0.816 against 0.587 for dense-only retrieval. And that improvement at the retrieval stage usually carries over into better faithfulness and groundedness scores once you look at the full RAG output.

Retrieval is really the backbone of any solid RAG system, and the retrieval strategy you pick has a massive impact on the quality of answers your AI ends up giving. Pure keyword search and pure semantic search both bring real strengths to the table, but they also both come with real blind spots. Hybrid search, by pairing BM25 with vector search and using fusion techniques like reciprocal rank fusion, gives you a retrieval setup that handles both exact terminology and natural language questions without dropping the ball on either.

For businesses building serious AI-powered search or RAG applications, hybrid search isn’t just a nice-to-have anymore, it’s fast becoming the standard way to do things. At Impressico Business Solutions, we help companies design and build retrieval pipelines made for messy, real-world data, not just tidy test cases, so the AI systems you rely on give answers your team can actually trust.