Chunking Strategies for RAG: What Actually Works in Production

Here’s a scene that plays out in almost every RAG project we’ve been part of. The demo looks fantastic. Everyone claps. Then it goes live, real users start typing real questions, and within a couple of weeks someone on the support team forwards a message that says “the bot gave me half an answer” or “it mixed up two different policies” or, worse, “it made something up, and it sounded so confident about it.”

The first instinct is almost always to blame the model. Swap in a bigger LLM, rewrite the prompt, add more instructions telling it to “only answer from the provided context.” Sometimes that helps a little. But in a lot of the projects we’ve worked on at Impressico Business Solutions, the real issue was sitting much earlier in the pipeline, at the point where documents got cut into pieces before they were ever stored or searched.

That step is called chunking, and honestly, it doesn’t get nearly enough attention. Most tutorials treat it as a two-line detail: “split your text into chunks of 500 tokens.” Nobody tells you that this one decision quietly shapes everything else your RAG system does.

Let’s back up for a second, in case chunking is new territory for you.

A RAG system can’t search an entire contract or a two hundred page manual every single time someone asks a question. So the document gets broken into smaller pieces, called chunks. Each chunk becomes a vector (an embedding) and lives in a database built for similarity search. When someone asks a question, the system looks for chunks that are closest in meaning to that question, then hands those chunks to the LLM to write the actual answer.

Once you see it laid out like that, it becomes obvious why chunking matters so much. It’s touching four things at once.

Retrieval accuracy takes the first hit. A chunk that’s too small might not carry enough context to match a real question. A chunk that’s too big can bury the right sentence under a pile of unrelated text, which weakens the match and pulls in noise.

Context quality comes next. Even when the system finds the right chunk, if it cuts off mid-sentence, or splits a table in half, the LLM is working with something broken. And a language model, being a language model, will usually try to smooth over that gap rather than admit it’s missing something.

Which brings us to hallucination risk. This is the part people underestimate the most. When the retrieved context is incomplete, the model doesn’t usually say “I don’t have enough information.” It fills the hole with something plausible sounding. That’s a hallucination, and a lot of the time, poor chunking is the actual root cause, not some flaw in the model itself.

And all three of those things roll up into overall answer quality. In our own client work, chunking has consistently been a bigger lever on answer quality than model choice. The model is very often not the bottleneck people assume it is.

There’s no shortage of chunking methods floating around online, and every vendor seems to claim theirs is the best. Here’s our honest read on the main ones, including where each tends to break down once it hits production traffic.

Figure 1 — The five strategies and what each one trades away.

Fixed-size chunking is the one everyone starts with. Pick a number, say 500 tokens, and cut the document into equal blocks without looking at sentences, paragraphs, or anything else. It’s fast and dead simple to implement, which is exactly why it’s the default in most tutorials. The catch is that it slices right through sentences, tables, and half-finished thoughts without any regard for what it’s cutting. Some benchmarks have found this approach can drop retrieval recall by 20 to 74 percent compared to smarter methods, depending on what kind of document you’re feeding it. Fine for a proof of concept. Not something we’d put in front of real customers.

Recursive chunking takes a smarter approach. Instead of cutting blindly, it tries a series of separators in order, first paragraph breaks, then line breaks, then sentence breaks, only going further down the list when a chunk is still too large. This has quietly become a favorite in production because it respects natural document structure most of the time without the overhead of anything fancier. A 2026 benchmark that ran seven different chunking strategies across fifty academic papers actually found recursive chunking with 512-token chunks scored highest for end-to-end accuracy, beating out several more complex methods. That result surprises people the first time they hear it. But it makes sense once you think about it. Recursive chunking rarely produces a badly broken chunk, and in practice, avoiding bad chunks matters more than chasing perfect ones.

Semantic chunking is the one that gets the most hype. It uses embeddings to detect where the topic actually shifts and splits there, rather than at some arbitrary length. In theory, every chunk ends up being a clean, self-contained idea. In practice, it’s a mixed bag. Some studies show it crushing fixed-size chunking, especially in dense, topic-heavy fields. One peer-reviewed study on clinical decision support found accuracy jumping from 13 percent with fixed-size chunking to 87 percent once chunks were aligned to actual topic boundaries. That’s not a small difference. But other benchmarks found semantic chunking producing chunks so tiny (some averaging under fifty tokens) that quality actually got worse, not better. It’s also noticeably slower to run, since every sentence needs its own embedding just to detect where a topic change happens. One benchmark clocked it at roughly fourteen times slower than basic token-based splitting. So semantic chunking is powerful, but it’s not something you flip on and forget about. It needs tuning, and it costs more.

Sentence-based chunking splits text sentence by sentence and groups sentences together until it hits a size limit. It respects grammar, so you almost never end up with a chunk that trails off mid-thought. What’s interesting is that some 2025 research found sentence-based chunking matching semantic chunking in quality for documents up to around five thousand tokens, at a fraction of the compute cost. If your documents aren’t packed with rapidly shifting topics, this is a genuinely solid, low-effort middle ground.

Structure-aware chunking respects the actual shape of the document, its headings, sections, sub-sections, bullet lists, tables, instead of treating the whole file as one endless stream of text. This is the one we reach for most often when working with enterprise document sets, and there’s a simple reason why. Business documents aren’t random blocks of prose. A contract is built from clauses. A technical manual is organized into sections and sub-sections. An FAQ page is already a series of question-and-answer pairs. When chunking follows these natural boundaries, each chunk tends to represent one complete idea, which is exactly what both the search step and the LLM actually need. It also makes it far easier to keep tables and lists intact as single units, rather than shredding them across two or three chunks, which is one of the most common and most painful failure modes we’ve seen in the wild.

People ask this question constantly, usually hoping for one clean number. The honest answer is that it depends, but we can at least give you real starting points instead of a shrug.

Guidance from Microsoft Azure, NVIDIA, and Arize AI points to a similar starting zone, though they don’t all land on one number. Azure recommends starting at 512 tokens with 25 percent overlap, counted in actual model tokens rather than characters. Arize found chunk sizes of roughly 300 to 500 gave the best speed-quality tradeoff. NVIDIA’s multi-dataset evaluation found 256 to 512 tokens worked well for fact-based questions, while analytical questions did better at 1,024 tokens or page-level chunks. Go below roughly 128 tokens and chunks start feeling too thin to carry real meaning. Push well past 1,000 tokens and the embedding tends to represent too many ideas at once, which can weaken how well it matches a specific question — though NVIDIA’s results show that is not universal.

Treat that as a starting point, not a rule carved in stone. The actual right size for your system depends on a handful of things.

One size never really fits all here. Below is how we tend to approach the document types we see most often in enterprise projects.

A chunk of text sitting by itself, disconnected from its source, isn’t worth much once you’ve pulled it out of the original document. This is why metadata matters just as much as the chunking method itself, even though it’s usually the thing teams bolt on last, if at all.

For every chunk, it’s worth keeping track of the document ID, so you always know which source it came from. The section or heading it belongs to, so the system understands where in the document this piece sits. The page number, which is genuinely useful for citations and for letting users double check an answer themselves. A timestamp, so you can tell whether the underlying information might already be stale. The source system it came from, especially relevant when you’re pulling content from multiple repositories at once. And access-control information, so sensitive documents only ever get retrieved for users who are actually allowed to see them.

One pattern that comes up a lot in mature RAG systems is parent-child chunking, sometimes called hierarchical retrieval.

The idea is fairly simple once you see it. You create small, tightly scoped chunks for search purposes, because small chunks tend to match specific questions much more precisely. But instead of only sending that small chunk to the LLM, you also pull in its “parent,” a larger chunk, or the full section it originally belonged to, so the model actually has enough surrounding context to give a complete, well-grounded answer.

Figure 2 — How parent-child retrieval resolves the precision-versus-context tension.

This solves a real tension that shows up constantly in RAG design. Small chunks are great for finding the right needle in the haystack, but they’re often too thin on their own for the LLM to build a solid answer from. Large chunks give the model plenty of room to work with, but they’re much harder to match precisely against a narrow question. Parent-child chunking lets you have both, precise matching when searching, full context when generating.

This pattern is especially useful for contracts, technical manuals, and long reports, where a single clause or paragraph often only makes complete sense once you read it alongside the section it lives in.

Too many teams pick a chunking method once, based on a blog post or a library’s default setting, and never look at it again. That’s a mistake, and it’s an easy one to avoid.

Chunking deserves the same kind of evaluation you’d give a model, tested against real queries, not made-up examples. Ideally you want a test set of fifty to a hundred real questions pulled from actual user behavior.

From there, a few metrics matter most. Retrieval recall tells you, out of all the chunks that actually contain the right answer, how many your system managed to pull back. Context relevance tells you how much of what was retrieved was genuinely useful, versus how much was noise the model had to wade through. Answer accuracy is the most obvious one, is the final answer actually correct. And faithfulness checks whether the answer sticks strictly to what’s in the retrieved context, or whether it wanders off into things the context never actually said. That last one is your clearest window into hallucination risk.

Chunking decisions don’t happen in a vacuum. Every choice you make trades off against something else somewhere in the system, and it’s worth being upfront about that instead of finding out the hard way.

None of this means better chunking isn’t worth it. It almost always is. It just means these are real costs worth planning for ahead of time, so nothing feels like a surprise once you move from a pilot into full production.

Given everything above, here’s roughly how we’d guide someone through choosing a strategy.

Figure 3 — Which strategy to start with, by situation.

If you’re just prototyping or testing whether RAG is even the right fit for your problem, start with fixed-size or basic recursive chunking. It’s quick to set up and good enough to validate the idea before you invest more.

If your documents are general business content with a clear structure, think manuals, policies, standard reports, structure-aware or recursive chunking will handle most of what you throw at it without a heavy engineering lift.

If your content is highly technical or topic-dense, things like clinical notes or deep research material, semantic chunking is probably worth the extra cost, but budget real time to tune it properly, since a poorly tuned setup can end up worse than something simpler.

If your users tend to ask broad questions that need the full picture, summaries, multi-step reasoning, cross-references, lean on parent-child or hierarchical chunking so the system can still search precisely while giving the model enough room to actually answer well.

If your documents are full of tables or structured data, financial reports, spreadsheets buried inside PDFs, pull tables out as their own units before running any text-based chunking on the rest of the document.

And if access control matters at all, internal knowledge bases with anything confidential, build metadata and access tags into your chunking pipeline from day one. Adding it later, after the system is already live, is a much harder conversation to have.

Chunking isn’t a task you finish once and move on from. It’s something that needs revisiting as your documents change, your users grow, and the kinds of questions people ask start to shift over time.

At Impressico Business Solutions, we spend a lot of our time helping enterprises build RAG pipelines around their actual documents and their actual users, not generic defaults copied from a tutorial. That means structure-aware chunking tuned to the specific document types a client actually has, metadata pipelines that bake in access control from the very start, and evaluation setups built around real queries, so we’re measuring what’s actually happening rather than guessing.

If your RAG system has been giving inconsistent or half-right answers, it’s worth checking chunking before reaching for a bigger model. More often than not, that’s where the real fix is hiding.