Every RAG failure I’ve debugged that wasn’t a prompt problem turned out to be a chunking problem. Not the retrieval loop, not the model, not the planner. The chunking. Someone split documents on a fixed character count, a table got cut down the middle, a heading got orphaned from the paragraph it governed, and from that moment the answer simply did not exist in the store in retrievable form. No loop recovers that. You cannot retrieve what was destroyed at index time.
This is the least glamorous layer in the stack and it sets the ceiling on everything above it. Agentic RAG, multi-hop reasoning, self-correction graders — all of it operates on whatever your chunking produced. If the chunk is garbage, the cleverness above it is rearranging garbage.
What chunking actually destroys
The default approach in most tutorials is brutal: split text every N characters with some overlap. It’s fast, it’s simple, and it quietly mutilates structured documents. Here’s what fixed-size splitting does to real content.
# A pricing table that meant something whole:
| Tier | Seats | Price/mo |
| Starter | 5 | $49 |
| Growth | 25 | $199 |
| Enterprise | 100+ | custom |
# After a 200-char split lands mid-table:
CHUNK A: "...| Tier | Seats | Price/mo | Starter | 5 | $49 | Grow"
CHUNK B: "th | 25 | $199 | Enterprise | 100+ | custom |"Now ask “what does the Growth tier cost?” The number $199 is in chunk B but the word “Growth” is split across the boundary, the column header “Price/mo” is in chunk A, and the embedding of chunk B is a soup of numbers with no clear subject. Retrieval might surface chunk A, which has the question’s keyword but not the answer. The model gets context that looks relevant and contains nothing useful. It either says it doesn’t know or, worse, guesses.
The same destruction happens to anything where structure carries meaning: a clause whose conditions are in the preceding sentence, a list whose items are meaningless without the lead-in, a section whose heading is the only thing that disambiguates it from three similar sections elsewhere.
Chunk to the structure, not to a number
The fix is to stop treating documents as character streams and start respecting their structure. Split on semantic boundaries — sections, paragraphs, table rows kept whole, list items kept with their lead-in. A markdown document chunks on its headings. A contract chunks on its clauses. A table is one chunk or splits only on complete rows, never mid-row.
Fixed-size chunking optimizes for the indexer’s convenience. Structural chunking optimizes for the retriever’s success. Only one of those is your actual goal.
Two techniques earn their keep here. Contextual headers: prepend each chunk with the title and section path it came from, so a chunk about “$199” also carries “Pricing > Growth tier” and embeds with its meaning intact. Parent-document retrieval: index small, precise chunks for matching, but when one hits, return the larger parent section it belongs to, so the model sees the full context and not just the fragment. You get retrieval precision and generation context at once.
Size is a tradeoff, not a setting
There’s no universal best chunk size, and anyone who hands you one number is guessing. It’s a tradeoff with a real shape:
| Chunk size | Precision | Context per chunk | Failure mode |
|---|---|---|---|
| Small (100-200 tokens) | high — sharp matches | low — fragments | answer split across chunks |
| Medium (300-500) | balanced | decent | usually the right default |
| Large (800-1000+) | low — diluted embedding | high | retrieves but buries the answer in noise |
Small chunks match precisely but slice answers apart. Large chunks keep answers whole but their embeddings get diluted — a 1000-token chunk averages so many concepts that it matches everything weakly and nothing strongly. The sweet spot for most prose sits in the middle, but the only way to know yours is to test on real questions, not to copy a number off a blog.
Indexing decisions that matter as much
Chunking is half of it. How you index the chunks is the other half, and two choices dominate.
The embedding model. This is the lens through which every query meets every document. A weak or domain-mismatched embedding model puts the right chunk and the right query far apart in vector space, and no loop closes that gap. Federal contracting language, medical terminology, legal phrasing — general embeddings handle these unevenly. Test your embedding model on your actual domain’s questions before you commit, because swapping it later means re-indexing everything.
Hybrid search. Pure vector search misses exact-match needs: a part number, a statute citation, an error code, an acronym. A precise identifier like a NAICS code is an exact token that semantic search will happily blur into nearby codes. Keyword (BM25) search nails it. Run both and combine the scores. The questions that pure vector search quietly fails on — exact identifiers — are exactly the ones keyword search was built for, and the fusion of the two covers far more ground than either alone.
The discipline that catches it
The reason chunking problems are so insidious is that they’re invisible from the top of the stack. The system returns something, the answer is fluent, and unless you look at what was actually retrieved you never see that the right context never made it in. So the discipline is simple and non-negotiable: read your retrieved chunks by hand on a sample of real questions, before you blame the model. For each question, ask one thing — was the fact needed to answer this actually present, intact, in the retrieved chunks?
When the answer is no, you’ve found your problem, and it’s almost never upstream of indexing. I’ve watched teams add planners, graders, multi-hop loops, and bigger models to fix systems whose real disease was a table sliced in half three weeks earlier. The loop kept looping over mangled chunks. Every cent of agentic cleverness was spent compensating for a 30-minute fix in the chunker.
Agentic RAG raises the ceiling on what retrieval can do. Chunking and indexing set where that ceiling is. When I build retrieval into the systems I ship — the AI-native operating system I’m building, Vontra, has to answer real questions over a company’s own documents — the boring index layer is where I spend the first week, every time. You can build the smartest loop in the world, and it will still only ever find what survived the moment you split your documents and embedded them. Spend the time there, because that’s the layer that decides whether everything you build on top has anything real to stand on.