Almost every enterprise RAG project hits 80% accuracy in the demo – and then refuses to climb any further. The missing 20% is not in the model; it is in the retrieval, the chunking and the evaluation harness nobody built. A bigger model will not save you.

Retrieval-augmented generation is the most widely deployed and least understood pattern in enterprise AI. The demo is deceptively simple: embed a few documents, drop the best hits into a prompt, and the answers look great. Then the system goes in front of real users with real questions, accuracy stalls at around 80%, and the team’s reflex is to reach for a more capable model. That almost never works, because the errors are retrieval errors – and even a cleverer model cannot reason over context it never received.

We have taken RAG systems from “impressive pilot” to “trusted part of a regulated workflow” several times over, and the work that closes that gap is deliberately unspectacular.

The real problem: retrieve rubbish, deliver confident nonsense

Take a wrong RAG answer apart and you will find one of three causes, in this order of frequency:

1. The right chunk was never retrieved. It exists in the corpus, but the retriever ranked it below the cut-off. The model then answers from irrelevant context – fluently, which is worse than an obvious failure.
2. The right chunk was retrieved, but mangled. A naive 512-token split cut a table in half, separated a clause from its condition or severed a heading from its content. The information is technically present and semantically destroyed.
3. The model ignored good context. The rarest case – and the only one a better model meaningfully fixes.

Teams spend their budget on case three, and their errors are in cases one and two. If you measure nothing else, measure retrieval recall – the share of questions where the necessary information actually landed in the context window. We regularly see pilots with 70% recall whose teams are wondering why end-to-end accuracy is stuck.

The architecture that gets you past 80%

Production RAG is a pipeline of five stages, each of which caps your ceiling on its own. A better model improves one stage; the gains are in the other four.

1. Parsing and chunking that respects structure

Stop splitting by token count. Split along document structure – sections, clauses, table boundaries – so that every chunk is a self-contained thought. Prefix every chunk with the heading hierarchy, so that “the rate is 1.5%” also carries “Section 4.2 – Interest on late payment” with it. For documents with tables, extract the tables separately and render them as Markdown; a chunk that separates a table row from its header is worthless. This single change has improved end-to-end accuracy by 8–12 points for us – more than any model swap.

2. Hybrid retrieval instead of vectors alone

Dense embeddings are excellent at semantics and weak at exact matches – product codes, account numbers, defined legal terms, anything where the literal token matters. Pure vector search misses “clause 7(b)”, because that is semantically meaningless. Run keyword retrieval (BM25) and vector retrieval in parallel and fuse the results. For enterprise corpora full of identifiers, hybrid retrieval is not optional.

3. Reranking before the context window

Retrieve broadly – the top 30 or 50 candidates – and then have a cross-encoder reranker score each candidate against the actual query; keep the best 5. Retrieval optimises for recall, reranking for precision; you need both. A reranker is cheap compared with generation and the most effective component that most pilots are missing entirely.

4. Grounding discipline in the prompt

Instruct the model to answer only from the supplied context, to cite the chunk it used and to say “not found in the supplied documents” when retrieval comes back empty. A RAG system that refuses to guess is the one a compliance team will sign off. No source, no answer.

5. An evaluation harness from day one

A pipeline you cannot measure is a pipeline you cannot improve. Build a set of 100–300 real question-answer pairs from actual user queries and score every change separately for retrieval recall and answer correctness, so you know which stage moved. Without that you are tuning on anecdotes, and the next model update will set you back without anyone noticing.

A worked example: a policy assistant for a telecoms company

A telecoms company wanted an internal assistant for its support staff covering roughly 12,000 pages of product, pricing and process documentation. The pilot, built the usual way (fixed 512-token chunks, pure vector search, a GPT-class model), reached 79% answer accuracy on a set of 200 questions. The team’s plan: wait for a better model.

Instead, we rebuilt the unspectacular layers and left the model untouched:

• Structure-aware chunking with heading prefixes and separate table extraction: recall 71% → 86%.
• Hybrid retrieval from BM25 and vectors (decisive for tariff codes): recall 86% → 92%.
• Cross-encoder reranking, retrieve 40 → keep 6: answer accuracy 84% → 91%.
• Grounding prompt with mandatory source references and an explicit “not found” path: hallucinated answers to out-of-scope questions fell from 18% to under 3%.

End-to-end answer accuracy: 79% → 93%, same model, in about five weeks. The later model upgrade added two more points – useful, but a rounding error next to the pipeline work.

The staff trust metric mattered more than the accuracy figure: once the system reliably said “not found” instead of inventing a tariff, usage went from sporadic to standard. A system that is wrong 7% of the time but handles its uncertainty honestly beats one that is wrong 7% of the time and sounds confident every time.

The metric that actually predicts trust: honest refusal

Accuracy gets the attention, but the metric that decides whether a RAG system is used is its refusal behaviour – what it does when it does not know something. A system that answers every question, including those outside its corpus, trains its users to distrust it, because they cannot tell a grounded answer from a confident guess. A system that says “that is not covered in my documents” for out-of-scope questions earns trust for the answers it does give, because users learn that when it answers, it means it. We measure the hallucination rate on a deliberately out-of-scope question set as a metric in its own right, separate from accuracy on in-scope questions – because the two error types have completely different remedies and completely different effects on adoption.

Getting refusal right is largely an engineering decision, not a prompting question. The retriever should return relevance scores, and below a calibrated threshold the system should decline rather than pass weak context to the model and hope for the best. This is the same confidence-threshold logic that governs escalation to humans: when retrieval confidence is low, the safe action is to admit the gap instead of generating. Teams that skip this end up with the worst combination – a system that is occasionally and confidently wrong on exactly the questions it should have declined. That is the fastest way to lose a user base for good.

One subtlety in the measurement deserves naming. If you only test with questions your corpus can answer, you will never see the refusal error type – and you will ship a system that invents answers to out-of-scope questions the moment real users ask them. And they will, constantly. A serious RAG eval set therefore contains a deliberate share of unanswerable and adversarial questions, scored on whether the system correctly declined. Recall and accuracy tell you how good the answers are; the refusal rate on out-of-scope input tells you whether you can trust the system at all. Both belong in the harness from day one.

Pitfalls

• Chasing the model first. If recall is below 90%, a better model is the wrong investment. Fix the retrieval.
• Not separating retrieval and generation metrics. A single end-to-end accuracy number hides where the error is and turns every improvement into guesswork.
• Embedding the whole document. Granularity matters; a single vector for a 40-page PDF finds everything and nothing.
• Ignoring access rights in retrieval. If the index does not respect document permissions, your assistant will happily display content the user should never see. Retrieval has to be permission-aware.
• Assuming a static corpus. Documents change. Without a reindexing pipeline, your assistant will be confidently quoting last year’s prices within a quarter.

Decision checklist before you scale a RAG pilot

1. What is our retrieval recall on a real question set – separate from answer accuracy?
2. Does the chunking preserve structure, tables and heading context?
3. Are we using hybrid retrieval, or do exact identifiers fail unnoticed?
4. Is there a reranker between retrieval and the context window?
5. Does the system cite sources and refuse to answer when retrieval is empty?
6. Does retrieval respect the same permissions as the source documents?
7. Do we have an evaluation harness that will catch regressions at the next model update?

The gridz view

RAG is 80% data plumbing and 20% model – and the industry distributes its attention in exactly the wrong proportion. The teams whose assistants are trusted and used are the ones that invested in parsing, hybrid retrieval, reranking and evaluation before they touched the model. If your RAG pilot is stuck in the high seventies, the bottleneck is almost certainly retrieval recall – and that can be fixed in weeks, not quarters. We build these systems for regulated environments where a confidently wrong answer is a liability: grounded, with source references, permission-aware and measured. Talk to us if your pilot needs to make the jump into production.