RAG without the hype: answering from your own documents without retraining anything

RAG is the short answer to a question we get almost every week: "we want the AI to know our documents — do we have to train it on them?" No. In 95 % of the business cases we see, the solution isn't training anything, it's retrieving the right passage and handing it to the model at answer time. That's RAG — retrieval-augmented generation — and whether you understand the distinction is worth months of budget.

This guide explains what RAG is without conference-stage metaphors: what each part does, when it beats the alternatives, how to measure whether the answer is any good, what it costs to keep running, and why most RAG projects that never reach production fail somewhere that has nothing to do with artificial intelligence.

What is RAG and how does it differ from training a model?

A language model knows what it learned when it was trained. It doesn't know your collective agreement, your 2026 price list or the penalty clause you signed on Tuesday. There are three ways to get that knowledge into it, and only one is reasonable for most companies:

• Retraining or fine-tuning. You take the model and adjust its weights with your data. Expensive, slow and — this is the common misunderstanding — not a way to memorise facts. It's a way to change style, output format or behaviour. Fine-tune on your 4,000 contracts, change one clause, and you're fine-tuning again.
• Pasting the whole document into the prompt. Works with one document. With 4,000 it's unworkable on cost, and quality drops when the model receives a hundred pages where the relevant fact is three lines.
• RAG. For each question, a search engine picks the handful of passages that contain the answer, and the model writes using only those, citing where every statement came from.

The practical consequence of RAG is that knowledge lives outside the model. Change a PDF, reindex, and the answer changes that same afternoon without touching anything else. It also means something less comfortable: if retrieval misses the right paragraph, no model will rescue the answer. The bottleneck is almost never the generator.

RAG doesn't make the AI "know" your company. It makes the AI read the right thing just before answering. All the work is in "the right thing".

When does RAG fit and when is it the wrong tool?

RAG fits when the answer is written down somewhere, changes over time, and has to be justifiable. It doesn't fit when the data lives in tables, or when what you want is for the model to write differently.

| Need | What fits | Why | |---|---|---| | Answering about your own policies, contracts or manuals | RAG | The facts change and the source must be cited | | Making the model write in the house tone and format | Fine-tuning or instructions | That's behaviour, not knowledge | | "How many orders are due this month?" | Database query / executive dashboard | That's aggregation, not documents | | One long document per conversation | Send it whole in the prompt | Building an index for a single PDF is over-engineering | | Extracting fields from invoices or delivery notes | Document data extraction | That's structured reading, not search | | A system that also acts (open a ticket, query the ERP) | An agent with RAG as one of its tools | RAG answers; it doesn't do |

Two warnings from the field. First: plenty of people ask for RAG when what they need is document housekeeping. If three live versions of the same policy exist and nobody knows which one rules, RAG will confidently hand you one of the three. Second: RAG and agents aren't competitors. In the projects that reach production, retrieval is one more tool the agent calls when it needs to check the documentation — not the centre of the architecture.

What parts does a production RAG actually have?

Six stages, each with its own way of ruining the final answer. The end user only sees the combined result, so it helps to know where to look when something goes wrong.

| Stage | What it does | Where quality is lost | |---|---|---| | Ingestion | Collects files from the DMS, shared folders or email | Drafts, copies and superseded versions get in | | Extraction | Turns PDF, Word or scans into plain text | Tables fall apart, columns mix, unchecked OCR | | Chunking | Splits documents into indexable passages | Cuts mid-clause that separate a condition from its exception | | Indexing | Builds the semantic and the exact-match index | Semantic only: fails on codes, references and proper nouns | | Retrieval | Picks candidate passages for each question | Five near-identical passages from the same irrelevant document | | Generation | Writes the answer citing retrieved sources | Fills gaps with plausible language when retrieval came up short |

Three concrete decisions move the needle more than picking a model:

• Chunk along the document's structure. Splitting every 500 characters works on a wiki and destroys a contract. If the document has titles, articles and sections, carry that context into every passage: the answer goes from "the term is 30 days" to "article 7.2 of the framework contract sets 30 days, except under 7.3".
• Hybrid search, not vectors alone. Semantic search finds "cure period" when you ask about "warranty term". Exact-match search is the only thing that finds file reference `EXP-2026-0431`. You need both, plus a reranking step that chooses between what each returns.
• Metadata and filters before searching. Effective date, version, legal entity, department and status (current/superseded). Without them the system competes against itself across five versions of the same document, and the winner is whichever matches the wording most literally — almost never the current one.

And one security rule that isn't optional: permissions are enforced at retrieval, not at answer time. Each user may only retrieve passages from documents they already had access to. Filtering after generation is too late: the text has already been built from information that person shouldn't have seen.

How do you evaluate whether a RAG answer is good?

This is where pilots that reach production part ways with committee demos. Evaluating isn't asking three questions and concluding "it answers pretty well". It's building a reference question set with known answers and running the whole thing every time something changes.

How you build it, with realistic effort: gather 50 to 150 real questions — they come from internal support tickets and from what new hires ask, not from a meeting room — record for each one the correct answer and the exact document and section where it lives, and deliberately include 10-15 trap questions: things that aren't in any document, and questions whose answer changed last year.

| Metric | What it measures | Rough threshold | |---|---|---| | Retrieval hit rate | % of questions whose correct passage is among those retrieved | > 90 % before looking at anything else | | Faithfulness | % of statements backed by the cited passage | > 95 % | | Correct answer | % of answers an expert signs off | > 85 % before opening to users | | Correct abstention | % of unanswerable questions where the system says "I don't know" | > 90 % | | Freshness | % of answers citing the current version | 100 % for policy documents |

The order matters. If retrieval hit rate sits at 60 %, swapping the generating model is a waste of time and money. And correct abstention is the most ignored metric: a system that admits it doesn't know 10 % of the time is usable; one that never admits it is dangerous, because users have no way to tell a good answer from an invented one. If you want the product that comes out of all this, we cover it in enterprise document search.

What does a RAG really cost to build and run?

No catalogue figures, because they depend on the vendor and the volume — but the real cost structure, which is what's usually missing from the quotes companies bring us to review:

• Indexing is a small, one-off cost. Converting and embedding a few thousand documents is cheap and happens once.
• Querying is the recurring cost. Driven by questions per day and how much text you pass the model each time. Retrieving ten passages instead of three multiplies the bill and rarely improves the answer.
• Index storage is lower than people fear, except with very large corpora or frequent full reindexing.
• What actually blows the estimate is cleanup hours. Marking currency, deciding what's in and what's out, fixing table extraction. In the projects we've seen, this groundwork takes more time than building the system.
• And live maintenance. Someone has to review the failure log, reindex what changes and re-run the evaluation set every quarter. It's the MLOps part almost nobody budgets, and it decides whether the system is still trustworthy in month nine.

The general framework for asking for comparable quotes is in what an AI project costs. For RAG, the single question that tells you most about a vendor is simple: "how will you measure retrieval hit rate, and with which question set?" If there's no clear answer, what you're being sold is a demo.

Why do most RAG projects die before production?

Because the system inherits the repository's mess and serves it with an air of authority. The patterns that repeat most:

1. Scope by volume instead of by value. "Let's index the whole network drive" guarantees noise and guarantees a permissions incident. One domain, one department, a few hundred well-chosen documents: that's a first case. 2. No document owner. If nobody is accountable for a procedure being up to date, a PDF that was correct in 2022 and never reviewed is false information in the format of good information. That's data quality applied to documents. 3. No visible citations. A clickable citation isn't decoration: it's what lets someone verify in five seconds and what earns the system trust. Without it, users can only believe or distrust. 4. No failure channel. A "this answer is wrong" button that logs the question, the answer and the retrieved passages. Without that log there's no improvement, only opinions. It's the same mechanism that separates live pilots from dead ones in why AI pilots fail. 5. No decision on where data is processed. If you're indexing case files, payroll or medical records, settle first where processing happens, what gets logged per query and which legal basis covers it. We develop that in GDPR and AI.

A starting route we've seen work, without a full-time dedicated team:

| Weeks | What happens | Deliverable | |---|---|---| | 1-2 | Pick a domain with real pain and an identifiable owner | Written scope: which folders are in and which aren't | | 3-4 | Build the reference question set | 50-150 questions with document and section | | 5-6 | Ingestion, extraction and structure-aware chunking | Index measured against those questions | | 7-8 | Hybrid retrieval, metadata filters and reranking | Retrieval hit rate above 90 % | | 9-10 | Generation with mandatory citation and allowed abstention | Answers traceable to the paragraph | | 11-12 | Pilot with 10-15 users and a failure log | A decision backed by data: expand, adjust or stop |

Frequently asked questions

Does RAG eliminate made-up answers?

It reduces them a lot and makes them detectable, but it doesn't eliminate them. If the system fails to retrieve the right passage and isn't configured to abstain, it will fill the gap with plausible language. You mitigate it with three combined measures: mandatory citation for every statement, explicit permission to say "I can't find this", and measuring correct abstention on the reference set.

How many documents make it worthwhile?

Fewer than people imagine. With 200 or 300 documents consulted daily by several people there's already a case: the value comes from consultation frequency and the cost of not finding, not from archive size. The real limit runs the other way — indexing a hundred thousand documents with no currency metadata produces a fast, unreliable system.

Do my documents leave the company?

It depends on the architecture, and it's a business decision before it's a technical one. You can build it with vendors that don't retain data, in an EU cloud with a signed processing agreement, or on your own infrastructure with open models — more control, more cost, more maintenance. What isn't acceptable is not knowing the answer: ask for it in writing before you upload a single file.

Do I need to fine-tune the model as well as build RAG?

Almost never at the start. Fine-tuning makes sense when the problem is output format or tone and you've exhausted instructions — not when the problem is that the system doesn't know a fact. Start with RAG, measure, and if the failure persists once retrieval hit rate is above 90 %, then consider touching the model.

If you're weighing up a RAG over your own documents, the useful first step isn't a proof of concept with three PDFs: it's looking at one real domain and counting how many versions coexist, how many documents carry an effective date and how long it takes someone to find an answer today. That's the audit, and it comes with an honest estimate of the groundwork needed before retrieval adds anything. If you'd rather sanity-check it in half an hour, let's talk and we'll tell you straight whether your case is RAG, document housekeeping, or both in that order.