RAG or fine-tuning: how do you put an LLM to work on your documents?

A language model was trained once, on public text, up to a certain date. It writes well and reasons correctly, but it knows nothing about your company. Ask it about your purchasing procedure and it makes up a plausible answer.

RAG, for retrieval-augmented generation, solves this without touching the model. Your documents are split into passages and stored in an index. For each question, the system finds the closest passages, attaches them to the question and asks the model to answer from what it has just read. The model works with the book open.

Fine-tuning goes the other way. You resume the model's training with your own examples, hundreds of questions paired with the expected answer, until it reproduces the behavior you want. The knowledge then moves into the model itself. The model works from memory.

That difference explains almost everything else: updates, sources, access rights and cost.

In a company, documents change all the time: a revised price list, a rewritten procedure, a contract signed yesterday. With RAG, the changed document is re-indexed and the next answer takes it into account. Nothing else changes.

With a retrained model, every change calls for a new training run, which means a new set of examples, a new evaluation and a new release. Between two runs, the model answers with the old version, without knowing it and without saying so.

RAG also lets you respect each person's access rights. The search only returns documents the user is allowed to read: a sales rep gets no answer drawn from HR files. In a retrained model, that separation does not exist. What it has learned, it can repeat to anyone.

One last point, often decisive in a regulated profession: the source. An answer produced by RAG points to the exact passage it rests on, and the reader checks it in one click. A retrained model cannot say where it got what it states.

Fine-tuning has real uses. They share one thing: you are asking the model to behave in a certain way, more than to know something.

First case, format. If every answer has to come out in a precise structure, always the same, thousands of times a day, a retrained model sticks to it better than a model that is reminded on every request.

Second case, classification. Sorting incoming emails by reason, tagging tickets, qualifying inquiries: with enough corrected examples, a small retrained model does this quickly and cheaply.

Third case, the tone or vocabulary of a profession, when a written instruction is not enough to get it.

In all three cases you need examples, many of them and of good quality. That is the real cost of fine-tuning: someone on your team has to produce and correct those examples, then start again when the need changes. Without that work the result disappoints, whatever the model.

Many projects need neither. Recent models accept very long texts as input. When the body of documents fits in a few dozen pages, you can hand the whole thing to the model with a clear instruction. It is simpler to build and to maintain, and it is enough for an internal guide or a price list.

Conversely, some tasks do not need a language model. Pulling an invoice number, a date or an amount out of a document that is always laid out the same way is a job for classic natural language processing: rules, or a small specialized model. These methods have been around for a long time. They are fast, cheap, and their output is easy to check.

The right sequence is to try things in this order: a simple rule, then a well-written instruction, then RAG, and fine-tuning last. Each step adds complexity and upkeep. You only take it if the previous one falls short, and you check that on a test set drawn from your real documents.

With RAG, your documents stay in an index you control. For each question, the model reads only the few relevant passages, and reading them does not change it. You can take a document out of the index overnight: the next answer will no longer use it.

With fine-tuning, your data goes into the model's parameters. Nobody knows how to cleanly remove a piece of information from a trained model. If a customer asks for their data to be erased, or if a document was included by mistake, the only reliable fix is to retrain without it. For personal data, that is a serious difficulty under the GDPR.

In both cases, the hosting question remains: where the model runs, where the index is stored, who has access. On our AI projects, data is hosted in France by default, or on your own infrastructure when its sensitivity requires it.

You want an assistant that answers your teams' questions from your procedures, contracts or documentation: RAG. It is the most common case, and the one where the cited source matters most.

You handle a large volume of requests to classify or format, always against the same grid, and you have a corrected history: fine-tune a small model, possibly backed by document search.

Your documents fit in a few dozen pages and rarely change: a well-written instruction, with neither RAG nor training.

You do not know yet: start by measuring. We gather real questions, write the expected answer for each, and try each approach on that set. That is what we do during scoping, which takes two weeks.