On this page
When a business adds a large language model (LLM) to its product, the first results are often impressive and then quickly frustrating. The model does not know your pricing rules, invents answers about your policies, or writes in the wrong tone. Two techniques are usually proposed to fix this: retrieval-augmented generation (RAG) and fine-tuning. They solve different problems, and choosing the wrong one can waste months.
Quick definitions
Retrieval-augmented generation (RAG)
With RAG, your application searches your own data for passages relevant to the user's question and gives them to the model together with the question. The model then answers using that material. The approach was described in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis and colleagues, and it has since become a standard way to connect models to private or changing information.
Fine-tuning
Fine-tuning continues training an existing model on your own examples, so its default behaviour changes. You show it many input and ideal output pairs, and it learns the pattern: a format, a tone, a classification scheme or a specialised task.
Start with the problem, not the technique
| Symptom | Likely cause | Start with |
|---|---|---|
| Answers miss or invent company-specific facts | The model lacks your knowledge | RAG |
| Information changes often (stock, prices, policies) | Training data is frozen in time | RAG |
| Users need to see where an answer came from | No link to sources | RAG with citations |
| Correct content, wrong format or tone | Behaviour, not knowledge | Prompting, then fine-tuning |
| A narrow, repetitive task at high volume | A large general model is more than needed | Fine-tuning a smaller model |
| Different users may see different data | Access control | RAG with permission filtering |
Try these first
Many problems disappear before you reach RAG or fine-tuning:
- Clear instructions that state the role, the audience, the rules and what to do when unsure.
- A few worked examples of good answers placed directly in the prompt.
- Structured outputs such as JSON schemas when your application needs to parse the result.
- Putting small reference material directly in the prompt. If your policy document is short, you may not need a retrieval system at all.
When RAG is the right choice
RAG fits when the answer depends on information that is private, large, or changes over time. Support assistants over help-centre articles, internal search across company documents, and assistants that answer questions about a customer's own account data are typical examples.
RAG has three practical advantages over trying to train knowledge into a model:
- Freshness. Update a document and the next answer reflects it. No retraining.
- Traceability. You can show users which sources an answer used, which builds trust and makes errors easy to spot.
- Access control. Retrieval can be filtered by the user's permissions, so the model only sees what that user is allowed to see. Anything trained into a model is available to every user, so it cannot be hidden from specific users.
When fine-tuning is the right choice
Fine-tuning fits when the model already has the knowledge it needs but must behave in a consistent, specialised way. Examples include classifying support tickets into your own categories, extracting fields from documents into a fixed structure, or matching a strict house writing style across thousands of outputs.
Fine-tuning can also let a smaller, cheaper model perform one narrow task as well as a larger general model, which matters at high volume. It is generally a less reliable way to teach a model new facts than retrieval, and those facts go stale as soon as your information changes.
What a production RAG system includes
- Ingestion: collect documents and data from their sources and keep them in sync.
- Chunking: split content into passages small enough to retrieve precisely, while keeping headings and context.
- Embedding and indexing: convert passages into vectors and store them. For teams already on PostgreSQL, the pgvector extension keeps this in the same database.
- Retrieval: combine vector search with keyword search, and filter by tenant and user permissions.
- Reranking: reorder the best candidates so the most relevant passages reach the model.
- Generation: instruct the model to answer only from the provided passages, to cite them, and to say when it does not know.
- Evaluation and monitoring: measure answer quality continuously, not just at launch.
Using both together
The two techniques are not exclusive. A common pattern uses RAG to supply current, permission-checked knowledge, and a fine-tuned model to produce answers in a consistent format. Start with RAG and strong prompts, and add fine-tuning only when evaluation shows a behaviour problem that prompting cannot fix.
Evaluate before and after every change
Build a golden dataset early: realistic questions with the answers and sources a good system should produce. Run it after every change to prompts, chunking, retrieval or models. Track at least these:
- Retrieval quality: did the right passages appear in the results?
- Groundedness: is every claim in the answer supported by the retrieved material?
- Answer quality: is it correct, complete and in the right format?
- Refusals: does the system admit when it lacks the information instead of guessing?
Security and privacy risks to plan for
- Prompt injection. Retrieved documents and user input can contain instructions that try to override your system. Treat all retrieved text as untrusted data, and never let the model take sensitive actions without checks in your own code.
- Data leakage between users. Enforce permissions in retrieval, before content reaches the model, not by asking the model to keep secrets.
- Personal data. Minimise what you send to model providers, review their data retention settings, and cover it in your data processing agreements.
- Over-trust. Show sources and make it easy for users to report wrong answers.
A simple decision guide
- Write down the exact failure you are trying to fix, with real examples.
- Build a small evaluation set from those examples.
- Improve prompts, examples and output structure. Re-evaluate.
- If failures are about missing or changing knowledge, add RAG. Re-evaluate.
- If failures are about consistent behaviour at scale, consider fine-tuning. Re-evaluate.
Adding AI to an existing product? We design and build RAG systems, AI agents and integrations with evaluation built in from the start. Learn more about our AI solutions, or talk to us about your use case.
Common questions
Is RAG cheaper than fine-tuning?
It depends on volume and complexity. RAG adds retrieval infrastructure and longer prompts on every request, while fine-tuning adds training work up front and again whenever requirements or base models change. Compare total cost against your expected usage rather than assuming either is cheaper.
Can fine-tuning stop a model from hallucinating?
Not reliably. Grounding answers in retrieved sources, requiring citations and instructing the model to say when it does not know are more effective ways to reduce invented answers.
Do we need a separate vector database for RAG?
Not necessarily. If you already use PostgreSQL, the pgvector extension can store and search embeddings alongside your existing data, which is often enough for early and mid-sized workloads.