"Can we train the AI on our documents?" is one of the first questions we hear from teams exploring AI. It is a fair question, but it hides a choice. There are three common ways to give an AI model your company's knowledge, and they behave very differently once real people start relying on the answers:
- Put it in the prompt: send the relevant documents along with every question.
- Retrieval (RAG): search your documents first, then answer from what was found.
- Fine-tuning: train a model on examples so the knowledge or behaviour is built into it.
Most teams that say "train" actually need retrieval. This guide explains why, when the other two options are the better choice, and how to decide for your own case.
Option 1: Put the knowledge in the prompt
The simplest approach is to include the documents directly in the request. Modern models can read a very large amount of text at once, so a staff handbook, a price list, or a set of product FAQs can often fit in a single prompt.
This works well when:
- The knowledge is small and fits comfortably in one request
- It changes often, because updating it is just editing the text you send
- Everyone asking questions is allowed to see all of it
Where it breaks down:
- Scale. Hundreds of documents, years of support tickets, or a full policy library will not fit, and sending everything with every question gets slow and expensive.
- Focus. The more text a model reads, the easier it is for the one relevant sentence to get lost. Answers become less precise as the prompt grows.
- Access control. If some documents are for finance only, you cannot send the whole library to everyone.
Option 2: Retrieval-augmented generation (RAG)
RAG adds a search step. Your documents are cleaned, split into passages, and indexed. When someone asks a question, the system finds the most relevant passages and gives only those to the model, which writes an answer based on them and points to its sources.
This is what most business knowledge systems need, because it solves the problems above:
- It scales. The library can hold thousands of documents; each answer only reads the few passages that matter.
- Answers can cite their sources. People can click through to the policy or SOP behind an answer, which is how a team learns to trust it.
- Updating is simple. When a document changes, you re-index it. Nothing has to be retrained.
- Permissions can be respected. Search can be filtered by team or role, so people only get answers from documents they are allowed to see.
- "Not found" is a valid answer. If nothing relevant comes back, the assistant can say so instead of guessing.
That last point is why we used retrieval for Atleet Buddy, a WhatsApp nutrition assistant that answers only from one book. Grounding each answer in a retrieved passage is what keeps it inside the book and lets it decline questions the book does not cover.
The catch is that a RAG system is only as good as its search. The most common reasons one disappoints have nothing to do with the AI model:
- Scanned PDFs, tables, and screenshots that were never turned into clean, searchable text
- Documents split in the wrong places, so an answer needs two passages that are never retrieved together
- Old and new versions of the same policy both in the index, contradicting each other
- Nobody owning the content, so it slowly goes stale
That is why the first part of our RAG Knowledge System Sprint is auditing, cleaning, and structuring the content, before any retrieval pipeline is built.
Option 3: Fine-tuning
Fine-tuning trains an existing model on many examples of inputs and the outputs you want. It changes how the model behaves. It is the right tool when the problem is behaviour, not facts:
- A consistent tone or writing style across thousands of replies
- A fixed output format, such as always producing the same structured fields
- Classification into your own categories, like routing tickets to your team names
- Specialised language the base model handles poorly
What fine-tuning is not good at is teaching a model facts that people will rely on:
- No sources. A fine-tuned model cannot tell you which document an answer came from, so nobody can check it.
- Hard to update. When a policy changes, the old version is still inside the model until you retrain it.
- No permissions. Everything the model learned is available to everyone who uses it.
- Blended answers. Models can mix up similar facts they absorbed in training, and do it with full confidence.
Fine-tuning also has a cost that is easy to underestimate. The training itself is rarely the expensive part; preparing hundreds or thousands of good examples is. And in many cases, clear instructions plus a handful of examples in the prompt get you the same tone and format without training anything.
How the three compare
When the knowledge changes, a prompt is updated by editing the text, RAG by re-indexing the document, and fine-tuning only by retraining.
When someone asks "where did that come from?", a prompt and RAG can point to the source; a fine-tuned model cannot.
When only some people should see some documents, RAG can filter by role; the other two cannot do it cleanly.
When the goal is a consistent style or format, fine-tuning or good prompt examples are the right tools; retrieval does not help.
Combining them
These options are not mutually exclusive. A common, sensible setup is retrieval for the facts plus instructions and examples for the behaviour: RAG finds the right passages, and the prompt sets the tone, the format, and the rules for what the assistant must refuse. Fine-tuning gets added later, if at all, when volume is high and prompt examples are no longer enough to keep the style consistent.
How to choose
Ask these questions in order:
- Does the knowledge fit in a few pages, and can everyone see all of it? Start by putting it in the prompt.
- Is it large, does it change, or do people need to check the source? Use retrieval.
- Do different people need access to different documents? Use retrieval with permission filtering.
- Is the real problem tone, format, or classification rather than facts? Try prompt instructions and examples first, and consider fine-tuning only if they fall short.
And before any of it, check whether you need AI at all. If the answer is always looked up in the same table by the same rule, a plain search or automation is cheaper and easier to trust.
If you have a library of SOPs, policies, or support docs that people struggle to search, our RAG & Knowledge Systems page explains how we build these systems, and the free AI Systems Audit is a quick way to find out which of the three approaches fits your case.
