RAG in business: getting an agent to answer from your documents

The principle, without jargon

A language model knows a lot about the world and nothing about your company. It does not know your terms and conditions, your complaints procedure or the latest version of your catalogue. Ask it anyway and it will answer, with a plausible sentence that is often wrong.

Retrieval augmented generation, usually shortened to RAG, fixes this with a simple mechanism. Before answering, the system searches your documents for the passages that relate to the question, then passes them to the model along with the question. Anthropic describes it as a method that retrieves relevant information from a knowledge base and appends it to the user's prompt. The model no longer answers from memory: it answers from what has just been put in front of it.

Three building blocks describe the whole thing:

• The corpus. The documents the agent is allowed to use: procedures, product sheets, standard contracts, FAQs, internal notes.
• Retrieval. The mechanism that, for a given question, finds the few useful passages among thousands.
• Generation. The model, which writes an answer from those passages and says where it comes from.

This architecture sits at the heart of most agents we build, whether an internal assistant for a support team or an agent that answers visitors on a website. Our page on designing an AI agent places it in the wider project. Here we look at what makes it work, and what makes it fail.

Before the technology: preparing the corpus

In this kind of project, a large share of the quality is decided before the first line of code. An agent that answers from your documents will never be better than those documents. If they contradict each other, are outdated or incomplete, so is the agent, only more confident.

Document quality

Start with an honest inventory. Where do the authoritative documents live: an intranet, a shared drive, a document management tool, one person's inbox? Which ones are actually up to date? Which ones exist in several versions, none marked as the right one?

Formats matter too. A crooked scanned PDF, a table laid out for printing or a slide deck where meaning depends on the position of boxes are hard for a machine to read. Sometimes they need converting, restructuring, or leaving out of the first batch. A small, clean corpus beats an exhaustive one where the right answer sits next to three obsolete ones.

A simple signal helps sort things out: if an experienced colleague hesitates over which document is authoritative for a given question, the agent will hesitate too. The business has to settle that before it gets indexed.

Keeping it current

A corpus is not a snapshot. Conditions change, procedures evolve, a product leaves the catalogue. The question to ask from day one: when a document changes at the source, how long until the agent knows, and who makes sure it does?

The technical answer is automatic synchronisation with the source, reindexing what changed and removing what was deleted. The organisational answer is a named owner for each family of documents. Without that owner, the corpus decays silently and nobody notices until a customer gets a wrong answer.

Access rights

This is the most neglected point. In your company, not everyone sees everything: the salary grid, a client's contracts, board notes. If those documents enter the corpus without their permissions, the agent becomes a way to read them for anyone who asks the right question.

OWASP, which publishes a reference list of risks specific to language model applications, classes this among vector database weaknesses. It recommends fine grained access control, with strict logical partitioning of data by user group, and logging of retrievals. In practice, every indexed passage keeps track of who may read it, and retrieval filters on the identity of the person asking, before anything reaches the model. Telling the model not to reveal certain information protects nothing: once the passage has been sent to it, it can come out. Our AI security and compliance team handles this from the design stage.

Splitting documents and finding the right passages

A model does not reread your entire documentation for each question. The system therefore splits documents into passages and only sends the most relevant ones. The split looks trivial. It decides a large part of answer quality.

The orphan passage problem

A passage lifted out of its document loses its context. Anthropic gives the example of a sentence from a financial filing announcing revenue growth without saying which company or which period. In your case it will be a paragraph stating "this warranty does not cover wear parts", with no way of knowing which warranty or which product range.

Several answers exist and can be combined:

• split along the document's structure (headings, articles, sections) rather than at a fixed length;
• keep the document title, original section and date with each passage;
• add a short context sentence to each passage, written from the whole document. This is the approach Anthropic calls contextual retrieval.

Two ways of searching, best combined

So called semantic search compares the meaning of the question with the meaning of passages. It finds a passage on termination when the user asks how to stop a subscription. It is less reliable on exact matches: a product reference, an article number, an internal code. Lexical keyword search does the opposite. Anthropic recommends combining both, then adding a reranking step that sorts retrieved passages and keeps only the most relevant before sending them to the model.

For a manager, the takeaway is simple: retrieval is a component in its own right, one that gets tuned and measured. When an agent answers badly, the first question is "did it find the right passage?", before "did the model misunderstand?". In the diagnostics we run, that is often where the problem lies.

Connecting the sources (drive, document management, knowledge base, ticketing tool) is data and integration work. It guarantees that what is indexed matches what exists, with the right permissions and the right version.

Citing sources, every time

An answer without a source asks for faith. An answer that cites the exact passage it relies on can be checked in seconds. That is the difference between an agent teams consult and an agent they work around.

Recent models do this reliably. Claude's citations feature, for example, returns for each claim the passage of the document that supports it, and the documentation states that these citations are guaranteed to point to the documents provided. The user sees the answer, the document name and the excerpt. They can open the source and check.

We draw three design rules from this:

• No source, no answer. If retrieval brings back nothing relevant, the agent says so and points to a person or a contact page. It does not fill the gap with general knowledge.
• The source is visible. The link to the original document appears in the interface, not only in technical logs.
• Dates matter. When a document has a validity date, it is shown with the citation. This stops an old procedure being taken for the current one.

Citation has a useful side effect: it makes corpus errors visible. When a user sees that a wrong answer relies on an outdated document, they know what to report, and to whom.

What document retrieval does not solve

Retrieval augmented generation clearly improves an agent's relevance. It does not solve everything, and two limits should be stated plainly before a project starts.

Data that commits the company

A price, an availability, a delivery lead time, a reference, an address: this information commits the company. Even when it sits in a corpus document, we never let the model copy or rephrase it. It can pick the wrong row in a table, mix two product ranges, or return an old rate left in an old PDF.

Our house rule: whatever commits the company is computed server side, from the source of truth (the catalogue, the sales system, the stock database), and checked before display. The model decides a price should be shown, the system fetches it. We covered this in why an AI agent must never quote a price.

Contradictory documents

Two documents say different things on the same subject. Retrieval brings back both. What does the model do? It picks one, often without saying so, or produces a synthesis that exists in neither. OWASP mentions this risk of knowledge conflicts when several sources feed the same base.

There is no purely technical fix, because the problem is organisational first. The safeguards we put in place:

• an explicit hierarchy of sources, for example the official procedure before an internal memo;
• version and date metadata to set aside the old;
• an instruction asking the agent to flag a contradiction rather than settle it;
• an automatic report to the corpus owner, who fixes it at the source.

A well built agent thus becomes a detector of documentation inconsistencies, which has value in itself, regardless of the answers it gives.

Evaluating before opening to users

An agent that answers from documents is evaluated at two levels, measured separately.

• Retrieval. For a given question, is the right passage among those retrieved? This is the kind of measure Anthropic uses in its contextual retrieval work: the share of relevant documents missing from the top results.
• The answer. Given the right passages, is the answer correct, complete, faithful to the source and in the expected tone?

Separating the two avoids looking in the wrong place. If retrieval finds the right passage and the answer is still wrong, the problem is in the instructions or the model. If the right passage is never found, no model will save the answer.

Measuring requires a test set: real questions, asked by real users or taken from support history, each with the expected answer and the document that contains it, validated by someone from the business. The set must include hard questions, ambiguous ones, and out of scope ones where the right answer is "I don't know, here is who to contact". Claude's documentation insists on this: a good evaluation set mirrors the real distribution of tasks, edge cases included.

The set is replayed after every change: new splitting, new model, new documents. Without it, you improve one thing and break another without knowing. Our article on deploying an AI agent in your company shows where this step fits in the project.

The mistakes we see most often

• Indexing everything at once. Plugging in the entire shared drive looks efficient. It is the surest way to surface drafts, duplicates and confidential documents. Start with a chosen scope, then widen it.
• Forgetting permissions. An agent that ignores access rights is a data leak waiting to happen. Filtering happens in retrieval, before the model.
• Blaming the model first. When an answer is wrong, check what retrieval returned. Changing models does not fix bad splitting.
• Neglecting updates. An index built once and never refreshed drifts at the first procedure change.
• Sending too much context. Sending many passages "just in case" dilutes useful information and adds to the bill on every call. That cost line is covered in our article on the cost of an AI agent in production.
• No way out. An agent must be able to say it does not know and hand over. If it cannot, it will make things up.
• Ignoring hosting. Your documents pass through the index, the search engine and the model. Where are they stored, who can access them, under which contract? See our article on hosting an AI project in France.

Where to start

A successful first project starts from a precise business question, asked often, whose answer already exists somewhere in your documents. A support team answering the same warranty questions every day. A purchasing department looking for the right clause in framework contracts. A finance department trying to find the rule that applies to a specific case.

Then, in this order: define the corpus and give it an owner, set access rights, build the test set with the business, and only then tune retrieval and choose the model. We work forward deployed: the engineer sits with you, on your real documents, alongside the people who will ask the questions. It is the most direct way to find out what the corpus really contains, and what it lacks.

If you have a use case in mind and want to know whether your documents are ready, our AI diagnostic reviews the corpus, permissions and scope before any development starts.

Sources

• Anthropic, Introducing Contextual Retrieval
• OWASP GenAI Security Project, LLM08 (2025), Vector and Embedding Weaknesses
• Claude documentation, Citations
• Claude documentation, Define success criteria and build evaluations

Frequently asked questions

Do we need to retrain a model on our documents?

Rarely. Retrieval leaves the model as it is and feeds it the right passages at question time. It is easier to keep current, since a changed document only needs reindexing, and easier to control, since every answer can cite its source.

Our documents are a mess. Do we need to tidy everything first?

No, but you need to choose. We start with a narrow scope on a precise use case and tidy that scope with the business. The rest of the corpus follows in batches as usage justifies it.

Can an agent like this answer our customers directly?

Yes, if the exposed corpus is limited to what a customer may read and if data that commits the company is produced by the system, never by the model. We often recommend starting internally, where errors are caught by people who know the right answer.

What happens when an employee asks about a document they are not allowed to see?

If permissions are carried properly into the index, retrieval simply does not return that document and the agent says it cannot find the information. The model never sees the passage, which is the only reliable protection.

Who maintains the agent once it is live?

Documents remain the responsibility of the teams that produce them, with a named owner per family. The technical side (synchronisation, index, monitoring answers, replaying tests) needs regular maintenance, which we can run or hand over to your team.