RAG on company data: how to build one that does not lie

RAG on company data does not lie when four things hold. Retrieval finds the right chunk from the current version of the document. Permissions are filtered in the index query, not in the prompt. The model may answer only from the retrieved chunks, with a citation checked by code, and refuses when it finds no source. And a set of questions with known answers measures all of it on every change.

This is a technical article for CTOs and developers. Why and when a company needs an internal assistant at all is covered in AI assistant for your company on your own data.

What does a RAG that does not lie consist of?

Most tutorials stop at "split the documents, store the vectors, send the top 5 to the model". That is a demo. Company data adds two steps the demo does not have: permissions and evals.

Steps 3 and 5 decide whether RAG lies. The model rarely makes things up from nothing. It answers wrongly from a chunk that retrieval handed it.

Where does retrieval break most often?

When RAG answers wrongly, first look at what the model got as input. That is where the problem usually was:

The last row is the most treacherous. Similarity search never returns "nothing". It returns the five least bad chunks and the model assembles something from them. That is why the model must be allowed, and trained, to refuse.

Document permissions: filter in the query, not in the prompt

An internal RAG reads policies, price lists and contracts. If the model gets a chunk from a contract, it can use it in an answer to anyone. An instruction in the prompt like "do not show contracts to sales" is not access control, it is a request. Same as with prompt injection: what the model never gets, it cannot reveal.

So the filter belongs in the query that selects the chunks. A simplified example in Postgres with the pgvector extension:

-- $1 = question embedding, $2 = user groups from the login, not from the question text
select c.id, c.document_id, c.content
from chunks c
join documents d on d.id = c.document_id
where d.allowed_groups && $2::text[]
  and d.valid_from <= now()
  and (d.valid_to is null or d.valid_to > now())
order by c.embedding <=> $1
limit 30;  -- wider set for reranking, only the best go to the model

What matters is where $2 comes from. From the logged-in user's session, never from what they typed into the question. And permissions sync from the source, for example from shared drives or the system the document lives in. If the document gets locked in the source, it gets locked in the index too.

How do you make the model refuse when it has no source?

A prompt like "answer only from the context" helps, but it is not enough. What you can rely on is what code checks. So the model does not return free text but structured output: either an answer with the ids of the cited chunks, or a refusal.

type RagAnswer = { answer: string; citations: string[] } | { refused: true; reason: string }

function check(out: RagAnswer, retrievedIds: Set<string>): RagAnswer {
  if ('refused' in out) return out
  // citation must exist and point only at chunks this user actually got
  if (out.citations.length === 0 || out.citations.some((id) => !retrievedIds.has(id)))
    return { refused: true, reason: 'no valid source' }
  return out
}

Code cannot verify that the answer really follows from the source. It can verify that the source exists and that the model was allowed to see it. Whether the answer matches the source is measured in evals, and for looser answers also with a judge working from criteria. How such a judge works and where it gets things wrong is in LLM as a judge.

The user sees a link to the document next to the answer. When RAG refuses, it says so plainly and shows where it looked. "I did not find this in the documents" beats a convincing mistake. More on why models make things up in AI hallucinations.

How do you measure that RAG does not lie?

With a set of questions people actually ask, each with the correct answer and the document that contains it. Not with questions a developer makes up while writing the prompt. One case looks like this, for example:

- id: business-trip-approval
  as_user: groups=[sales]
  question: Who approves my business trip abroad?
  expect_source: travel-expenses-policy.pdf#approval
  expect_answer_contains: ["head of department"]
- id: salaries-no-access
  as_user: groups=[sales]
  question: How much does the warehouse manager earn?
  expect: refused
- id: outside-documents
  as_user: groups=[sales]
  question: What is the euro exchange rate today?
  expect: refused

The example is made up, it is about the shape. On every change to the prompt, model, chunking or sources the whole set runs and four numbers are tracked separately:

• Retrieval: was the expected source among the chunks the model got?
• Answer: does it contain the right fact and cite the right source?
• Refusal: did it refuse where there is no answer, and not refuse where there is one?
• Permissions: cases where someone got a chunk they must not see. The only acceptable value there is zero.

Wrong answers from production that users flag go back into the set as new cases. That way the set grows with what people really ask.

Is your RAG ready for company data?

Go through your setup:

When not to build RAG

• There are a few dozen documents and they rarely change. Putting them whole into the model's context is often enough, and you skip the index.
• The answer is in a database, not in text. Ask for order status or stock with a query to the system, not with a document search.
• Nobody knows which documents are valid. RAG over a mess answers fast and wrong. Clean up the sources first.
• You have nobody to write evals. Without a set of questions with known answers you do not know whether RAG lies, and you will not find out after a model change either.

Need a RAG people will trust?

We build RAG on company data as part of our custom AI agent for business service: retrieval, permissions, citations and evals right in your repository. The code, the index and the data are yours from day one.

We get back to you within two business days with a few questions about your work. Then we agree on what we do first.

If you would rather talk first, book a 15-minute intro call.

What does your RAG do today when you ask it something the documents do not contain?

Frequently asked questions

What is RAG?

RAG (retrieval-augmented generation) means the system first retrieves passages from your documents for the question, and only then does the language model write an answer from them. The model answers from your data, not from what it remembers from training.

Why does RAG on company data lie?

Most often because retrieval returned a wrong or outdated chunk and the model dutifully answered from it. Less often because the model added something that was not in the chunk. The first is fixed with metadata, hybrid search and retrieval evals, the second with code-checked citations and a rule to refuse without a source.

How do you handle document permissions in RAG?

With a filter in the index query, based on the user identity, before the model sees any chunk. An instruction in the prompt like "do not show salaries" is not access control. What the model never gets, it cannot reveal.

How do you test whether RAG answers correctly?

With a set of real questions, each with the correct answer and the source that contains it. It also includes questions the documents do not answer and questions from a user who must not see the document. It runs on every change to the prompt, model, chunking and documents.

Do I need a vector database for RAG?

Not necessarily a separate one. For thousands of documents, your existing database with a vector extension and full-text search is often enough. More important than the database is that it can filter by permissions and version in the same query.