The Enterprise RAG Guide: how to build an AI that talks to your company data

What is RAG, and why do you need it?

RAG (Retrieval-Augmented Generation) is a technique in which a large language model, before it writes an answer, finds the relevant passages in your data sources and receives them as context. It makes the model speak from your documents, not from what it "memorized."

In an enterprise setting this has three concrete results: answers are current (the model isn't stuck at its training date), sourced (you can show which document every claim came from) and proprietary (they rely only on your data, so a competitor can't get the same answer from a public model).

The system's 4 layers

Every RAG system in production goes through the same four layers:

1. Ingestion: PDFs, Word files, wikis, CRM records and emails are read, cleaned and split.
2. Indexing: Each chunk is turned into a vector by an embedding model and written to a vector database.
3. Retrieval: The user's question is vectorized and the most relevant chunks are found — usually combined with keyword search (hybrid).
4. Generation: The LLM writes the answer using the retrieved chunks as context, and cites its sources.

Most projects are won or lost at layer 3: no matter how good the model is, it can't produce the right answer if you retrieve the wrong chunks. That's why the data layer in our capability map is so crowded.

Data preparation: 60% of the work

After forty-some deployments, we can say it plainly: in RAG projects, most of the time goes not into the model but into the data. Typical problems and their fixes:

• Duplicates and version clutter — if 6 versions of the same contract make it into the index, the model gives conflicting answers. Define a single source of truth and keep old versions out of the index.
• Scanned PDFs — documents without a text layer are invisible until they go through OCR.
• Tables — a table flattened into plain text loses its meaning; a table-aware parser is a must.
• Access permissions — a bot that answers HR documents for everyone is a data leak. Add permission labels (ACLs) to the index and filter on them at retrieval time.

Choosing a vector database

An honest comparison of the popular options:

The rule: start with whatever is already in your infrastructure. Switching vector databases is easy; reprocessing 100,000 documents indexed with the wrong chunking strategy is expensive.

Chunking strategy

Documents can't be handed to the model whole; they're split into pieces (chunks). Chunk size is a balancing act: with chunks that are too small, context breaks apart; with chunks that are too large, the relevance signal gets diluted.

# Solid defaults to start with
chunk_size      = 512 tokens    # split along section headings
chunk_overlap   = 64 tokens     # context bridge between chunks
strategy        = "semantic"    # split at meaning boundaries, not fixed sizes
metadata        = doc_name, section, date, access_level

For structured text such as contracts and regulations, splitting at clause boundaries — and for free text, at paragraph/heading boundaries — noticeably improves accuracy.

Preventing hallucinations

Four safeguards that are standard in our production systems:

• Mandatory citations — the model must quote the retrieved chunks for every claim in its answer.
• Confidence threshold — if the retrieval score falls below the threshold, the model says "this information isn't in the archive" instead of making up an answer.
• Answer-source consistency check — a second model checks whether the generated answer contradicts its sources.
• Human escalation — legal and financial question types are routed to a person automatically.

Measuring success

"It looks good in the demo" is not a metric. Before going live, build an evaluation set of at least 100 questions and measure these three: retrieval accuracy (is the right document in the top 5?), answer correctness (signed off by an expert) and source faithfulness (does the answer actually appear in the source it cites?). This set becomes your regression test for every model or prompt change.

If you're curious how this works on our side, take a look at our 4-phase methodology; for the commercial side of a similar setup, the AI Solutions page will point the way.

Frequently asked questions