How to add RAG to your SaaS product

Short answer: Ingest your customers’ documents into chunks, store their embeddings next to your tenant data, retrieve only that tenant’s most relevant chunks for each question, and have the model answer with citations. Then measure answer quality on a fixed test set before you ship.

What is RAG and when does a SaaS product need it?

Retrieval-augmented generation (RAG) gives a language model the right context at question time. Instead of relying on what the model learned in training, your application searches your own data and passes the relevant passages to the model with the question.

A SaaS product needs RAG when customers ask questions whose answers live in their own content: policies, contracts, help articles, tickets or product documentation. If the answer depends on data the model has never seen, RAG is usually the right starting point, before any fine-tuning.

How should you ingest and chunk documents?

Ingestion turns uploaded files into searchable pieces:

1. Extract text from each format (PDF, Word, HTML) and keep structure such as headings and page numbers.
2. Split into chunks that hold one idea each. Splitting on headings and paragraphs works better than fixed character counts, because it keeps related sentences together.
3. Store metadata with every chunk: tenant, document, section and page. You need it for filtering and for citations.
4. Re-ingest on change. When a document is updated or deleted, its chunks must follow.

Test chunking on real customer documents early. Problems that never show up in demo files, such as scanned pages, tables and long appendices, show up immediately in real data.

Where should embeddings live?

For most SaaS products, the simplest good option is the database you already run. PostgreSQL with the pgvector extension stores embeddings next to tenant data, so tenant filters are plain SQL and there is no second system to secure, back up and keep in sync.

A dedicated vector database can make sense at very large scale or for specialized search needs. Start simple, and move only when measurements show you need to.

How do you keep tenants’ data separate?

Tenant isolation is the most important rule in a multi-tenant RAG system: one customer’s documents must never appear in another customer’s answer.

• Store the tenant ID on every chunk.
• Apply the tenant filter inside the retrieval query itself, not after results come back.
• Enforce it in one place in your application layer, so no code path can forget it.
• Add automated tests that try to retrieve another tenant’s data and expect nothing.

How do you evaluate answer quality?

Build a test set before you tune anything: real questions, the documents that answer them, and what a good answer contains. Then measure two things separately:

• Retrieval: did the right chunks come back for each question?
• Answers: is the answer correct, supported by the cited sources, and does it say “I don’t know” when the documents do not contain the answer?

Run the same test set after every change to chunking, prompts or models. Without it, improvements are guesses.

What does it cost to run?

Costs come from three places: creating embeddings at ingestion, storing them, and the model calls for each question. Question-time model calls usually dominate.

Keep costs predictable by limiting how many chunks you send per question, caching repeated questions where it is safe, setting per-tenant usage limits, and logging token usage per feature. A provider-agnostic model layer also lets you move to a cheaper model for simple questions without rewriting the feature.