Building a Production RAG Pipeline: From Prototype to Scale

Why Production RAG Is Hard

A RAG prototype takes an afternoon. A production RAG system that is accurate, fast, and cheap at scale is a multi-week engineering effort. The gap comes from: chunking strategy, retrieval quality, answer faithfulness, latency, and ongoing evaluation.

Architecture Overview

User Query
    ↓
Query Rewriting (expand synonyms, fix spelling)
    ↓
Hybrid Retrieval (vector + BM25)
    ↓
Re-ranking (cross-encoder)
    ↓
Context Assembly (deduplicate, truncate)
    ↓
LLM Generation
    ↓
Faithfulness Check (optional)
    ↓
Response

Chunking for Production

Do not use fixed-size chunking. Use semantic chunking — split at paragraph boundaries, section headers, or sentence breaks. Preserve metadata (source URL, page number, section title) in every chunk for citation generation.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    separators=["

", "
", ". ", " "],
    chunk_size=800,
    chunk_overlap=100,
    length_function=tiktoken_len,  # Token-based, not character-based
)

Evaluating RAG Quality with Ragas

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall]
)
print(results)
# {'faithfulness': 0.89, 'answer_relevancy': 0.91, 'context_recall': 0.83}

Build your evaluation suite before your RAG pipeline. Without metrics, you are flying blind — every "improvement" might be making the system worse in ways you cannot see.

Latency Optimisation

• Cache embeddings for unchanged documents — recomputing embeddings on every query is expensive.
• Use approximate nearest neighbour (ANN) search for large corpora.
• Stream the LLM response — first token in <1s even if full response takes 5s.
• Pre-warm: keep your vector store client connection alive, not re-initialised per request.