The Problem RAG Solves
LLMs are trained on data up to a certain date and know nothing about your proprietary documents, your product, or events after their cutoff. RAG solves this by retrieving relevant information from an external knowledge base and injecting it into the LLM's context at inference time.
How RAG Works
- Indexing: Split documents into chunks, embed each chunk as a vector, store in a vector database.
- Retrieval: Embed the user's question, find the most similar document chunks by vector similarity.
- Generation: Pass retrieved chunks + user question to the LLM. The LLM answers using the retrieved context.
Building a RAG Pipeline
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
# 1. Ingest
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(documents)
vs = Chroma.from_documents(chunks, OpenAIEmbeddings())
# 2. Query
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o"),
retriever=vs.as_retriever(search_kwargs={"k": 5})
)
result = qa.run("What is our refund policy?")Chunking Strategy
Chunking is the most impactful parameter you can tune. Smaller chunks (200–500 tokens) retrieve more precisely; larger chunks (1000–2000 tokens) provide more context per retrieval. Experiment with your specific data.
The single biggest improvement to RAG accuracy is usually better chunking, not a bigger model or more sophisticated retrieval.
Advanced RAG Techniques
- Hybrid search: Combine vector similarity with BM25 keyword search. Catches both semantic and lexical matches.
- Re-ranking: Use a cross-encoder to re-rank top-k candidates before passing to the LLM.
- Query decomposition: Break complex questions into sub-queries, retrieve for each, then synthesise.
- Self-RAG: The LLM decides whether to retrieve at all, and critiques its own generated answers.
Other Articles
