How to Build a RAG Pipeline

This is the hands-on companion to What is RAG? — if that article explained the concept, this guide walks through actually building one. It's the same stack we teach in the Generative AI course's RAG module: one embedding model, one vector database (Chroma), one LLM, done properly end to end.

Before you start

You'll need: Python 3.10+, an API key for an LLM provider (OpenAI, Gemini, or Groq all work), the chromadb package, and a document to test with — a PDF or a set of text files works well. This guide shows the shape of the pipeline; treat the code as a reference, not a copy-paste production script.

Step 1: Chunk your documents

Split documents into overlapping chunks — small enough to be specific, large enough to retain context. 300–800 tokens per chunk with 10–20% overlap is a reasonable starting point; the right size depends on your content (dense technical text often needs smaller chunks than narrative text).

def chunk_text(text, chunk_size=500, overlap=75):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

Step 2: Generate embeddings

Convert each chunk into a vector using an embedding model. Keep the embedding model consistent for both your documents and your queries — mixing embedding models breaks similarity search entirely, since the two sets of vectors won't share the same geometric space.

from chromadb.utils import embedding_functions

embed_fn = embedding_functions.DefaultEmbeddingFunction()
embeddings = embed_fn(chunks)  # one vector per chunk

Step 3: Store & index vectors

Store each chunk's embedding alongside its original text and useful metadata (source filename, page number, section heading, date). Metadata is what lets you filter later — don't skip it, even in a prototype.

import chromadb

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("my_documents")

collection.add(
    documents=chunks,
    embeddings=embeddings,
    metadatas=[{"source": "handbook.pdf", "chunk_index": i} for i in range(len(chunks))],
    ids=[f"chunk-{i}" for i in range(len(chunks))],
)

Step 4: Retrieve relevant chunks

At query time, embed the user's question with the same embedding model, then ask the vector store for the top-K most similar chunks. K=3 to K=6 is a common starting range — too few and you risk missing the answer, too many and you dilute the LLM's context with irrelevant text.

results = collection.query(
    query_texts=[user_question],
    n_results=4,
)
retrieved_chunks = results["documents"][0]

Step 5: Generate a grounded answer

Insert the retrieved chunks into the prompt explicitly, and instruct the model to answer only from that context — this single instruction meaningfully reduces hallucination compared to letting the model blend retrieved context with its own memorized knowledge.

context = "\n\n".join(retrieved_chunks)
prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {user_question}"""

response = llm_client.generate(prompt)

Step 6: Evaluate before you ship

Before trusting this pipeline, test retrieval and generation separately: does the right chunk show up in the top-K results for a set of known questions (retrieval quality), and does the generated answer actually stick to the retrieved context (faithfulness)? See our LLM Evaluation Guide for how to build this test set properly.

Common pitfalls

• Chunks too large or too small — test a few chunk sizes against real questions rather than guessing once and moving on.
• No metadata filtering — pure semantic search without filters returns "similar but wrong document" results more often than expected once you have more than one source document.
• Ignoring the "I don't know" case — always instruct the model to say when the context doesn't contain an answer, otherwise it will guess.
• Skipping evaluation — a RAG demo that works on your three test questions can fail on a large fraction of real ones. See vector databases explained for what's happening under the hood when retrieval misses.

This exact pipeline — plus source citation, a real UI, and your own PDFs — is the Chat With Your Documents project students ship in Sessions 9–10 of the Generative AI course, with instructor feedback while you build it.