Admissions open for our live online programs · Starts 5 December 2026 · Live online, learn from anywhere

Generative AI

How to Build a RAG Pipeline: A Step-by-Step Guide with Python Code

Build a RAG pipeline step by step: load, chunk, index, retrieve, prompt, generate and evaluate. Runnable Python code, architecture choices and common mistakes.

Short answer: A RAG (retrieval-augmented generation) pipeline has seven steps: load your documents, chunk them into small passages, index the chunks (embeddings in a vector store, keyword search, or both), retrieve the most relevant chunks for each question, augment a prompt with those chunks, generate an answer that cites them, and evaluate retrieval and answers on a fixed test set. Start with the simplest version that works end to end (the short Python script below), measure it, and only then add embeddings, hybrid search, reranking or an agent.

Not sure RAG is the right approach for your problem? Read RAG vs fine-tuning first.

What a RAG pipeline is (and why it exists)

A language model only knows what was in its training data. It doesn't know your company policies, last week's product changes or the PDF your customer uploaded. RAG fixes that by fetching relevant text at question time and putting it in front of the model.

The term comes from a 2020 research paper by Lewis and colleagues, which described models that "combine pre-trained parametric and non-parametric memory for language generation" (Lewis et al., arXiv:2005.11401). In plain words: the model's own weights are the parametric memory; your searchable document store is the non-parametric memory.

Today "RAG pipeline" usually means the whole system around that idea. LlamaIndex, for example, describes "five key stages within RAG": loading, indexing, storing, querying and evaluation (LlamaIndex, Introduction to RAG). This guide splits querying into its parts (retrieve, augment, generate) so you can see where things go wrong.

Do you even need RAG?

Before you build anything, check whether a simpler option works:

  • Small knowledge base: Anthropic notes that "if your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt" (Anthropic, Introducing Contextual Retrieval). Context window sizes vary by model, so check yours.
  • Style or format problem: if answers are factually fine but in the wrong tone or structure, better prompts or fine-tuning fit better than retrieval.
  • Structured data: if the answer lives in a database table, a text-to-SQL tool is often more accurate than embedding rows as text.

RAG is the right call when the knowledge is large, changes often, must be cited, or must respect who is allowed to see what.

The RAG pipeline architecture at a glance

The RAG pipeline architecture at a glance
StepWhat happensSimple first choiceUpgrade when needed
1. LoadRead PDFs, docs, web pages, tickets, database rowsPlain text filesDocument parsers for PDFs, tables and scans
2. ChunkSplit into passages and keep metadata (source, page, date)Paragraph splitsSection-aware or overlapping chunks
3. IndexTurn chunks into something searchableTF-IDF or BM25 keyword searchEmbeddings in a vector store, or hybrid search
4. RetrieveFind the top-k chunks for a questionTop 2–5 above a score thresholdHybrid search + reranking, metadata filters
5. AugmentPut the chunks into a prompt with clear rules"Answer only from context, cite ids"Query rewriting, context compression
6. GenerateCall the modelOne model callStreaming, structured output, guardrails
7. EvaluateCheck retrieval and answer qualityHit rate on 20–50 test questionsFaithfulness and context metrics, regression tests in CI

Step-by-step: build a working RAG pipeline in Python

The script below is deliberately small. It uses TF-IDF from scikit-learn for retrieval so you can run it on any laptop without a GPU, a vector database or an API key, and see every step. Once it works, you swap pieces out (see "Upgrading each step" below).

Setup: create a folder with a `docs/` subfolder holding a few `.txt` files (we used two short HR policy files, with paragraphs separated by blank lines), then run `pip install scikit-learn`.

"""A minimal, runnable RAG pipeline: load -> chunk -> index -> retrieve -> prompt -> generate -> evaluate.
pip install scikit-learn   (optional: pip install openai, then set OPENAI_API_KEY and MODEL to a current model
name from your provider's models page to call a real model)"""
import os, pathlib
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# 1. Load: read every .txt file in ./docs
def load_docs(folder="docs"):
    return {p.name: p.read_text(encoding="utf-8") for p in sorted(pathlib.Path(folder).glob("*.txt"))}

# 2. Chunk: split on blank lines (paragraphs) and keep the source name with each chunk
def chunk(docs):
    chunks = []
    for name, text in docs.items():
        for i, para in enumerate(p.strip() for p in text.split("\n\n")):
            if para:
                chunks.append({"id": f"{name}#{i+1}", "text": para})
    return chunks

# 3. Index: TF-IDF vectors here; swap in an embedding model + vector store later
def build_index(chunks):
    vec = TfidfVectorizer(stop_words="english")
    matrix = vec.fit_transform([c["text"] for c in chunks])
    return vec, matrix

# 4. Retrieve: top-k chunks above a minimum similarity score
def retrieve(question, chunks, vec, matrix, k=2, min_score=0.1):
    scores = cosine_similarity(vec.transform([question]), matrix)[0]
    ranked = sorted(zip(scores, chunks), key=lambda x: x[0], reverse=True)[:k]
    return [(round(float(s), 3), c) for s, c in ranked if s >= min_score]

# 5. Augment: build a grounded prompt that forces citations and allows "I don't know"
def build_prompt(question, hits):
    context = "\n".join(f"[{c['id']}] {c['text']}" for _, c in hits)
    return ("Answer using ONLY the context below. Cite chunk ids in square brackets. "
            "If the context does not contain the answer, say \"I don't know\".\n\n"
            f"Context:\n{context}\n\nQuestion: {question}\nAnswer:")

# 6. Generate: call a model if a key is set, otherwise show the prompt (useful for testing)
def generate(prompt):
    if not os.getenv("OPENAI_API_KEY"):
        return "(no API key set; prompt that would be sent)\n" + prompt
    from openai import OpenAI
    resp = OpenAI().responses.create(model=os.environ["MODEL"], input=prompt)
    return resp.output_text

def answer(question, chunks, vec, matrix):
    hits = retrieve(question, chunks, vec, matrix)
    if not hits:  # refuse before calling the model when retrieval finds nothing
        return "I don't know: nothing relevant was found in the documents.", hits
    return generate(build_prompt(question, hits)), hits

# 7. Evaluate retrieval: does the right chunk come back for each test question?
TESTS = [("How many days of leave can I carry forward?", "leave-policy.txt#1"),
         ("Do I need a medical certificate for sick leave?", "leave-policy.txt#2"),
         ("Can I fly business class?", "expense-policy.txt#2"),
         ("When must travel expenses be filed?", "expense-policy.txt#1")]

if __name__ == "__main__":
    chunks = chunk(load_docs())
    vec, matrix = build_index(chunks)
    hit = sum(any(c["id"] == want for _, c in retrieve(q, chunks, vec, matrix)) for q, want in TESTS)
    print(f"{len(chunks)} chunks indexed | retrieval hit rate: {hit}/{len(TESTS)}")
    for q in ["Can I fly business class?", "What is the office Wi-Fi password?"]:
        text, hits = answer(q, chunks, vec, matrix)
        print("\nQ:", q, "| retrieved:", [(s, c["id"]) for s, c in hits])
        print(text)

What you should see (output from our test run on 17 November 2026, with no API key set):

5 chunks indexed | retrieval hit rate: 4/4

Q: Can I fly business class? | retrieved: [(0.574, 'expense-policy.txt#2')]
(no API key set; prompt that would be sent)
Answer using ONLY the context below. Cite chunk ids in square brackets. If the context does not contain the answer, say "I don't know".
...
Q: What is the office Wi-Fi password? | retrieved: []
I don't know: nothing relevant was found in the documents.

To call a real model, `pip install openai`, set `OPENAI_API_KEY`, and set `MODEL` to a current model name from your provider's models page. Model names change and older ones get deprecated, which is why the script doesn't hard-code one.

What each step teaches you

  1. Load. Real projects spend more time here than anywhere else: scanned PDFs, tables, headers and footers repeated on every page. Garbage in, garbage retrieved.
  2. Chunk. Each chunk keeps an `id` with its source, so the answer can cite it. Chunks that are too large dilute the match; chunks that are too small lose meaning. Paragraphs or document sections are a sensible first unit.
  3. Index. TF-IDF matches words, not meaning. That's a feature while learning, because you can see exactly why a chunk matched, and a limitation in production, where users phrase things differently from your documents.
  4. Retrieve. Note the `min_score` threshold. Without it, the pipeline always returns something, and the model will happily answer from irrelevant text.
  5. Augment. The prompt sets three rules: use only the context, cite chunk ids, and say "I don't know" otherwise. Those three lines prevent a large share of beginner RAG failures.
  6. Generate. The model call is the smallest part of the code. Notice the pipeline refuses before calling the model when retrieval finds nothing, which saves cost and avoids invented answers.
  7. Evaluate. Four test questions are enough to prove the loop works and to catch a broken change instantly. Real projects need dozens of questions written by people who know the documents.

Upgrading each step for real-world RAG

  • Embeddings turn text into vectors that capture meaning. A common small open model, all-MiniLM-L6-v2, "maps sentences & paragraphs to a 384 dimensional dense vector space". The Sentence Transformers library makes it a few lines of code.
  • Vector stores hold those vectors and find the nearest ones fast. Options range from the FAISS library ("a library for efficient similarity search and clustering of dense vectors") to pgvector ("Open-source vector similarity search for Postgres"), which is a good fit if your data already lives in Postgres.
  • Hybrid search combines embeddings with keyword ranking such as BM25, "a bag-of-words retrieval function". Keywords catch exact terms such as product codes, error messages and names that embeddings can miss. pgvector's README shows hybrid search with Postgres full-text search.

Better retrieval: reranking and context

A reranker re-scores the top 20–50 candidates with a more precise (and slower) model and keeps the best few. Anthropic's write-up on Contextual Retrieval, which adds a short explanation of each chunk's place in its document before indexing, reports that the method "can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%" on their benchmarks (Anthropic). Your numbers will differ; the point is that retrieval quality, not the model, is usually the bottleneck. How you write the instructions around retrieved chunks, and how much context you include, is covered in prompt engineering vs context engineering.

Better chunking

  • Split on document structure (headings, sections, table rows) rather than a fixed character count where you can.
  • Store metadata with every chunk: source, page, date, department, access level. You'll need it for citations, filters and permissions.
  • Re-index when documents change, and delete chunks for removed documents, or the assistant will quote outdated policy.

Evaluation that goes beyond "looks right"

Measure retrieval and generation separately:

  • Retrieval: did the right chunk appear in the top-k (hit rate), and how high (rank)?
  • Generation: is the answer supported by the retrieved text, and does it answer the question?

Libraries such as Ragas provide metrics including Context Precision, Context Recall, Faithfulness and Response Relevancy. Keep a fixed test set and re-run it after every change to chunking, embeddings, prompts or models.

Security and access control

The OWASP Top 10 for LLM Applications 2025 includes LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure and LLM08 Vector and Embedding Weaknesses, all directly relevant to RAG. In practice:

  • Filter retrieval by the user's permissions before chunks reach the prompt.
  • Treat retrieved text as untrusted: a document can contain instructions aimed at the model.
  • Mask personal data you don't need, and log what was retrieved for each answer.

Should you use a framework (LangChain, LlamaIndex)?

Frameworks such as LangChain and LlamaIndex give you document loaders, splitters, vector store integrations and retrievers out of the box. They save time once you know what each step does. Build the plain-Python version above first; then you'll know which part of the framework to tune when answers go wrong.

When retrieval becomes one tool among several (search, SQL, calculators, APIs), you are moving from RAG to an agent. Our agentic AI roadmap shows where RAG fits in that path, and how to build an MCP server shows how to expose a retriever as a reusable tool.

Common RAG pipeline mistakes

  • No "I don't know" path. Without a score threshold and a refusal rule, the model answers from whatever it gets.
  • Judging by demo questions. Five questions you wrote yourself will always pass. Test with real user questions.
  • Only measuring the final answer. When an answer is wrong, you need to know whether retrieval or generation failed.
  • Losing metadata. No source or page means no citations, no filters and no permission checks.
  • Jumping to fine-tuning. Most "the model doesn't know our data" problems are retrieval problems.
  • Ignoring updates. A RAG system over stale documents gives confident, outdated answers.

RAG project ideas to practise

  1. Policy assistant: answers HR or travel policy questions with citations (extend the script above).
  2. Course syllabus Q&A: students ask about modules and deadlines; the bot says "I don't know" for anything off-syllabus.
  3. Support ticket helper: retrieves similar past tickets and drafts a reply for a human to approve.
  4. Hybrid search comparison: run the same 30 questions through keyword, embedding and hybrid retrieval and publish the hit rates in your README.

For more ideas organised by level, see agentic AI projects for beginners.

Learn to build RAG systems with a mentor

If you'd like structure, live sessions and code review while you build:

  • The Generative AI & Agentic AI Engineer program spends Weeks 4–7 on RAG systems: embeddings and vector databases, advanced RAG and evaluation, knowledge graphs and GraphRAG, and multimodal document intelligence, ending in a Knowledge Copilot project with hybrid RAG and role-based access.
  • The Applied Agentic AI Certification covers production RAG in Week 3, where you build a knowledge-base chatbot with hybrid search, reranking and RAGAS evaluation. No prior AI experience is required.
  • Already have a RAG prototype? LLMOps & AI Evaluation focuses on evaluating, observing and deploying it, including retrieval quality.
  • For teams, the same material is available as private cohorts through corporate training.

More free articles are on the Datamarcos guides hub.

FAQ

What are the steps in a RAG pipeline?

Load documents, split them into chunks with metadata, index the chunks (keyword search, embeddings or both), retrieve the most relevant chunks for a question, add them to a prompt with clear rules, generate an answer that cites them, and evaluate retrieval and answers on a fixed test set.

Can I build a RAG pipeline in Python without LangChain?

Yes. A basic pipeline needs only a way to split text, a way to search it (TF-IDF, BM25 or embeddings) and a model call. Building it in plain Python first helps you understand what frameworks such as LangChain or LlamaIndex do for you.

Do I need a vector database to build RAG?

Not to start. Keyword search or an in-memory index works for small collections. A vector store such as pgvector or FAISS becomes useful when you have many documents and want meaning-based search, usually combined with keyword search.

What is the best chunk size for RAG?

There is no single best size. Start with natural units such as paragraphs or sections, keep metadata with each chunk, and compare a few options on your own test questions. Measure retrieval hit rate rather than guessing.

How do I stop a RAG chatbot from hallucinating?

Set a minimum retrieval score, instruct the model to answer only from the provided context and to say "I don't know" otherwise, require citations, and check faithfulness on a test set. Better retrieval (hybrid search, reranking) reduces hallucinations more than prompt tweaks alone.

What is the difference between RAG and fine-tuning?

RAG gives the model new information at question time by retrieving documents; fine-tuning changes the model's behaviour by training it on examples. Use RAG for knowledge that changes or must be cited, and fine-tuning for consistent style or format. Our RAG vs fine-tuning guide covers this in detail.

How do I evaluate a RAG pipeline?

Build a test set of real questions with the source passage that should answer each one. Measure retrieval (is the right chunk in the top results?) separately from generation (is the answer supported and relevant?). Tools such as Ragas offer metrics like context precision, context recall and faithfulness.

Related program

Generative AI & Agentic AI Engineer program

18 weeks live online, 130+ live hours, three client-style projects and a capstone defended at Demo Day. Built for people who already code or work with data.

See the 18-week syllabus

Sources

Accessed 17 November 2026.

Contact Us