|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/rag

$cat docs/rag-(retrieval-augmented-generation).md

updated Recently·35 min read·published

RAG (Retrieval-Augmented Generation)

◆RAG◆AI◆Intermediate to Advanced

What is RAG?

Retrieval-Augmented Generation (RAG) is an architecture that enhances large language models by retrieving relevant information from an external knowledge base before generating a response. Instead of relying solely on the model's parametric memory, RAG grounds generation in retrieved context, reducing hallucinations and enabling access to up-to-date or private information.

A RAG system combines two core components: a retriever that searches a corpus for relevant documents and a generator that produces an answer conditioned on both the query and the retrieved documents. This separation of concerns allows each component to be optimized independently.

ℹ

info

RAG effectively solves the problem of LLMs being limited to their training cutoff date. By retrieving fresh information at inference time, RAG systems can answer questions about recent events, proprietary data, or domain-specific knowledge the model was never trained on.

RAG Architecture Overview

A complete RAG system involves two distinct pipelines: the indexing pipeline (run offline to prepare the knowledge base) and the retrieval pipeline (run at query time to fetch and generate).

High-Level Flow

1. User submits a query

2. Query is embedded into a vector representation

3. Vector database performs similarity search

4. Top-k most relevant documents are retrieved

5. Retrieved documents are concatenated as context

6. LLM generates answer conditioned on query + context

📝

note

RAG is not a single algorithm but a family of techniques. The simplest variant is often called "naive RAG," while more sophisticated approaches add query rewriting, re-ranking, and iterative retrieval.

Indexing Pipeline

The indexing pipeline transforms raw documents into a searchable vector index. This is an offline process that runs whenever the knowledge base is updated.

Step 1: Ingestion

Raw documents are collected from various sources — PDFs, databases, web pages, APIs. Each document is extracted and normalized into plain text with associated metadata (source, date, author, etc.).

ingestion.py

Python

1	def ingest_documents(sources: list[DocumentSource]) -> list[RawDocument]:
2	documents = []
3	for source in sources:
4	raw_text = extract_text(source)
5	metadata = {
6	"source": source.url,
7	"date": source.last_modified,
8	"format": source.format,
9	}
10	documents.append(RawDocument(text=raw_text, metadata=metadata))
11	return documents

Step 2: Chunking

Documents are split into smaller chunks to fit within the embedding model's context window and to improve retrieval precision. Chunking strategies include fixed-size, semantic, and recursive splitting.

chunking.py

Python

1	from langchain.text_splitter import RecursiveCharacterTextSplitter
2
3	splitter = RecursiveCharacterTextSplitter(
4	chunk_size=512,
5	chunk_overlap=64,
6	separators=["\n\n", "\n", ".", " ", ""]
7	)
8
9	chunks = splitter.split_documents(raw_documents)

Step 3: Embedding

Each chunk is passed through an embedding model to produce a dense vector representation. Similar chunks produce similar vectors, enabling semantic search.

embedding.py

Python

1	from openai import OpenAI
2
3	client = OpenAI()
4	embedding_model = "text-embedding-3-small"
5
6	def embed_chunks(chunks: list[str]) -> list[list[float]]:
7	response = client.embeddings.create(
8	model=embedding_model,
9	input=chunks
10	)
11	return [e.embedding for e in response.data]

Step 4: Storing

Embeddings and their associated text chunks are stored in a vector database with an index structure for fast approximate nearest neighbor search.

storing.py

Python

1	import chromadb
2
3	client = chromadb.PersistentClient(path="./index")
4	collection = client.create_collection(
5	name="knowledge_base",
6	metadata={"hnsw:space": "cosine"}
7	)
8
9	collection.add(
10	ids=[str(i) for i in range(len(embeddings))],
11	embeddings=embeddings,
12	documents=chunks,
13	metadatas=metadatas
14	)

Retrieval Pipeline

At query time, the retrieval pipeline converts the user's question into a vector, searches the index, retrieves the most relevant chunks, and passes them to the LLM alongside the original query.

retrieval.py

Python

1	def rag_retrieve_and_generate(query: str, k: int = 5) -> str:
2	# Step 1: Embed the query
3	query_embedding = client.embeddings.create(
4	model="text-embedding-3-small",
5	input=[query]
6	).data[0].embedding
7
8	# Step 2: Search vector database
9	results = collection.query(
10	query_embeddings=[query_embedding],
11	n_results=k,
12	include=["documents", "metadatas"]
13	)
14
15	# Step 3: Build context from retrieved documents
16	context = "\n\n---\n\n".join(results["documents"][0])
17
18	# Step 4: Generate with context
19	response = client.chat.completions.create(
20	model="gpt-4o",
21	messages=[
22	{"role": "system", "content":
23	"Answer based on the provided context."},
24	{"role": "user", "content":
25	f"Context:\n{context}\n\nQuery: {query}"}
26	]
27	)
28
29	return response.choices[0].message.content

✓

best practice

The system prompt plays a critical role in RAG quality. Instruct the model to answer based solely on the provided context and to say "I don't know" if the context lacks the answer. This prevents the model from falling back to its parametric knowledge.

Naive RAG vs. Advanced RAG

The RAG ecosystem has evolved significantly. Understanding the progression from naive to advanced approaches helps you choose the right level of complexity.

Approach	Characteristics	When to Use
Naive RAG	Single-shot retrieve & generate, no query rewriting, no re-ranking	Simple Q&A, prototyping
Advanced RAG	Query rewriting, re-ranking, hybrid search, sliding window retrieval	Production systems, complex queries
Modular RAG	Pluggable modules: query routing, iterative retrieval, self-RAG, corrective RAG	Multi-step reasoning, high-accuracy needs
Agentic RAG	LLM agent decides when and what to retrieve, uses tools like web search and APIs	Complex multi-source research tasks

Advanced RAG Pipeline

advanced_rag.py

Python

1	def advanced_rag(query: str) -> str:
2	# Step 1: Query rewriting — improve the raw query
3	rewritten = rewrite_query(query)
4
5	# Step 2: Hybrid search — dense + sparse
6	dense_results = vector_search(rewritten, k=10)
7	sparse_results = bm25_search(rewritten, k=10)
8
9	# Step 3: Fusion — combine and deduplicate
10	fused = reciprocal_rank_fusion(
11	[dense_results, sparse_results]
12	)
13
14	# Step 4: Re-ranking — refine relevance
15	reranked = cross_encoder_rerank(query, fused, top_k=5)
16
17	# Step 5: Generate with ranked context
18	return generate(query, reranked)

🔥

pro tip

Start with naive RAG and measure your baseline. Only add complexity (rewriting, re-ranking, hybrid search) after identifying specific failure modes like poor first-retrieval quality or query ambiguity. Premature optimization adds latency and maintenance overhead.

RAG Evaluation

Evaluating a RAG system requires measuring both retrieval quality and generation quality. These are orthogonal dimensions that together determine the user experience.

Metric	What It Measures	Target
Hit Rate	Does the retrieved set contain at least one relevant document?	> 0.9
MRR	Mean reciprocal rank of the first relevant document	> 0.8
NDCG	Normalized discounted cumulative gain (position-aware relevance)	> 0.8
Answer Relevancy	How well does the answer address the query?	> 0.9
Faithfulness	Is the answer grounded in the retrieved context?	> 0.95
Context Precision	Are relevant documents ranked higher than irrelevant ones?	> 0.8

evaluation.py

Python

1	from ragas import evaluate
2	from ragas.metrics import (
3	faithfulness,
4	answer_relevancy,
5	context_precision,
6	context_recall
7	)
8
9	dataset = {
10	"question": ["What is RAG?"],
11	"answer": [response],
12	"contexts": [[retrieved_text]],
13	"ground_truth": [expected_answer]
14	}
15
16	scores = evaluate(dataset, metrics=[
17	faithfulness,
18	answer_relevancy,
19	context_precision,
20	context_recall
21	])
22
23	print(scores)
24	# {'faithfulness': 0.95, 'answer_relevancy': 0.92, ...}

⚠

warning

Ground truth evaluation is expensive and does not scale. For production monitoring, use LLM-as-a-judge metrics (like faithfulness and relevancy) that compare retrieved context to generated output without requiring human labels. These correlate well with human judgment when using a strong evaluator model.

Common Pitfalls

✗ Problem

Chunk size too large or too small

Large chunks dilute relevance; small chunks lose context. Tune based on your embedding model's context length.

✗ Problem

No metadata filtering

Without pre-filtering by source, date, or domain, retrieval can return irrelevant results. Use metadata filters alongside vector search.

✗ Problem

No prompt instruction to use context

Without explicit instruction to ground answers in retrieved context, the LLM may ignore it and use its parametric knowledge instead.

✗ Problem

Fixed k without relevance thresholds

Always returning k results regardless of relevance scores introduces noise. Use a relevance threshold to filter low-quality matches.

$Blueprint — Engineering Documentation·Section ID: AI-RAG·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor