|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/chunking

$cat docs/chunking-strategies.md

updated Recently·30 min read·published

Chunking Strategies

◆Chunking◆RAG◆AI◆Intermediate

Why Chunking Matters

Chunking is the process of splitting documents into smaller pieces before embedding and indexing. It is one of the most impactful design decisions in a RAG system. The quality of retrieved context — and therefore the quality of generated answers — depends heavily on how documents are broken down.

A good chunking strategy balances competing goals: each chunk must be semantically self-contained, small enough for precise retrieval, yet large enough to provide sufficient context for the LLM. There is no one-size-fits-all approach; the optimal strategy depends on your document types, embedding model, and use case.

ℹ

info

Chunking is often the highest-leverage optimization you can make to a RAG pipeline. A well-chosen chunking strategy can improve retrieval accuracy by 20-40% with zero model changes.

Fixed-Size Chunking

The simplest approach: split documents into chunks of a fixed number of characters or tokens. This is fast, deterministic, and easy to implement. The chunk size is typically chosen based on the embedding model's context window (e.g., 256 or 512 tokens).

fixed_chunking.py

Python

1	def fixed_size_chunking(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
2	chunks = []
3	start = 0
4	while start < len(text):
5	end = min(start + chunk_size, len(text))
6	chunks.append(text[start:end])
7	start += chunk_size - overlap
8	return chunks
9
10	# Example
11	text = "Long document text..."
12	chunks = fixed_size_chunking(text, chunk_size=512, overlap=64)
13	print(f"Generated {len(chunks)} chunks")

📝

note

Fixed-size chunking at the token level is generally preferred over character-level chunking because embedding models operate on tokens. Libraries like tiktoken can count tokens accurately for a given model.

Semantic Chunking

Semantic chunking splits documents at natural boundaries — paragraphs, sections, or sentences — preserving semantic coherence. The goal is to ensure each chunk represents a complete idea or topic. This approach requires more computation but produces higher-quality chunks.

semantic_chunking.py

Python

1	from langchain.text_splitter import RecursiveCharacterTextSplitter
2	from langchain.text_splitter import SentenceTransformersTokenTextSplitter
3
4	# Recursive splitting: tries separators in order
5	splitter = RecursiveCharacterTextSplitter(
6	chunk_size=512,
7	chunk_overlap=64,
8	separators=[
9	"\n\n", # Paragraph breaks first
10	"\n", # Line breaks second
11	".", # Sentences third
12	" ", # Words fourth
13	"" # Characters last
14	]
15	)
16
17	semantic_chunks = splitter.split_text(document)
18
19	# Sentence-aware splitting
20	sentence_splitter = SentenceTransformersTokenTextSplitter(
21	chunk_size=256,
22	chunk_overlap=32,
23	)
24	sentence_chunks = sentence_splitter.split_text(document)

Embedding-Based Semantic Splitting

A more advanced approach uses embedding similarity between adjacent sentences to detect topic shifts. When the similarity drops below a threshold, a new chunk begins.

embedding_semantic_split.py

Python

1	def embedding_semantic_split(sentences: list[str], threshold: float = 0.7):
2	chunks = []
3	current_chunk = [sentences[0]]
4
5	for i in range(1, len(sentences)):
6	similarity = cosine_similarity(
7	embed(sentences[i-1]),
8	embed(sentences[i])
9	)
10	if similarity < threshold:
11	chunks.append(" ".join(current_chunk))
12	current_chunk = []
13	current_chunk.append(sentences[i])
14
15	chunks.append(" ".join(current_chunk))
16	return chunks

✓

best practice

Semantic chunking produces chunks that are more coherent and interpretable, but at a higher computational cost. For production, benchmark both fixed-size and semantic chunking on your data — the improvement varies significantly by domain and document structure.

Recursive Chunking

Recursive chunking attempts to split text at natural boundaries (paragraphs, sentences, words) and falls back to smaller separators when a chunk still exceeds the target size. This produces the most naturally readable chunks while respecting size constraints.

recursive_chunking.py

Python

1	def recursive_chunk(text: str, max_size: int,
2	separators: list[str] = None) -> list[str]:
3	if separators is None:
4	separators = ["\n\n", "\n", ".", " "]
5
6	def split(text: str, sep_idx: int) -> list[str]:
7	if len(text) <= max_size or sep_idx >= len(separators):
8	return [text]
9
10	parts = text.split(separators[sep_idx])
11	chunks = []
12	current = []
13
14	for part in parts:
15	candidate = separators[sep_idx].join(current + [part])
16	if len(candidate) <= max_size:
17	current.append(part)
18	else:
19	if current:
20	chunks.extend(
21	split(separators[sep_idx].join(current),
22	sep_idx + 1)
23	)
24	current = [part]
25
26	if current:
27	chunks.extend(split(
28	separators[sep_idx].join(current), sep_idx + 1
29	))
30	return chunks
31
32	return split(text, 0)

🔥

pro tip

LangChain's RecursiveCharacterTextSplitter is the most widely used implementation. It defaults to splitting on ["\n\n", "\n", " ", ""] — customize the separator list to match your document structure. For code, use language-aware separators; for Markdown, split on heading boundaries.

Document-Based Chunking

Many document formats have inherent structural boundaries that make natural chunking points: headings in Markdown or HTML, slides in presentations, pages in PDFs. Leveraging these structures produces chunks that align with the document's intended organization.

Markdown Headers

markdown_chunking.py

Python

1	import re
2
3	def markdown_chunking(md_text: str) -> list[dict]:
4	chunks = []
5	current_chunk = {"heading": None, "content": []}
6	heading_pattern = re.compile(r'^(#{1,6})s+(.+)$', re.MULTILINE)
7
8	for line in md_text.split("\n"):
9	match = heading_pattern.match(line)
10	if match:
11	if current_chunk["content"]:
12	chunks.append({
13	"heading": current_chunk["heading"],
14	"text": "\n".join(current_chunk["content"])
15	})
16	current_chunk = {
17	"heading": match.group(2),
18	"content": [line]
19	}
20	else:
21	current_chunk["content"].append(line)
22
23	if current_chunk["content"]:
24	chunks.append(current_chunk)
25	return chunks

HTML Structure

html_chunking.py

Python

1	from bs4 import BeautifulSoup
2
3	def html_chunking(html: str) -> list[dict]:
4	soup = BeautifulSoup(html, "html.parser")
5	chunks = []
6
7	for section in soup.find_all(["section", "article", "div"]):
8	heading = section.find(["h1", "h2", "h3", "h4"])
9	heading_text = heading.get_text() if heading else None
10	content = section.get_text(strip=True)
11
12	if len(content) > 50: # Skip empty sections
13	chunks.append({
14	"heading": heading_text,
15	"text": content
16	})
17
18	return chunks

⚠

warning

Document-based chunking is format-specific and can be brittle. A PDF parser may produce different output than a Markdown parser for the same semantic content. Always validate the extracted text before committing to a chunking strategy.

Chunk Overlap

Chunk overlap ensures that boundary information is not lost. When chunks are created from consecutive parts of a document, the tail of one chunk overlaps with the head of the next. This prevents relevant context from being split across chunk boundaries.

overlap.py

Python

1	# Visual representation of chunk overlap:
2	#
3	# Chunk 1: [--- 512 tokens ---]
4	# Chunk 2: [--- 512 tokens ---]
5	# Chunk 3: [--- 512 tokens ---]
6	#
7	# Overlap region: 64 tokens between consecutive chunks
8
9	def overlapping_chunks(text: str, size: int, overlap: int):
10	step = size - overlap
11	return [text[i:i + size]
12	for i in range(0, max(1, len(text) - size + 1), step)]

ℹ

info

A good rule of thumb is 10-15% overlap relative to chunk size. For 512-token chunks, use 64-80 tokens of overlap. Too much overlap creates redundant embeddings and increases storage costs; too little risks losing context at boundaries.

Chunk Size Considerations

Chunk size is the single most important parameter in your chunking strategy. It directly impacts retrieval precision, context quality, and system latency.

Chunk Size	Pros	Cons	Best For
Small (128-256 tokens)	High precision, fast retrieval, low embedding cost	Loses context, may miss relevant signals	Factoid Q&A, entity extraction
Medium (256-512 tokens)	Good balance of precision and context	Moderate storage and latency	General RAG, most use cases
Large (512-1024 tokens)	Rich context, fewer chunks to search	Lower precision, higher embedding cost	Summarization, complex reasoning
Very Large (1024+ tokens)	Maximum context, minimal chunk management	Poor retrieval precision, noisy results	Full-document retrieval, specific use cases

✓

best practice

Match your chunk size to your embedding model's context window. OpenAI's text-embedding-3-small supports 8192 tokens, but smaller chunks (256-512) typically perform better for retrieval because they are more semantically focused.

Impact on Retrieval Quality

Chunking strategy directly affects three key retrieval metrics: precision (are the results relevant?), recall (are all relevant results found?), and latency (how fast does retrieval complete?).

The Precision-Recall Tradeoff

Smaller chunks → Higher precision, lower recall. Each chunk is tightly focused, so retrieved results are highly relevant. But the model may miss information that spans multiple chunks.

Larger chunks → Lower precision, higher recall. Each chunk contains more information, increasing the chance of including relevant content. But irrelevant content within the chunk dilutes the signal.

Optimal strategy → Medium chunks with semantic boundaries and overlap, tuned on your specific dataset and queries.

evaluate_chunking.py

Python

1	# Evaluate chunking strategies
2	from ragas.metrics import context_precision, context_recall
3
4	strategies = [
5	("fixed_256", lambda t: fixed_chunking(t, 256, 32)),
6	("fixed_512", lambda t: fixed_chunking(t, 512, 64)),
7	("semantic", semantic_chunking),
8	("recursive", lambda t: recursive_split(t, 512, 64)),
9	]
10
11	for name, strategy in strategies:
12	chunks = strategy(example_doc)
13	# Index chunks, run evaluation queries
14	precision = evaluate_precision(chunks, queries)
15	recall = evaluate_recall(chunks, queries)
16	print(f"{name}: precision={precision:.3f}, recall={recall:.3f}")

🔥

pro tip

When evaluating chunking strategies, use a representative set of queries that reflect real user behavior. Synthetic queries generated from chunk content will bias toward good results — always validate with held-out or human-written queries.

$Blueprint — Engineering Documentation·Section ID: AI-CHUNKING·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor