|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/tokenization

$cat docs/tokenization-&-embeddings.md

updated Recently·35 min read·published

Tokenization & Embeddings

◆AI Engineering◆Tokenization◆Embeddings◆Intermediate

Introduction

Tokenization and embeddings are the two fundamental preprocessing steps that enable LLMs to work with text. Tokenization converts raw text into discrete units (tokens) that the model can process, while embeddings convert those tokens into dense vector representations that capture semantic meaning.

Understanding these concepts is critical for managing context windows, estimating costs, optimizing RAG pipelines, and debugging unexpected model behavior. This guide covers the major tokenization algorithms, embedding models, and practical tools for working with them.

ℹ

info

Token count directly determines your cost (most LLM APIs charge per token), your context window usage, and your model's ability to process information. Mastering tokenization is one of the highest-leverage skills in AI engineering.

What Are Tokens?

Tokens are the atomic units that LLMs process. They are not words, characters, or bytes — they are subword units determined by the tokenizer's vocabulary. A single token might represent a whole word ("hello"), a subword ("ing"), a single character ("a"), or even a whitespace sequence.

The tokenizer converts input text into a sequence of token IDs (integers) that index into the model's embedding table. The model processes these IDs through its transformer layers. At output, the model predicts token IDs which the tokenizer converts back to text.

tokenization-basics.py

Python

1	# Tokenization visualized
2	text = "The quick brown fox jumps over the lazy dog."
3
4	# With tiktoken (OpenAI's tokenizer)
5	import tiktoken
6	enc = tiktoken.get_encoding("cl100k_base")
7	tokens = enc.encode(text)
8
9	print(f"Text: {text}")
10	print(f"Token IDs: {tokens}")
11	print(f"Tokens: {[enc.decode([t]) for t in tokens]}")
12	print(f"Token count: {len(tokens)}")
13
14	# Output:
15	# Text: The quick brown fox jumps over the lazy dog.
16	# Token IDs: [791, 4062, 14198, 10246, 31373, 1110, 279, 16084, 4106, 13]
17	# Tokens: ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog', '.']
18	# Token count: 10

⚠

warning

Never rely on word count or character count to estimate tokens. A rough rule of thumb: 1 token ~= 0.75 words in English, but this varies significantly by language. For example, "Hello" is 1 token, but "Héllö" might be 3 tokens due to Unicode encoding.

Tokenization Strategies

Different model families use different tokenization algorithms. The three dominant approaches are BPE, WordPiece, and SentencePiece (Unigram). Each has trade-offs in vocabulary size, compression efficiency, and language coverage.

Byte-Pair Encoding (BPE)

BPE is the most widely used tokenization algorithm, employed by GPT models, Llama, and many others. It starts with individual bytes as tokens and iteratively merges the most frequent adjacent pairs. The result is a vocabulary where common words become single tokens while rare words are split into multiple subword tokens. BPE is data-driven — the merges are learned from the training corpus.

bpe-training.py

Python

1	# BPE training algorithm (simplified)
2	from collections import Counter
3
4	def train_bpe(texts, vocab_size=1000):
5	# Start with character vocabulary
6	word_freqs = Counter()
7	for text in texts:
8	words = text.split()
9	for word in words:
10	word_freqs[word] += 1
11
12	# Split each word into characters
13	splits = {word: list(word) + ['</w>']
14	for word in word_freqs}
15
16	# Iteratively merge most frequent pairs
17	vocab = set()
18	for char in set(''.join(word for word in word_freqs)):
19	vocab.add(char)
20
21	while len(vocab) < vocab_size:
22	# Count adjacent pairs
23	pair_freqs = Counter()
24	for word, freq in word_freqs.items():
25	split = splits[word]
26	for i in range(len(split) - 1):
27	pair_freqs[(split[i], split[i+1])] += freq
28
29	if not pair_freqs:
30	break
31
32	# Merge the most frequent pair
33	best_pair = max(pair_freqs, key=pair_freqs.get)
34	merged = ''.join(best_pair)
35	vocab.add(merged)
36
37	# Update splits
38	for word in splits:
39	new_split = []
40	i = 0
41	while i < len(splits[word]):
42	if (i < len(splits[word]) - 1 and
43	(splits[word][i], splits[word][i+1]) == best_pair):
44	new_split.append(merged)
45	i += 2
46	else:
47	new_split.append(splits[word][i])
48	i += 1
49	splits[word] = new_split
50
51	return vocab

WordPiece

WordPiece is used by BERT and some other Transformer models. Unlike BPE which merges based on frequency, WordPiece merges based on likelihood — it selects the pair that maximizes the likelihood of the training data. WordPiece also uses a special "##" prefix to denote subword tokens that are continuations rather than word-initial.

wordpiece-example.py

Python

1	# WordPiece tokenization example
2	from transformers import BertTokenizer
3
4	tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
5	text = "Tokenization is fascinating!"
6
7	tokens = tokenizer.tokenize(text)
8	print(tokens)
9	# Output: ['token', '##ization', 'is', 'fascinating', '!']
10
11	# Note the "##" prefix on continuation tokens
12	# 'tokenization' was split into 'token' + '##ization'
13
14	# Token IDs
15	ids = tokenizer.encode(text)
16	print(ids)
17	# Output: [101, 19204, 12339, 2003, 6202, 999, 102]
18	# (101 = [CLS], 102 = [SEP])

SentencePiece & Unigram

SentencePiece is a language-independent tokenizer that treats the input as a raw byte stream, removing the need for pre-tokenization (splitting on whitespace). It supports both BPE and Unigram algorithms. Unigram starts with a large vocabulary and iteratively removes tokens that minimize the loss. Llama, Gemma, and many newer models use SentencePiece.

sentencepiece-example.py

Python

1	# SentencePiece usage
2	import sentencepiece as spm
3
4	# Train a SentencePiece model
5	spm.SentencePieceTrainer.Train(
6	input='corpus.txt',
7	model_prefix='mymodel',
8	vocab_size=16000,
9	model_type='bpe', # or 'unigram'
10	character_coverage=1.0,
11	normalization_rule_name='nmt_nfkc'
12	)
13
14	# Load and use
15	sp = spm.SentencePieceProcessor()
16	sp.Load('mymodel.model')
17
18	text = "Learning tokenization is key."
19	encoded = sp.EncodeAsIds(text)
20	decoded = sp.DecodeIds(encoded)
21	pieces = sp.EncodeAsPieces(text)
22
23	print(f"IDs: {encoded}")
24	print(f"Pieces: {pieces}")
25	print(f"Decoded: {decoded}")
26
27	# SentencePiece handles all Unicode transparently
28	japanese = "トークン化は面白い"
29	print(f"JP Pieces: {sp.EncodeAsPieces(japanese)}")

Algorithm	Used By	Pre-Tokenization	Merge Criterion	Special Tokens
BPE	GPT-4, Llama, Mistral	Whitespace-based	Frequency	None
WordPiece	BERT, DistilBERT	Whitespace-based	Likelihood	## prefix
Unigram	Gemma, T5, XLNet	None (raw bytes)	Loss minimization	▁ (underscore)
SentencePiece BPE	Llama 2/3, Gemma	None (raw bytes)	Frequency	▁ (underscore)

🔥

pro tip

Different models use different tokenizers. Always use the correct tokenizer for your model — mismatched tokenizers produce garbage. OpenAI models use cl100k_base (GPT-4, GPT-3.5), p50k_base (code models), or r50k_base (older models). Use tiktoken for OpenAI, transformers.AutoTokenizer for HuggingFace models.

Token Counting & Costs

Accurate token counting is essential for cost estimation, context window management, and prompt optimization. Every LLM API has a maximum context window (the total tokens in the prompt plus the generated response).

token-cost-estimator.py

Python

1	# Token counting utilities
2	import tiktoken
3
4	def count_tokens(text: str, model: str = "gpt-4o") -> int:
5	"""Count tokens for any OpenAI model."""
6	enc = tiktoken.encoding_for_model(model)
7	return len(enc.encode(text))
8
9	def estimate_cost(prompt_tokens: int, output_tokens: int, model: str) -> float:
10	"""Estimate API cost for a completion."""
11	pricing = {
12	"gpt-4o": (0.00250, 0.01000), # input, output per 1K
13	"gpt-4o-mini": (0.00015, 0.00060),
14	"gpt-4-turbo": (0.01000, 0.03000),
15	"claude-3-5-sonnet":(0.00300, 0.01500),
16	}
17	input_rate, output_rate = pricing.get(model, (0, 0))
18	cost = (prompt_tokens / 1000 * input_rate +
19	output_tokens / 1000 * output_rate)
20	return round(cost, 6)
21
22	# Example usage
23	prompt = "Write a 500-word essay on tokenization."
24	prompt_tokens = count_tokens(prompt)
25	# Typically ~7-10 tokens for this prompt
26
27	# Estimate 500 words ~= 667 tokens output
28	cost = estimate_cost(prompt_tokens, 667, "gpt-4o")
29	print(f"Prompt: {prompt_tokens} tokens")
30	print(f"Estimated cost: ${cost}")

Model	Tokenizer	Vocab Size	Max Context	Tokens per Word (EN)
GPT-4o	cl100k_base	100K	128K	~1.3
Claude 3	Custom BPE	~80K	200K	~1.4
Llama 3	SentencePiece (BPE)	128K	128K	~1.2
Gemini 2.0	SentencePiece	~256K	1M	~1.1
BERT	WordPiece	30K	512	~1.4

✓

best practice

Always truncate prompts to stay within the model's context window. A common pattern is to reserve ~20% of the context window for the response. For GPT-4o (128K context), aim for prompts of at most ~100K tokens, leaving 28K for generation. Use the tokenizer to check before sending.

Embeddings & Semantic Meaning

Embeddings convert tokens (or entire texts) into dense numerical vectors — typically 256 to 4096 floating-point numbers. These vectors capture semantic meaning: texts with similar meaning produce vectors that are close together in the embedding space, regardless of exact word choice.

This property makes embeddings the foundation of semantic search, clustering, recommendation systems, and RAG pipelines. Instead of matching keywords, you measure the distance between vector representations of queries and documents.

semantic-similarity.py

Python

1	# Semantic similarity with embeddings
2	from openai import OpenAI
3	import numpy as np
4
5	client = OpenAI()
6
7	def get_embedding(text: str, model="text-embedding-3-small"):
8	response = client.embeddings.create(
9	model=model,
10	input=text
11	)
12	return np.array(response.data[0].embedding)
13
14	def cosine_similarity(a, b):
15	return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
16
17	# Compare semantic similarity
18	texts = [
19	"The cat sat on the mat",
20	"A feline rested on the rug",
21	"The stock market rallied today",
22	"Quantum computing uses qubits",
23	]
24
25	embeddings = [get_embedding(t) for t in texts]
26
27	# Similarity matrix
28	for i, t1 in enumerate(texts):
29	for j, t2 in enumerate(texts):
30	if i < j:
31	sim = cosine_similarity(embeddings[i], embeddings[j])
32	print(f"[{sim:.3f}] {t1[:20]}... <-> {t2[:20]}...")
33
34	# Output:
35	# [0.921] The cat sat on the mat... <-> A feline rested on...
36	# [0.342] The cat sat on the mat... <-> The stock market...
37	# [0.289] The cat sat on the mat... <-> Quantum computing...
38	# [0.314] A feline rested on... <-> The stock market...
39	# ...

preview

📝

note

Embedding models are distinct from generative models. OpenAI's text-embedding-3-small is a dedicated embedding model (it outputs vectors, not text). You can use different embedding models for retrieval than the model you use for generation — this is common and often optimal.

Embedding Models

Choosing the right embedding model impacts retrieval quality, storage requirements, and latency. Here are the major options:

Model	Dimensions	Max Input	Pricing (per 1K tokens)	Best For
text-embedding-3-small	512-1536	8K tokens	$0.00002	General purpose, cost-sensitive
text-embedding-3-large	256-3072	8K tokens	$0.00013	High-accuracy retrieval
Cohere Embed v3	1024 or 384	512 tokens	$0.00010	Classification, search
BGE (BAAI)	1024	512 tokens	Free (open-source)	Self-hosted retrieval
E5 (Microsoft)	1024	512 tokens	Free (open-source)	Multilingual retrieval
Jina Embeddings	768 or 1024	8K tokens	Free (open-source)	Long document retrieval

Dimensions & Performance Trade-offs

OpenAI's embedding models support a dimensions parameter that lets you truncate the output vector. This is a practical technique for reducing storage and computation costs while maintaining most of the retrieval quality.

embedding-dimensions.py

Python

1	# Dimensionality reduction with OpenAI embeddings
2	from openai import OpenAI
3
4	client = OpenAI()
5
6	# text-embedding-3-small supports 512-1536 dimensions
7	# text-embedding-3-large supports 256-3072 dimensions
8
9	# Full 1536 dimensions
10	full = client.embeddings.create(
11	model="text-embedding-3-small",
12	input="Hello world"
13	).data[0].embedding
14	print(f"Full dimension: {len(full)}") # 1536
15
16	# Truncated to 256 dimensions (faster, cheaper storage)
17	small = client.embeddings.create(
18	model="text-embedding-3-small",
19	input="Hello world",
20	dimensions=256
21	).data[0].embedding
22	print(f"Reduced dimension: {len(small)}") # 256
23
24	# MTEB benchmark scores:
25	# 1536 dims: 62.3%
26	# 512 dims: 61.8% (-0.5%)
27	# 256 dims: 61.0% (-1.3%)
28	# Rule of thumb: 512 dims captures ~99% of quality

ℹ

info

Use 256 or 512 dimensions for most applications. The quality drop from reducing 1536 to 256 is typically less than 2% on retrieval benchmarks, but storage costs drop by 6x and retrieval latency improves significantly.

Open-Source Embedding Models

For self-hosted deployments, open-source embedding models from Sentence Transformers provide competitive performance with no API costs.

open-source-embeddings.py

Python

1	# Self-hosted embeddings with Sentence Transformers
2	from sentence_transformers import SentenceTransformer
3
4	# Load model (downloads on first use)
5	model = SentenceTransformer("BAAI/bge-small-en-v1.5")
6
7	texts = [
8	"How to implement RAG pipeline",
9	"Building retrieval augmented generation",
10	"Stock market analysis for beginners",
11	]
12
13	embeddings = model.encode(texts, normalize_embeddings=True)
14
15	print(f"Embedding shape: {embeddings.shape}") # (3, 384)
16	print(f"Similarity: {embeddings[0] @ embeddings[1]:.3f}") # High
17	print(f"Similarity: {embeddings[0] @ embeddings[2]:.3f}") # Low
18
19	# For multilingual use:
20	model = SentenceTransformer("intfloat/multilingual-e5-small")
21	texts = [
22	"Hello, how are you?",
23	"Bonjour, comment allez-vous ?",
24	"你好，你好吗？",
25	]
26	embeddings = model.encode(texts)
27	# All three will be close in embedding space

Practical Tokenization Tools

Here are the essential libraries and tools for working with tokenization in production AI systems.

tiktoken — OpenAI Tokenizer

OpenAI's official tokenizer library. Fast (Rust-based), supports all OpenAI models, and provides encoding/decoding with proper handling of special tokens.

tiktoken-usage.py

Python

1	# Comprehensive tiktoken usage
2	import tiktoken
3
4	# Get encoder for specific model
5	enc = tiktoken.encoding_for_model("gpt-4o")
6
7	# Encode and decode
8	text = "Hello, how are you?"
9	tokens = enc.encode(text)
10	decoded = enc.decode(tokens)
11
12	print(f"Tokens: {tokens}")
13	print(f"Decoded: {decoded}")
14
15	# Token-by-token decoding (for visualization)
16	for token_id in tokens:
17	token_bytes = enc.decode_single_token_bytes(token_id)
18	print(f" {token_id:6d} -> {token_bytes}")
19
20	# Count tokens without decoding
21	print(f"Token count: {enc.encode(text, disallowed_special=())}")
22
23	# Working with chat messages
24	def count_chat_tokens(messages, model="gpt-4o"):
25	enc = tiktoken.encoding_for_model(model)
26	tokens_per_message = 3 # role + content + formatting
27	tokens = 0
28	for msg in messages:
29	tokens += tokens_per_message
30	tokens += len(enc.encode(msg["content"]))
31	tokens += 3 # assistant prefix
32	return tokens
33
34	messages = [
35	{"role": "system", "content": "You are a helpful assistant."},
36	{"role": "user", "content": "Hello!"},
37	]
38	print(f"Chat tokens: {count_chat_tokens(messages)}")

HuggingFace Tokenizers

The transformers library provides tokenizers for thousands of models with a unified API. It is the standard tool for working with open-source models.

hf-tokenizers.py

Python

1	# Unified tokenizers with HuggingFace
2	from transformers import AutoTokenizer
3
4	# Auto-detects correct tokenizer
5	tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
6
7	text = "Tokenization is the foundation of LLMs."
8
9	# Basic usage
10	tokens = tokenizer.tokenize(text)
11	ids = tokenizer.encode(text)
12	decoded = tokenizer.decode(ids)
13
14	print(f"Tokens: {tokens}")
15	print(f"IDs: {ids}")
16	print(f"Decoded: '{decoded}'")
17
18	# With special tokens
19	chat = [
20	{"role": "user", "content": "Hello!"}
21	]
22	prompt = tokenizer.apply_chat_template(
23	chat, tokenize=False
24	)
25	print(f"Chat template: {prompt}")
26
27	# Padding and truncation for batch processing
28	batch = tokenizer(
29	["Short text", "A much longer text here for testing"],
30	padding=True,
31	truncation=True,
32	max_length=512,
33	return_tensors="pt"
34	)
35	print(f"Input IDs shape: {batch['input_ids'].shape}")
36	print(f"Attention mask: {batch['attention_mask']}")

🔥

pro tip

For production RAG systems, pre-tokenize and cache all documents. Store the token count as metadata alongside each chunk. This allows you to estimate context window usage without re-tokenizing, and enables smart chunking strategies like dynamic chunk merging based on available context.

Context Windows

The context window is the maximum number of tokens a model can process in a single forward pass. This includes both the input prompt and the generated output. Context windows have grown from 512 tokens (BERT era) to over 1 million tokens (Gemini 2.0), enabling entirely new use cases.

However, all models degrade in quality as the context window fills. The "lost in the middle" phenomenon (Liu et al., 2024) shows that models pay less attention to information in the middle of long contexts. For RAG applications, this means you should put the most relevant documents at the beginning and end of the prompt.

context-window-strategy.py

Python

1	# Context window management strategies
2
3	def build_prompt_with_context(
4	question: str,
5	documents: list[str],
6	max_context: int = 100_000,
7	reserve_output: int = 20_000
8	):
9	"""
10	Build a prompt that respects context window limits.
11	Puts most relevant docs at start and end.
12	"""
13	max_prompt = max_context - reserve_output
14	available = max_prompt - len(question)
15
16	# Rank documents by relevance (already sorted)
17	prompt = ""
18	used = 0
19
20	# Add from both ends
21	first_half = []
22	second_half = []
23
24	for i, doc in enumerate(documents):
25	doc_len = len(doc)
26	if used + doc_len <= available:
27	if i % 2 == 0:
28	first_half.append(doc)
29	else:
30	second_half.append(doc)
31	used += doc_len
32
33	# Assemble: most relevant at ends
34	context = "\n".join(first_half)
35	context += "\n" + "\n".join(reversed(second_half))
36
37	return f"Context:\n{context}\n\nQuestion: {question}"
38
39	# Usage
40	docs = ["Document 1...", "Document 2...", ...]
41	prompt = build_prompt_with_context(
42	"What is RAG?",
43	docs,
44	max_context=128_000,
45	reserve_output=20_000
46	)

⚠

warning

Do not assume you can reliably use the full context window. Model performance degrades on long contexts — retrieval accuracy, instruction following, and factual recall all suffer as the prompt approaches the context limit. For critical applications, stay within 75% of the stated context window.

$Blueprint — Engineering Documentation·Section ID: AI-03·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor