|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/embeddings

$cat docs/embedding-models.md

updated Recently·30 min read·published

Embedding Models

◆Embeddings◆AI◆Models◆Intermediate

What are Embeddings?

Embeddings are dense vector representations of data — text, images, audio, or any modality — that capture semantic meaning in a high-dimensional space. The core idea is simple: similar items are placed close together in vector space, while dissimilar items are far apart.

A good embedding model transforms complex input into a fixed-size vector that preserves semantic relationships. For example, the embedding of "cat" will be closer to "kitten" than to "car". This property enables semantic search, clustering, classification, and retrieval-augmented generation.

ℹ

info

Think of embeddings as a universal translator: they convert any modality into a shared numeric language that machines can compare and reason about. The same vector space can contain text, images, and audio simultaneously if using a multimodal model.

What Embeddings Represent

An embedding vector is a point in a learned semantic space. Each dimension captures some latent feature — not directly interpretable by humans — that the model learned during training. The aggregate of all dimensions encodes meaning.

Properties of Good Embeddings

Semantic proximity — Similar concepts map to nearby vectors. "Dog" and "puppy" are closer than "dog" and "truck".

Directional meaning — Vector arithmetic works: king - man + woman ≈ queen.

Dimensional efficiency — Most information is captured in the first few principal components. Later dimensions encode finer-grained distinctions.

Robustness — Invariant to surface-level changes like synonyms, paraphrasing, and typos (to varying degrees depending on the model).

📝

note

The dimensions of an embedding vector are not individually meaningful. Unlike bag-of-words features where dimension 42 might correspond to the word "the," embedding dimensions are distributed representations — each input activates many dimensions in complex patterns.

Embedding Dimensions

The dimensionality of an embedding determines how much information it can capture. Higher dimensions provide more capacity but increase storage and compute costs. Modern embedding models offer a range of dimensionalities via a technique called Matryoshka Representation Learning.

Dimensions	Tradeoff	Common Models
256-384	Smallest, fastest, lowest storage — good for high-throughput, lower-accuracy needs	MiniLM, all-MiniLM-L6-v2
768-1024	Sweet spot — strong accuracy with reasonable storage and speed	OpenAI text-embedding-3-small, Cohere embed-english-v3
1536-3072	Highest accuracy, largest storage — for precision-critical applications	OpenAI text-embedding-3-large, Cohere embed-multilingual-v3

dimensions.py

Python

1	# OpenAI with dimensionality control (Matryoshka)
2	from openai import OpenAI
3
4	client = OpenAI()
5
6	# text-embedding-3-small: 1536 dims, can be reduced
7	response = client.embeddings.create(
8	model="text-embedding-3-small",
9	input="Hello, world!",
10	dimensions=512 # Trade accuracy for speed/storage
11	)
12
13	print(len(response.data[0].embedding)) # 512
14
15	# Full dimensionality
16	response = client.embeddings.create(
17	model="text-embedding-3-large",
18	input="Hello, world!",
19	dimensions=3072 # Maximum
20	)

🔥

pro tip

OpenAI's Matryoshka embeddings let you truncate the dimension at inference time without retraining. Use 256 dimensions for initial filtering, 1536 for final ranking. This two-stage approach dramatically reduces vector database costs while maintaining accuracy.

Text Embedding Models

Text embedding models convert natural language into vectors. They are the most widely used class of embedding models and form the foundation of most RAG systems.

OpenAI Embeddings

OpenAI offers two text embedding models: text-embedding-3-small (cost-effective) and text-embedding-3-large (highest accuracy). Both support Matryoshka dimensionality reduction.

openai_embeddings.py

Python

1	import openai
2
3	client = openai.OpenAI()
4
5	# Batch embedding for efficiency
6	texts = [
7	"RAG combines retrieval with generation",
8	"Vector databases enable semantic search",
9	"Embeddings capture semantic meaning"
10	]
11
12	response = client.embeddings.create(
13	model="text-embedding-3-small",
14	input=texts
15	)
16
17	embeddings = [e.embedding for e in response.data]
18	# Each embedding is a list of 1536 floats
19	# Cost: ~$0.02 per 1M tokens

Cohere Embeddings

Cohere offers specialized embedding models with distinct modes for search, classification, and clustering. They also provide multilingual and late-interaction models.

cohere_embeddings.py

Python

1	import cohere
2
3	co = cohere.Client(api_key="COHERE_API_KEY")
4
5	# Embed with search mode (optimized for RAG)
6	response = co.embed(
7	texts=["What is RAG?"],
8	model="embed-english-v3.0",
9	input_type="search_query", # or "search_document"
10	embedding_types=["float"]
11	)
12
13	# For document-side embedding
14	doc_response = co.embed(
15	texts=["Document text here..."],
16	model="embed-english-v3.0",
17	input_type="search_document",
18	embedding_types=["float"]
19	)

Voyage AI

Voyage provides domain-specific embedding models optimized for code, finance, law, and medical domains, often outperforming general-purpose models on specialized tasks.

voyage_embeddings.py

Python

1	import voyageai
2
3	vo = voyageai.Client(api_key="VOYAGE_API_KEY")
4
5	# Domain-optimized embeddings
6	response = vo.embed(
7	texts=["def fibonacci(n): return n if n <= 1 else ..."],
8	model="voyage-code-2", # Code-optimized model
9	input_type="document"
10	)
11
12	# Voyage 3 for general text
13	response = vo.embed(
14	texts=["Semantic search with embeddings"],
15	model="voyage-3",
16	input_type="query"
17	)

Multimodal Embeddings

Multimodal embedding models map different data types — text, images, audio — into a shared vector space. This enables searching images by text descriptions, finding similar audio clips, or cross-modal retrieval.

multimodal.py

Python

1	# CLIP: Contrastive Language-Image Pre-training
2	import clip
3	import torch
4
5	model, preprocess = clip.load("ViT-B/32")
6
7	# Encode text and images into shared space
8	text_features = model.encode_text(
9	clip.tokenize(["A cat on a mat", "A dog playing fetch"])
10	)
11
12	image = preprocess(Image.open("cat.jpg")).unsqueeze(0)
13	image_features = model.encode_image(image)
14
15	# Compare: which text best matches the image?
16	similarity = (text_features @ image_features.T).softmax(dim=-1)
17
18	# Yes, "A cat on a mat" has highest similarity score

✓

best practice

Multimodal embeddings are powerful but computationally expensive. For most RAG systems, stick to text-only embeddings. Use multimodal embeddings when your use case requires searching across modalities — for example, finding product images from text descriptions.

Embedding Quality Evaluation (MTEB)

The Massive Text Embedding Benchmark (MTEB) is the standard benchmark for evaluating embedding models. It covers 8 tasks across 58 datasets, providing a comprehensive view of model quality.

Task	Description	Metric
Classification	Label text into categories	Accuracy / F1
Clustering	Group similar texts	V-Measure
Pair Classification	Predict if two texts are similar	AP / F1
Reranking	Order texts by relevance to query	MAP / MRR
Retrieval	Find relevant documents for a query	NDCG@10 / Recall@100
Semantic Similarity (STS)	Score similarity between sentence pairs	Spearman / Pearson
Summarization	Score summary quality against source	Spearman
Bitext Mining	Find parallel sentences across languages	Accuracy@1

ℹ

info

Always check the MTEB leaderboard when choosing an embedding model. Pay attention to the retrieval column — that is the most relevant metric for RAG. The top models change frequently; as of late 2025, Cohere Embed v3 and Voyage 3 lead for retrieval.

Fine-Tuning Embeddings

For domain-specific applications, fine-tuning an embedding model on your data can significantly improve retrieval quality. The most common approach uses contrastive learning with pairs of similar and dissimilar documents.

finetune_embeddings.py

Python

1	# Fine-tuning with sentence-transformers
2	from sentence_transformers import (
3	SentenceTransformer,
4	losses,
5	InputExample
6	)
7	from torch.utils.data import DataLoader
8
9	model = SentenceTransformer("all-MiniLM-L6-v2")
10
11	# Training data: (anchor, positive, negative) triplets
12	train_data = [
13	InputExample(
14	texts=[
15	"What is RAG?",
16	"RAG combines retrieval with generation",
17	"The weather is nice today"
18	],
19	label=1.0 # Similarity score (not used in TripletLoss)
20	),
21	# More examples...
22	]
23
24	train_dataloader = DataLoader(train_data, shuffle=True, batch_size=16)
25	train_loss = losses.TripletLoss(model)
26
27	model.fit(
28	train_objectives=[(train_dataloader, train_loss)],
29	epochs=3,
30	warmup_steps=100,
31	output_path="./fine-tuned-embedding"
32	)

🔥

pro tip

Fine-tuning is most effective when your domain has specialized vocabulary that general models handle poorly (medical terms, legal jargon, code). Collect 1000+ high-quality (query, relevant_doc, irrelevant_doc) triplets from your own user data. Just 500-1000 examples can yield a 5-15% improvement in retrieval NDCG.

$Blueprint — Engineering Documentation·Section ID: AI-EMBEDDINGS·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor