|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/multimodal

$cat docs/multimodal.md

updated Recently·35 min read·published

Multimodal

◆AI◆Advanced

Introduction

Multimodal AI models process and generate content across multiple modalities — text, images, audio, and video. Unlike single-modality models (text-only LLMs), multimodal models can understand the visual world, listen to audio, and reason across different types of information simultaneously.

The multimodal landscape has evolved rapidly: GPT-4V and GPT-4o set the standard for vision-language understanding, Claude 3.5 Sonnet and Gemini 1.5 Pro extended context and vision capabilities, and Llama 3.2 Vision brought open-weight multimodal models to parity with proprietary ones. The trend is toward native multimodality where a single model handles all modalities without separate specialist components.

This guide covers multimodal model capabilities, API usage patterns, image and video understanding, audio processing, multimodal embeddings, cross-modal retrieval, and practical applications.

Vision-Language Models

Vision-language models (VLMs) accept images as input alongside text, enabling tasks like image description, visual question answering, document analysis, and UI understanding. The leading models support high-resolution images, multiple images, and in some cases, video frames.

Model	Vision	Max Image Inputs	Max Resolution
GPT-4o	Native vision	Unlimited	16K pixels (long edge)
Claude 3.5 Sonnet	Vision	Up to 20 images	8K pixels (long edge)
Gemini 1.5 Pro	Native multimodality	Up to 3,600 images	4K pixels
Llama 3.2 90B Vision	Cross-attention vision	Up to 10 images	4K pixels

Image Understanding with GPT-4o

gpt4-vision.py

Python

1	from openai import OpenAI
2	import base64
3
4	client = OpenAI()
5
6	def analyze_image(image_path: str, prompt: str) -> str:
7	with open(image_path, "rb") as f:
8	image_data = base64.b64encode(f.read()).decode()
9
10	response = client.chat.completions.create(
11	model="gpt-4o",
12	messages=[
13	{
14	"role": "user",
15	"content": [
16	{"type": "text", "text": prompt},
17	{
18	"type": "image_url",
19	"image_url": {
20	"url": f"data:image/jpeg;base64,{image_data}",
21	"detail": "high"
22	}
23	}
24	]
25	}
26	],
27	max_tokens=1024
28	)
29	return response.choices[0].message.content
30
31	# Usage
32	result = analyze_image(
33	"chart.png",
34	"Explain this chart. What are the key trends?"
35	)
36	print(result)

Claude 3.5 Vision

claude-vision.py

Python

1	import anthropic
2
3	client = anthropic.Anthropic()
4
5	def analyze_with_claude(image_path: str, prompt: str) -> str:
6	with open(image_path, "rb") as f:
7	image_data = f.read()
8
9	response = client.messages.create(
10	model="claude-3-5-sonnet-20241022",
11	max_tokens=1024,
12	messages=[
13	{
14	"role": "user",
15	"content": [
16	{
17	"type": "image",
18	"source": {
19	"type": "base64",
20	"media_type": "image/jpeg",
21	"data": base64.b64encode(image_data).decode()
22	}
23	},
24	{"type": "text", "text": prompt}
25	]
26	}
27	]
28	)
29	return response.content[0].text
30
31	# Claude excels at document analysis
32	result = analyze_with_claude(
33	"invoice.pdf",
34	"Extract the following: invoice number, date, total amount, vendor name"
35	)
36	print(result)

ℹ

info

When analyzing images, the "detail" parameter (or equivalent) controls the trade-off between understanding and cost. "high" detail creates more tokens and costs more but is necessary for small text, complex diagrams, or detailed scenes. "low" detail is sufficient for simple images, photographs, and general scene understanding.

Audio Processing

Multimodal models increasingly support audio input — both speech transcription and direct audio understanding. GPT-4o and Gemini 1.5 can process audio natively, enabling tasks like meeting transcription, sentiment analysis from voice tone, and audio content summarization.

audio-processing.py

Python

1	from openai import OpenAI
2
3	client = OpenAI()
4
5	# Speech-to-text transcription
6	def transcribe_audio(audio_path: str) -> str:
7	with open(audio_path, "rb") as f:
8	transcript = client.audio.transcriptions.create(
9	model="whisper-1",
10	file=f,
11	response_format="text"
12	)
13	return transcript
14
15	# Audio understanding with GPT-4o (native audio)
16	def understand_audio(audio_path: str, prompt: str) -> str:
17	with open(audio_path, "rb") as f:
18	audio_data = base64.b64encode(f.read()).decode()
19
20	response = client.chat.completions.create(
21	model="gpt-4o-audio-preview",
22	messages=[
23	{
24	"role": "user",
25	"content": [
26	{"type": "text", "text": prompt},
27	{
28	"type": "input_audio",
29	"input_audio": {
30	"data": audio_data,
31	"format": "wav"
32	}
33	}
34	]
35	}
36	]
37	)
38	return response.choices[0].message.content
39
40	# Gemini audio understanding
41	import google.generativeai as genai
42
43	genai.configure(api_key="GEMINI_API_KEY")
44	model = genai.GenerativeModel("gemini-1.5-pro")
45
46	def gemini_audio(audio_path: str, prompt: str) -> str:
47	audio_file = genai.upload_file(audio_path)
48	response = model.generate_content([prompt, audio_file])
49	return response.text

Video Analysis

Video analysis with multimodal models can be done by sampling frames at regular intervals and passing them as a sequence of images. Gemini 1.5 Pro can process entire videos natively, while GPT-4o and Claude require frame extraction first.

video-analysis.py

Python

1	import cv2
2	import base64
3
4	def extract_frames(video_path: str, num_frames: int = 10) -> list:
5	cap = cv2.VideoCapture(video_path)
6	total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
7	frame_indices = [
8	int(total_frames * i / num_frames)
9	for i in range(num_frames)
10	]
11	frames = []
12	for idx in frame_indices:
13	cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
14	ret, frame = cap.read()
15	if ret:
16	_, buffer = cv2.imencode(".jpg", frame)
17	frames.append(base64.b64encode(buffer).decode())
18	cap.release()
19	return frames
20
21	def analyze_video(video_path: str, prompt: str) -> str:
22	frames = extract_frames(video_path, num_frames=15)
23	content = [{"type": "text", "text": prompt}]
24	for frame in frames:
25	content.append({
26	"type": "image_url",
27	"image_url": {
28	"url": f"data:image/jpeg;base64,{frame}",
29	"detail": "low"
30	}
31	})
32
33	response = client.chat.completions.create(
34	model="gpt-4o",
35	messages=[{"role": "user", "content": content}],
36	max_tokens=1024
37	)
38	return response.choices[0].message.content
39
40	# Native video with Gemini
41	def gemini_analyze_video(video_path: str, prompt: str) -> str:
42	video_file = genai.upload_file(video_path)
43	# Gemini processes the video directly — no frame extraction needed
44	response = model.generate_content([prompt, video_file])
45	return response.text

✓

best practice

For video analysis, sample 1-2 frames per second for most use cases. Use "low" detail for frames to minimize token costs — the temporal reasoning across frames often matters more than fine-grained detail in any single frame. For action recognition, increase frame rate. For static scene understanding, fewer frames suffice.

Multimodal Embeddings & Cross-Modal Retrieval

Multimodal embeddings map different modalities (text, images, audio) into a shared vector space, enabling cross-modal retrieval — search images with text, find text with images, or match audio to text descriptions. This enables powerful RAG systems over multimodal data.

multimodal-embeddings.py

Python

1	from openai import OpenAI
2	import numpy as np
3
4	client = OpenAI()
5
6	class MultimodalEmbedding:
7	def __init__(self):
8	self.model = "text-embedding-3-small" # Also supports images via GPT-4o
9
10	def embed_text(self, text: str) -> np.ndarray:
11	response = client.embeddings.create(
12	model=self.model,
13	input=text
14	)
15	return np.array(response.data[0].embedding)
16
17	def embed_image(self, image_path: str) -> np.ndarray:
18	# Use GPT-4o to generate a text description, then embed
19	description = analyze_image(
20	image_path,
21	"Describe this image in detail for search indexing."
22	)
23	return self.embed_text(description)
24
25	def search(self, query: str, index: list, top_k: int = 5) -> list:
26	query_emb = self.embed_text(query)
27	similarities = []
28	for item in index:
29	sim = np.dot(query_emb, item["embedding"]) / (
30	np.linalg.norm(query_emb) * np.linalg.norm(item["embedding"])
31	)
32	similarities.append((sim, item))
33	similarities.sort(reverse=True, key=lambda x: x[0])
34	return [
35	{"content": item["content"], "score": float(sim)}
36	for sim, item in similarities[:top_k]
37	]
38
39	# CLIP-based multimodal search (open source)
40	import clip
41	import torch
42
43	class CLIPSearch:
44	def __init__(self):
45	self.model, self.preprocess = clip.load("ViT-L/14")
46	self.model.eval()
47
48	def embed_text(self, text: str) -> torch.Tensor:
49	tokens = clip.tokenize([text])
50	with torch.no_grad():
51	return self.model.encode_text(tokens)
52
53	def embed_image(self, image_path: str) -> torch.Tensor:
54	image = self.preprocess(Image.open(image_path)).unsqueeze(0)
55	with torch.no_grad():
56	return self.model.encode_image(image)

🔥

pro tip

For production multimodal search, use CLIP or SigLIP embeddings for zero-shot cross-modal retrieval, or fine-tune a multimodal embedding model on your domain data. Consider a two-stage pipeline: (1) CLIP for initial candidate retrieval, (2) GPT-4o or similar for re-ranking candidates with detailed visual understanding.

Practical Applications

Document Processing

Extract data from invoices, forms, receipts, and contracts. Multimodal models handle complex layouts, handwriting, and mixed text-image documents.

doc-processing.py

Python

1	def process_invoice(invoice_path: str) -> dict:
2	prompt = """Extract the following from this invoice:
3	- Invoice number
4	- Date
5	- Vendor name and address
6	- Line items (description, quantity, unit price, total)
7	- Subtotal, tax, total amount
8	- Payment terms
9	Return as JSON."""
10	result = analyze_image(invoice_path, prompt)
11	return json.loads(result)

UI / Screen Understanding

Analyze screenshots for testing, accessibility auditing, and automation. Multimodal models can identify UI elements, suggest improvements, and generate code.

ui-analysis.py

Python

1	def analyze_ui(screenshot_path: str) -> str:
2	prompt = """Analyze this UI screenshot. Identify:
3	1. Layout structure (header, nav, content, footer)
4	2. Interactive elements (buttons, forms, links)
5	3. Accessibility issues (color contrast, missing labels)
6	4. Suggestions for improvement"""
7	return analyze_image(screenshot_path, prompt)

$Blueprint — Engineering Documentation·Section ID: AI-MM-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor