|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/few-shot

$cat docs/few-shot-learning.md

updated Recently·30 min read·published

Few-Shot Learning

◆AI◆Prompts◆Learning◆Intermediate

Introduction

Few-shot learning in the context of LLMs refers to the ability of a model to perform a task after being shown only a small number of input-output examples within the prompt. This is a form of in-context learning — the model never updates its weights; instead, it uses the examples as implicit instructions.

Few-shot prompting is one of the most powerful and reliable techniques in prompt engineering. A well-chosen set of examples can outperform meticulously crafted instructions, especially for nuanced tasks where rules are hard to articulate.

In-Context Learning

In-context learning (ICL) is the mechanism by which few-shot examples influence model behavior. Unlike fine-tuning (which updates model weights), ICL works entirely through the attention mechanism — examples in the prompt condition the model's next-token predictions by establishing patterns in the context window.

Method	Weight Updates	Examples Needed	Cost Per Task
Zero-Shot	None	0	Lowest
Few-Shot (ICL)	None	1-100	Low (per-token cost)
Fine-Tuning	All or LoRA	100-10K+	High (training cost)
RLHF / DPO	All	1K-100K+	Very High

zero-vs-few-shot.txt

TEXT

1	# Zero-shot: no examples, relies entirely on instructions
2	Classify the sentiment of this review: "The battery life is terrible."
3
4	# Few-shot: examples establish the pattern
5	Classify each review as Positive, Negative, or Neutral.
6
7	Review: "Best purchase I've ever made!"
8	Sentiment: Positive
9
10	Review: "It's okay, does what it says."
11	Sentiment: Neutral
12
13	Review: "Completely useless, broke in a day."
14	Sentiment: Negative
15
16	Review: "The battery life is terrible."
17	Sentiment:

ℹ

info

In-context learning works best when examples are diverse, representative of the task distribution, and presented consistently. The order of examples matters — models tend to be biased toward examples near the end of the prompt (recency bias).

K-Shot Examples

The "k" in k-shot refers to the number of examples provided. The optimal k varies by task, model size, and available context window. Understanding how performance scales with k helps you trade off cost against accuracy.

1-Shot (Single Example)

Even a single example dramatically improves output consistency. Use 1-shot when the task is simple but the format is non-obvious.

1-shot-date.txt

TEXT

1	Convert dates to ISO 8601 format.
2
3	Input: March 15, 2026
4	Output: 2026-03-15
5
6	Input: April 1st, 2025
7	Output:

2-5 Shot (Standard Range)

This is the sweet spot for most tasks. 2-5 examples provide enough signal without excessive token consumption.

few-shot-extraction.txt

TEXT

1	Extract structured data from product descriptions.
2
3	Description: "Sony WH-1000XM5 Wireless Noise Cancelling Headphones, 30hr battery, Black"
4	Product: {"brand": "Sony", "model": "WH-1000XM5", "type": "Headphones", "features": ["Wireless", "Noise Cancelling"], "battery": "30hr", "color": "Black"}
5
6	Description: "Apple MacBook Pro 16-inch M3 Max, 36GB RAM, 1TB SSD, Space Black"
7	Product: {"brand": "Apple", "model": "MacBook Pro 16-inch M3 Max", "type": "Laptop", "ram": "36GB", "storage": "1TB SSD", "color": "Space Black"}
8
9	Description: "Nike Air Force 1 '07 Men's Shoe, White/White, Size 10"
10	Product: {"brand": "Nike", "model": "Air Force 1 '07", "type": "Shoe", "gender": "Men's", "color": "White/White", "size": "10"}
11
12	Description: "Instant Pot Duo Plus 6-Quart 9-in-1 Pressure Cooker, Stainless Steel"
13	Product:

10+ Shot (Context-Intensive)

For complex tasks with many edge cases, 10-50+ examples may be needed. This consumes significant context but can approach fine-tuned performance.

dynamic-few-shot.py

Python

1	import openai
2
3	client = openai.OpenAI()
4
5	def build_few_shot_prompt(examples, query):
6	"""Build a few-shot prompt with dynamic example count."""
7	messages = [
8	{"role": "system", "content": "Classify email intent."},
9	]
10
11	for example in examples:
12	messages.append({
13	"role": "user",
14	"content": example["email"]
15	})
16	messages.append({
17	"role": "assistant",
18	"content": example["intent"]
19	})
20
21	messages.append({"role": "user", "content": query})
22
23	return messages
24
25
26	# With 15+ diverse examples, this approaches fine-tuned accuracy
27	examples = [
28	{"email": "Can you send me the invoice?", "intent": "billing_request"},
29	{"email": "I want to cancel my subscription.", "intent": "cancellation"},
30	# ... 13+ more diverse examples covering edge cases
31	]
32
33	messages = build_few_shot_prompt(examples, "Where is my refund?")
34	response = client.chat.completions.create(
35	model="gpt-4o",
36	messages=messages,
37	temperature=0,
38	)
39	print(response.choices[0].message.content)

✓

best practice

Monitor the relationship between k and accuracy on your validation set. Diminishing returns typically set in after 10-20 examples. If accuracy still climbs linearly at k=20, consider fine-tuning instead — you may need more examples than ICL can efficiently provide.

Example Selection Strategies

The choice of which examples to include is as important as how many. Random selection is rarely optimal. Several principled strategies exist for selecting the most effective examples.

Random Sampling

Simplest baseline. Works adequately when the task has low variance and examples are broadly representative.

Risk: may miss edge cases or over-represent common patterns.

Diversity Sampling

Select examples that are maximally different from each other, covering the full input space. Use clustering or embedding similarity to ensure coverage.

diversity-sampling.py

Python

1	from sentence_transformers import SentenceTransformer
2	from sklearn.metrics.pairwise import cosine_similarity
3	import numpy as np
4
5	model = SentenceTransformer('all-MiniLM-L6-v2')
6
7	def select_diverse_examples(pool, k=5):
8	"""Select k diverse examples using maximum marginal relevance."""
9	embeddings = model.encode(pool)
10	selected = []
11
12	# Pick the first example farthest from the centroid
13	centroid = np.mean(embeddings, axis=0)
14	distances = cosine_similarity(embeddings, centroid.reshape(1, -1))
15	selected_idx = np.argmin(distances)
16	selected.append(selected_idx)
17
18	# Iteratively pick examples most different from selected set
19	for _ in range(k - 1):
20	remaining = [i for i in range(len(pool)) if i not in selected]
21	max_min_sim = -1
22	best_idx = remaining[0]
23
24	for i in remaining:
25	sim_to_selected = max(
26	cosine_similarity(
27	embeddings[i].reshape(1, -1),
28	embeddings[j].reshape(1, -1)
29	)[0][0]
30	for j in selected
31	)
32	if sim_to_selected < max_min_sim:
33	max_min_sim = sim_to_selected
34	best_idx = i
35
36	selected.append(best_idx)
37
38	return [pool[i] for i in selected]

Dynamic (Test-Time) Selection

Select examples at inference time based on similarity to the current query. This adapts the prompt to each specific input, maximizing relevance.

dynamic-few-shot-selection.py

Python

1	import openai
2	import numpy as np
3	from openai import OpenAI
4
5	client = OpenAI()
6
7	def dynamic_few_shot(query, example_pool, k=3):
8	"""Select k examples most similar to the query."""
9	# Get embeddings for all examples and query
10	all_texts = [e["input"] for e in example_pool] + [query]
11	resp = client.embeddings.create(
12	model="text-embedding-3-small",
13	input=all_texts
14	)
15	embeddings = [d.embedding for d in resp.data]
16	example_embs = np.array(embeddings[:-1])
17	query_emb = np.array(embeddings[-1])
18
19	# Find nearest neighbors
20	similarities = example_embs @ query_emb
21	top_k = np.argsort(similarities)[-k:][::-1]
22
23	# Build prompt with selected examples
24	selected = [example_pool[i] for i in top_k]
25	messages = [{"role": "system", "content": "Classify intent."}]
26	for ex in selected:
27	messages.append({"role": "user", "content": ex["input"]})
28	messages.append({"role": "assistant", "content": ex["label"]})
29	messages.append({"role": "user", "content": query})
30
31	return messages
32
33
34	example_pool = [
35	{"input": "How do I reset my password?", "label": "account"},
36	{"input": "I was charged twice!", "label": "billing"},
37	{"input": "Your app keeps crashing", "label": "bug"},
38	# ... thousands more
39	]
40
41	query = "I forgot my login credentials"
42	messages = dynamic_few_shot(query, example_pool, k=3)
43	response = client.chat.completions.create(
44	model="gpt-4o", messages=messages, temperature=0
45	)

🔥

pro tip

Dynamic few-shot selection (also called "retrieval-augmented in-context learning") is the most effective strategy for large example pools. By retrieving the k most semantically similar examples for each query, you provide maximally relevant demonstrations while keeping the prompt compact. This is the same retrieval pattern used in RAG systems.

Impact on Output Quality

Few-shot examples influence multiple dimensions of output quality. Understanding these effects helps you design better example sets.

Dimension	Effect	Example Design Strategy
Format Compliance	Model mimics output format of examples	Ensure all examples have identical structure
Label Distribution	Model mirrors label frequency in examples	Balance label counts (or match real distribution)
Tone & Style	Model adopts tone and vocabulary of examples	Curate examples with desired voice
Error Patterns	Errors in examples propagate to outputs	Validate every example's correctness
Bias	Biases in examples amplify in outputs	Audit examples for fairness and representation

example-quality-comparison.txt

TEXT

1	# Poor examples — inconsistent format, incorrect label
2	Review: "This movie was amazing"
3	Sentiment: Positive
4
5	Review: "not good"
6	Sentiment: negative
7
8	Review: "It was okay I guess."
9	Sentiment: Neutral
10
11	# Good examples — consistent format, correct labels, diverse inputs
12	Review: "This movie was absolutely incredible, best of the year!"
13	Sentiment: Positive
14
15	Review: "The plot was confusing and the acting felt flat."
16	Sentiment: Negative
17
18	Review: "It was an average film, neither great nor terrible."
19	Sentiment: Neutral
20
21	Review: "Not bad, but not good either. Just okay."
22	Sentiment: Neutral

⚠

warning

The label distribution in your few-shot examples strongly biases the model's outputs. If 3 of 4 examples are "Positive," the model will over-predict "Positive." Either balance labels evenly or match the real-world distribution you expect in production.

Ordering & Recency Effects

The order of examples in a few-shot prompt significantly impacts performance. Models exhibit recency bias — examples placed near the end of the prompt have disproportionate influence.

Recency Bias

Examples closest to the query (at the end of the context) have the strongest influence. Place your most representative or important example last.

Primacy Bias

The first example also has outsized influence as it establishes the initial pattern. Place your cleanest, most unambiguous example first.

Optimal Ordering Strategy

Place the best "anchor" example first (primacy), fill the middle with diverse examples, and place an example highly similar to the expected query last (recency).

optimal-ordering.txt

TEXT

1	# Optimal ordering pattern:
2	# 1st: Strong anchor example (establishes pattern)
3	Input: "What is the capital of France?"
4	Output: "Paris"
5
6	# Middle: Diverse edge cases
7	Input: "How many people live in Tokyo?"
8	Output: "37.2 million"
9
10	Input: "What is the area of Vatican City?"
11	Output: "0.44 km²"
12
13	# Last: Similar to expected query (recency bias)
14	Input: "What is the tallest mountain?"
15	Output: "Mount Everest"
16
17	# Actual query
18	Input: "What is the longest river?"
19	Output:

Common Pitfalls

✗ Label Leakage

Including the answer format or label in the query portion of examples — the model learns to pattern-match rather than reason.

✗ Example Contamination

Using examples that overlap with the model's training data can cause memorization artifacts rather than genuine in-context learning.

✗ Token Waste

Including overly verbose examples eats context window. Each example should be as concise as possible while remaining representative.

✗ Ignoring the System Prompt

When few-shot examples contradict the system prompt, the model may follow examples over instructions. Keep them aligned.

✓

best practice

Always validate your few-shot examples on a held-out test set before deploying. A single bad example can silently degrade accuracy by 10-20%. Automate example selection and validation as part of your prompt management workflow.

$Blueprint — Engineering Documentation·Section ID: AI-05·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor