|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/structured-outputs

$cat docs/structured-outputs.md

updated Recently·35 min read·published

Structured Outputs

◆AI◆Prompts◆Engineering◆Advanced

Introduction

Structured outputs are the practice of constraining LLM outputs to follow a predefined schema — typically JSON, but also XML, YAML, or custom formats. This transforms LLMs from free-text generators into reliable data extraction and transformation engines.

For production systems, unstructured text outputs are a liability. Structured outputs enable deterministic parsing, type-safe integration, validation, and error handling. They are the bridge between probabilistic LLM outputs and deterministic software systems.

JSON Mode

JSON mode instructs the model to return valid JSON that matches a specified schema. Most major LLM providers support this as a built-in parameter or via structured prompting.

json-mode.py

Python

1	import openai
2	from openai import OpenAI
3
4	client = OpenAI()
5
6	# JSON mode via response_format parameter (OpenAI)
7	response = client.chat.completions.create(
8	model="gpt-4o",
9	messages=[
10	{"role": "system", "content": "Extract user information."},
11	{"role": "user", "content": "John Doe, 30, john@example.com"}
12	],
13	response_format={"type": "json_object"},
14	temperature=0,
15	)
16
17	result = response.choices[0].message.content
18	print(result)
19	# {"name": "John Doe", "age": 30, "email": "john@example.com"}

ℹ

info

When using JSON mode, always include the desired schema in the system prompt. Without schema constraints, the model will produce valid JSON but with unpredictable keys and structure. Be explicit about required fields, types, and nullability.

json-schema-prompt.txt

TEXT

1	Extract invoice data as JSON with this exact schema:
2
3	{
4	"invoice_number": "string (required)",
5	"vendor_name": "string (required)",
6	"issue_date": "string (ISO 8601 date)",
7	"due_date": "string (ISO 8601 date)",
8	"line_items": [
9	{
10	"description": "string",
11	"quantity": "number",
12	"unit_price": "number",
13	"total": "number"
14	}
15	],
16	"subtotal": "number",
17	"tax": "number",
18	"total": "number",
19	"currency": "string (ISO 4217, default: USD)"
20	}
21
22	For any field that cannot be determined from the text, use null rather than guessing.

✓

best practice

Always set temperature=0 when using JSON mode for extraction or structured generation. Even with JSON mode, higher temperatures can produce keys that do not match your expected schema or introduce hallucinated fields.

Constrained Decoding

Constrained decoding forces the model to generate tokens that conform to a context-free grammar (CFG) or regular expression. This guarantees well-formed output at the token level, not just at the semantic level.

constrained-decoding.py

Python

1	from openai import OpenAI
2	import json
3
4	client = OpenAI()
5
6	# Using function calling for constrained structured output
7	response = client.chat.completions.create(
8	model="gpt-4o",
9	messages=[
10	{
11	"role": "system",
12	"content": "Extract product information from the description."
13	},
14	{
15	"role": "user",
16	"content": "Apple MacBook Pro 16-inch, M3 Max, 36GB RAM, 1TB SSD, Space Black, $3499"
17	}
18	],
19	tools=[
20	{
21	"type": "function",
22	"function": {
23	"name": "extract_product",
24	"description": "Extract structured product data",
25	"parameters": {
26	"type": "object",
27	"properties": {
28	"name": {"type": "string"},
29	"brand": {"type": "string"},
30	"model": {"type": "string"},
31	"specifications": {
32	"type": "array",
33	"items": {
34	"type": "object",
35	"properties": {
36	"key": {"type": "string"},
37	"value": {"type": "string"}
38	}
39	}
40	},
41	"price": {"type": "number", "nullable": true},
42	"currency": {"type": "string"}
43	},
44	"required": ["name", "brand", "model", "specifications"]
45	}
46	}
47	}
48	],
49	tool_choice={"type": "function", "function": {"name": "extract_product"}},
50	temperature=0,
51	)
52
53	tool_call = response.choices[0].message.tool_calls[0]
54	product = json.loads(tool_call.function.arguments)
55	print(json.dumps(product, indent=2))
56	# {
57	# "name": "MacBook Pro 16-inch",
58	# "brand": "Apple",
59	# "model": "M3 Max",
60	# "specifications": [
61	# {"key": "RAM", "value": "36GB"},
62	# {"key": "Storage", "value": "1TB SSD"},
63	# {"key": "Color", "value": "Space Black"}
64	# ],
65	# "price": 3499,
66	# "currency": "USD"
67	# }

For even stronger guarantees, libraries like LMQL, Guidance, and Outlines provide grammar-based sampling that constrains the token generation process at the logit level:

grammar-constrained.py

Python

1	# Using Outlines for regex-constrained generation
2	import outlines
3
4	model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")
5
6	# Define a regex for structured output
7	pattern = r"{ "name": "[A-Za-z ]+", "age": d+, "email": "[a-z@.]+" }"
8
9	generator = outlines.generate.regex(model, pattern)
10
11	response = generator("Extract info: Jane Smith, 28, jane@example.com")
12	print(response)
13	# { "name": "Jane Smith", "age": 28, "email": "jane@example.com" }
14
15
16	# Using Guidance for grammar-based control
17	import guidance
18
19	program = guidance('''
20	{{#system~}}
21	Extract user info as JSON.
22	{{~/system}}
23	{{#user~}}
24	{{input}}
25	{{~/user}}
26	{{#assistant~}}
27	{
28	"name": "{{gen 'name' pattern='[A-Za-z ]+'}}",
29	"age": {{gen 'age' pattern='\d+'}},
30	"email": "{{gen 'email' pattern='[a-z@.]+'}}"
31	}
32	{{~/assistant}}
33	''')
34
35	result = program(input="John Doe, 35, john.doe@company.com")
36	print(result["name"], result["age"], result["email"])

🔥

pro tip

Constrained decoding is the gold standard for production systems that cannot tolerate malformed output. The tradeoff is flexibility — grammar constraints can interfere with the model's ability to express complex or nuanced information. Use the loosest constraint that guarantees parseability. For most production use cases, function calling (tool use) provides the best balance of constraint and flexibility.

Function Calling for Structured Data

Function calling (also called tool use) is the most robust approach for structured outputs in production. By defining functions with JSON Schema parameters, you get guaranteed valid output that integrates directly with your type system.

function-calling-structured.py

Python

1	import openai
2	import json
3	from typing import List, Optional
4	from pydantic import BaseModel
5
6	client = openai.OpenAI()
7
8
9	# Define your data model
10	class ResumeExtraction(BaseModel):
11	name: str
12	email: str
13	phone: Optional[str] = None
14	skills: List[str]
15	years_experience: Optional[float] = None
16	education: List[dict] = []
17	certifications: List[str] = []
18
19
20	# Convert Pydantic model to OpenAI function schema
21	def pydantic_to_function(model_class):
22	schema = model_class.model_json_schema()
23	return {
24	"type": "function",
25	"function": {
26	"name": model_class.__name__,
27	"description": f"Extract {model_class.__name__} data",
28	"parameters": schema,
29	}
30	}
31
32
33	def extract_structured(text: str, model_class) -> BaseModel:
34	"""Extract structured data using function calling."""
35	response = client.chat.completions.create(
36	model="gpt-4o",
37	messages=[
38	{
39	"role": "system",
40	"content": "Extract the requested information from the text. "
41	"Do not guess values not present in the text."
42	},
43	{"role": "user", "content": text}
44	],
45	tools=[pydantic_to_function(model_class)],
46	tool_choice={
47	"type": "function",
48	"function": {"name": model_class.__name__}
49	},
50	temperature=0,
51	)
52
53	tool_call = response.choices[0].message.tool_calls[0]
54	data = json.loads(tool_call.function.arguments)
55	return model_class(**data)
56
57
58	resume_text = """
59	John Smith
60	john.smith@email.com \| (555) 123-4567
61
62	Senior Software Engineer with 8 years of experience in Python,
63	TypeScript, and cloud infrastructure. AWS Certified Solutions Architect.
64
65	Education:
66	- M.S. Computer Science, Stanford University (2018)
67	- B.S. Computer Science, UC Berkeley (2016)
68
69	Certifications: AWS Solutions Architect, Google Cloud Professional
70	"""
71
72	result = extract_structured(resume_text, ResumeExtraction)
73	print(f"Name: {result.name}")
74	print(f"Skills: {', '.join(result.skills)}")
75	print(f"Experience: {result.years_experience} years")

✓

best practice

Use Pydantic (Python) or Zod (TypeScript) to define your data models and convert them to JSON Schema for function definitions. This ensures your LLM output types match your application types exactly. Never manually write JSON Schema for function definitions — generate it from your type system.

JSON Schema Validation

Even with constrained decoding and function calling, validation is essential. LLMs can produce structurally valid JSON that is semantically wrong — for example, extracting a salary field as a string instead of a number, or omitting required fields.

json-schema-validation.py

Python

1	import json
2	from jsonschema import validate, ValidationError
3
4	# Define your schema
5	invoice_schema = {
6	"$schema": "http://json-schema.org/draft-07/schema#",
7	"type": "object",
8	"required": ["invoice_number", "vendor", "total", "currency"],
9	"properties": {
10	"invoice_number": {"type": "string", "pattern": "^INV-\d{6}$"},
11	"vendor": {"type": "string", "minLength": 1},
12	"issue_date": {"type": "string", "format": "date"},
13	"due_date": {"type": "string", "format": "date"},
14	"line_items": {
15	"type": "array",
16	"items": {
17	"type": "object",
18	"required": ["description", "total"],
19	"properties": {
20	"description": {"type": "string"},
21	"quantity": {"type": "number", "minimum": 0},
22	"unit_price": {"type": "number", "minimum": 0},
23	"total": {"type": "number", "minimum": 0}
24	}
25	}
26	},
27	"subtotal": {"type": "number", "minimum": 0},
28	"tax": {"type": "number", "minimum": 0},
29	"total": {"type": "number", "minimum": 0},
30	"currency": {
31	"type": "string",
32	"enum": ["USD", "EUR", "GBP", "JPY", "CAD"]
33	}
34	}
35	}
36
37
38	def extract_and_validate(text, schema):
39	"""Extract data and validate against JSON Schema."""
40	response = client.chat.completions.create(
41	model="gpt-4o",
42	messages=[
43	{"role": "system", "content": f"Extract invoice data matching this schema: {json.dumps(schema)}"},
44	{"role": "user", "content": text}
45	],
46	response_format={"type": "json_object"},
47	temperature=0,
48	)
49
50	try:
51	data = json.loads(response.choices[0].message.content)
52	validate(instance=data, schema=schema)
53	return data, None
54	except json.JSONDecodeError as e:
55	return None, f"Invalid JSON: {e}"
56	except ValidationError as e:
57	return None, f"Schema validation failed: {e.message}"
58
59
60	# Usage
61	invoice_text = """
62	INV-001234 \| Acme Corp
63	Issued: 2026-03-15
64	Due: 2026-04-14
65
66	Item: Server hosting (3 months @ $200/mo) = $600
67	Item: Domain renewal (1 year) = $15
68	Subtotal: $615
69	Tax (8%): $49.20
70	Total: $664.20
71	Currency: USD
72	"""
73
74	data, error = extract_and_validate(invoice_text, invoice_schema)
75	if error:
76	print(f"Extraction failed: {error}")
77	else:
78	print(f"Invoice: {data['invoice_number']}")
79	print(f"Total: {data['currency']} {data['total']}")

⚠

warning

Never trust LLM outputs without validation. Even models with JSON mode enabled occasionally produce outputs that violate your schema. Always validate with a JSON Schema validator and implement retry logic for failed extractions. A robust pipeline: extract → validate → retry (with error feedback) → fallback.

Parsing Strategies

Choosing the right parsing strategy depends on your reliability requirements, latency budget, and the complexity of your data model.

Strategy	Reliability	Latency	Complexity	Best For
Prompt + Regex	Low	Lowest	Low	Quick prototypes, simple extractions
JSON Mode	Medium	Low	Low	General structured output
Function Calling	High	Medium	Medium	Production extraction pipelines
Grammar Sampling	Highest	High	High	Mission-critical, zero-tolerance for errors
Two-Stage (Extract + Validate)	Very High	High	High	Complex multi-schema extraction

parse-with-retry.py

Python

1	import openai
2	import json
3	import re
4
5	client = openai.OpenAI()
6
7
8	def parse_with_retry(text, schema, max_retries=3):
9	"""Extract structured data with retry on validation failure."""
10	system_prompt = f"""Extract the data as JSON matching this schema.
11	Output ONLY valid JSON. No explanations.
12
13	Schema:
14	{json.dumps(schema, indent=2)}"""
15
16	for attempt in range(max_retries):
17	response = client.chat.completions.create(
18	model="gpt-4o",
19	messages=[
20	{"role": "system", "content": system_prompt},
21	{"role": "user", "content": text}
22	],
23	response_format={"type": "json_object"},
24	temperature=0,
25	)
26
27	content = response.choices[0].message.content
28
29	try:
30	data = json.loads(content)
31	# Validate against schema (simplified)
32	for key in schema.get("required", []):
33	if key not in data:
34	raise ValueError(f"Missing required field: {key}")
35
36	# Return successful extraction
37	return data, True, attempt + 1
38
39	except (json.JSONDecodeError, ValueError) as e:
40	if attempt < max_retries - 1:
41	# Feed error back to model for correction
42	text = f"""Previous extraction failed: {e}
43	Original text: {text}
44	Please try again, ensuring valid JSON matching the schema."""
45	continue
46	return None, False, attempt + 1
47
48	return None, False, max_retries
49
50
51	schema = {
52	"type": "object",
53	"required": ["name", "amount", "date"],
54	"properties": {
55	"name": {"type": "string"},
56	"amount": {"type": "number"},
57	"date": {"type": "string"}
58	}
59	}
60
61	result, success, attempts = parse_with_retry(
62	"Transaction: Coffee shop - $4.50 on March 20",
63	schema
64	)
65
66	if success:
67	print(f"Extracted: {result} ({attempts} attempt(s))")
68	else:
69	print(f"Failed after {attempts} attempts")

🔥

pro tip

The two-stage approach — extract then validate with retry — achieves near-100% structural validity in production. On each retry, feed the validation error back to the model so it can correct its output. After 3 failed attempts, fall back to a default or human review. This pattern reliably handles the 1-5% of cases where the model produces invalid output on the first try.

Production Best Practices

◆Always validate LLM output against your schema — never trust the model to produce correct JSON

◆Use function calling (tool use) instead of JSON mode when you need guaranteed schema adherence

◆Implement retry with error feedback for failed extractions — 3 retries catches ~99% of failures

◆Use temperature=0 for all structured extraction tasks

◆Define schemas in your type system (Pydantic/Zod) and auto-convert to JSON Schema

◆Monitor extraction quality — track schema validation pass rate, retry count, and field-level accuracy

◆For PII-heavy extraction, implement field-level redaction and never log raw extracted data

✓

best practice

Structured outputs are not optional for production systems. They are the difference between a demo and a product. Invest in your extraction pipeline: schema design, validation, retry logic, and monitoring. The upfront engineering effort pays for itself in reduced debugging, improved reliability, and seamless system integration.

$Blueprint — Engineering Documentation·Section ID: AI-06·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor