|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/cost

$cat docs/ai-engineering-—-cost-optimization.md

updated Recently·35 min read·published

AI Engineering — Cost Optimization

◆AI Engineering◆Operations◆Intermediate

Introduction

AI inference costs can quickly become a significant line item as applications scale. A typical production application using GPT-4o may spend $50-500 per day at moderate traffic. Cost optimization is not about avoiding spending — it's about maximizing the value per dollar by choosing the right model, deployment strategy, and usage patterns.

Effective cost management combines model selection, prompt engineering, caching, batching, distillation, and deployment architecture. Most teams can reduce costs by 60-90% with systematic optimization without sacrificing quality.

Pricing Models

Understanding how providers charge is the first step to optimization. Each pricing model has different cost characteristics for different usage patterns.

Model	How It Works	Best For	Cost Profile
Pay-per-Token	Charge per input + output token	Variable workloads, startups	Predictable per-request, variable monthly
Provisioned Throughput	Reserve TPM capacity, hourly/monthly fee	High-volume, steady workloads	Higher fixed cost, lower per-request cost
Batch API	50% discount, 24-hour completion window	Async offline processing	50% cheaper, delayed results
Self-Hosted	GPU cost + electricity + ops	High volume, data-sensitive	Fixed GPU cost, volume-dependent ops
Serverless GPU	Pay per second of GPU time	Sporadic workloads, prototyping	Zero when idle, premium when active

ℹ

info

Use pay-per-token for development and low volume. At approximately 1M+ tokens/day, evaluate provisioned throughput for API providers or self-hosting. Batch API is excellent for offline tasks like data enrichment, classification, and bulk summarization — 50% savings with no quality impact.

Caching Strategies

Caching is the single most impactful cost optimization. Many queries are repeated — exact duplicates, semantic equivalents, or variations that can be served from cache. Implement multi-level caching for maximum savings.

semantic-cache.ts

TypeScript

1	// Multi-level cache for LLM responses
2	class SemanticCache {
3	private exactCache = new Map<string, CacheEntry>();
4	private embeddingCache: EmbeddingCache;
5
6	constructor(
7	private embedder: EmbeddingFunction,
8	private similarity: CosineSimilarity,
9	private threshold: number = 0.95
10	) {
11	this.embeddingCache = new EmbeddingCache(embedder);
12	}
13
14	async get(query: string): Promise<string \| null> {
15	// Level 1: Exact match
16	const exact = this.exactCache.get(query);
17	if (exact && !this.isExpired(exact)) {
18	return exact.response;
19	}
20
21	// Level 2: Semantic match
22	const similar = await this.embeddingCache.findSimilar(
23	query, this.threshold
24	);
25	if (similar) {
26	return similar.response;
27	}
28
29	return null;
30	}
31
32	async set(query: string, response: string): Promise<void> {
33	this.exactCache.set(query, {
34	response,
35	timestamp: Date.now(),
36	ttl: 3600000 // 1 hour default
37	});
38
39	await this.embeddingCache.store(query, response);
40	}
41
42	private isExpired(entry: CacheEntry): boolean {
43	return Date.now() - entry.timestamp > entry.ttl;
44	}
45	}
46
47	// Prefix caching (for streaming responses)
48	class PrefixCache {
49	// Cache common prompt prefixes (system prompts, few-shot examples)
50	private prefixHashes = new Map<string, string>();
51
52	async getPrefixHash(systemPrompt: string): Promise<string> {
53	const hash = await sha256(systemPrompt);
54	if (!this.prefixHashes.has(hash)) {
55	// In production, this would use prompt caching APIs
56	// (Claude prompt caching, OpenAI prefix caching)
57	this.prefixHashes.set(hash, hash);
58	}
59	return hash;
60	}
61	}

Cache Type	Hit Rate	Latency Savings	Implementation
Exact Match	10-30%	100% (no API call)	Redis, in-memory Map
Semantic Cache	20-60%	~95% (embedding lookup)	Vector DB + embedding
Prefix Cache	N/A (reduces input tokens)	10-80% token reduction	API prompt caching, KV cache
Response Cache	40-70%	100% (pre-computed)	CDN, edge cache, Redis

✓

best practice

Implement exact-match caching first — it's the simplest and catches 10-30% of requests. Then add semantic caching with a vector database. For chat applications, cache common queries like "What can you do?", "Help", and greetings which often constitute 20% of all requests.

Prompt Optimization

Every token costs money. Reducing prompt length through careful engineering directly lowers costs. Common techniques: shorter system prompts, concise few-shot examples, token-efficient formatting, and removing redundant instructions.

Before — 412 tokens

"You are an AI assistant that helps users with their questions. You should be helpful, friendly, and accurate. Please provide detailed responses that are informative and well-structured. When you don't know something, admit it. Always cite sources when possible..."

After — 98 tokens (76% reduction)

"Helpful, accurate assistant. Be concise. Cite sources. Admit uncertainty."

prompt-optimization.ts

TypeScript

1	// Token-efficient prompt engineering
2
3	// ❌ Verbose: ~150 tokens
4	const verbosePrompt = `
5	You are a classification system. Your task is to classify
6	the sentiment of the following text as either positive,
7	negative, or neutral. Please analyze the text carefully
8	and return only one word: positive, negative, or neutral.
9	Here is the text to classify: "${text}"
10	`;
11
12	// ✅ Concise: ~30 tokens (80% reduction)
13	const concisePrompt = `Classify sentiment (pos/neg/neu): "${text}"`;
14
15	// Efficient few-shot: use structured format instead of prose
16	const fewShot = JSON.stringify([
17	{ text: "Great product!", label: "pos" },
18	{ text: "Terrible experience.", label: "neg" },
19	{ text: "It's okay I guess.", label: "neu" }
20	]);
21
22	// Dynamic prompt trimming
23	function trimPrompt(prompt: string, maxTokens: number): string {
24	const tokens = prompt.split(/s+/);
25	if (tokens.length <= maxTokens) return prompt;
26
27	// Keep the beginning and end, trim the middle
28	const keep = Math.floor(maxTokens * 0.4);
29	return [
30	...tokens.slice(0, keep),
31	"...[trimmed]...",
32	...tokens.slice(-keep)
33	].join(" ");
34	}

🔥

pro tip

Run every system prompt through a tokenizer before deployment. A reduction of 100 tokens per request at 1M requests/month saves $250/month (GPT-4o). Over a year, that's $3,000 saved from a single prompt optimization. Automate prompt length checks in CI/CD.

Model Selection by Task

Different tasks need different model capabilities. Using the most powerful model for every request is extremely wasteful. Route simple tasks to cheaper, faster models and reserve advanced models for complex reasoning.

Task Type	Recommended Model	Cost Ratio	Example
Classification	GPT-4o-mini / Haiku	1x	"Classify this email"
Extraction	GPT-4o-mini / Haiku	1x	"Extract name, date, amount"
Summarization	GPT-4o-mini / Sonnet	1-5x	"Summarize this article"
Chat / Q&A	Sonnet / GPT-4o	5-10x	User-facing chatbot
Code Generation	GPT-4o / Sonnet / Opus	10-40x	"Write a React component"
Complex Reasoning	o1 / o3 / Opus	40-200x	Math, logic, planning
Embeddings	text-embedding-3-small	1x	Vector search

model-router.ts

TypeScript

1	// Smart model router
2	class ModelRouter {
3	private routes = [
4	{
5	pattern: /classify\|extract\|categorize\|tag/i,
6	model: "gpt-4o-mini",
7	costPerKTokens: 0.00015
8	},
9	{
10	pattern: /summarize\|translate\|rewrite/i,
11	model: "claude-3-haiku-20240307",
12	costPerKTokens: 0.00025
13	},
14	{
15	pattern: /code\|implement\|debug\|refactor/i,
16	model: "gpt-4o",
17	costPerKTokens: 0.01
18	},
19	{
20	pattern: /.*/,
21	model: "claude-3-5-sonnet-20241022",
22	costPerKTokens: 0.003
23	}
24	];
25
26	async route(prompt: string, messages: any[]) {
27	const matched = this.routes.find(r => r.pattern.test(prompt));
28	const model = matched ? matched.model : "gpt-4o-mini";
29
30	console.log(`Routing to ${model} (cost: ${matched?.costPerKTokens}/1K tokens)`);
31
32	return await callModel(model, messages);
33	}
34
35	async estimateCost(prompt: string, messages: any[]) {
36	const tokens = estimateTokens(messages);
37	const route = this.routes.find(r => r.pattern.test(prompt));
38	return tokens * (route?.costPerKTokens \|\| 0.01) / 1000;
39	}
40	}

✓

best practice

Implement a model router that classifies each request and routes to the appropriate model. Many teams save 70-80% by routing 80% of requests to cheap models while maintaining quality. Use GPT-4o-mini for classification/extraction as your default, and escalate to stronger models only when the cheap model's confidence is low.

Distillation for Cost Reduction

Model distillation uses a large "teacher" model to generate training data for a smaller "student" model. The student learns to imitate the teacher's outputs on your specific task. This can achieve 90%+ of teacher quality at 5-10% of the cost.

distillation.ts

TypeScript

1	// Distillation data generation
2	async function generateDistillationData(
3	teacher: string,
4	student: string,
5	examples: { input: string; task: string }[]
6	) {
7	const trainingData = [];
8
9	for (const example of examples) {
10	// Teacher generates the output
11	const teacherResponse = await callModel(teacher, [
12	{ role: "system", content: example.task },
13	{ role: "user", content: example.input }
14	]);
15
16	trainingData.push({
17	messages: [
18	{ role: "system", content: example.task },
19	{ role: "user", content: example.input },
20	{ role: "assistant", content: teacherResponse }
21	]
22	});
23	}
24
25	// Fine-tune student model on this data
26	// This creates a custom version of the cheap model
27	// that performs like the expensive one on this specific task
28	return await fineTuneModel(student, trainingData);
29	}
30
31	// Cost comparison after distillation
32	console.log({
33	teacherCost: "$10.00/1M tokens",
34	studentCost: "$0.15/1M tokens", // GPT-4o-mini
35	savings: "98%",
36	quality: "95% of teacher on specific task"
37	});

ℹ

info

Distillation works best for narrow, well-defined tasks (classification, extraction, structured output). For open-ended chat or creative tasks, distilled models lose too much quality. Generate at least 500-1000 teacher examples for good results, and evaluate student outputs against teacher outputs on a held-out test set.

Batching & Concurrency

Batch processing reduces costs by: using Batch API (50% discount for async), batching multiple requests into a single API call (via multi-prompt or JSON mode), and maximizing throughput on self-hosted GPUs through continuous batching.

batching.ts

TypeScript

1	// OpenAI Batch API — 50% discount
2	async function createBatch(
3	requests: { custom_id: string; params: any }[]
4	) {
5	const batch = await openai.batches.create({
6	input_file_id: await uploadBatchFile(requests),
7	endpoint: "/v1/chat/completions",
8	completion_window: "24h"
9	});
10
11	console.log(`Batch created: ${batch.id}`);
12	return batch.id;
13	}
14
15	// Self-hosted continuous batching
16	// vLLM handles this automatically — just send concurrent requests
17	async function continuousBatch(
18	client: OpenAI,
19	prompts: string[],
20	batchSize: number = 32
21	) {
22	const results = [];
23
24	for (let i = 0; i < prompts.length; i += batchSize) {
25	const batch = prompts.slice(i, i + batchSize);
26
27	const batchResults = await Promise.all(
28	batch.map(prompt =>
29	client.chat.completions.create({
30	model: "meta-llama/Llama-3.2-8B-Instruct",
31	messages: [{ role: "user", content: prompt }],
32	max_tokens: 256
33	})
34	)
35	);
36
37	results.push(...batchResults.map(r => r.choices[0].message.content));
38	}
39
40	return results;
41	}

✓

best practice

For self-hosted inference engines, send requests concurrently rather than sequentially. vLLM and TGI handle continuous batching internally — you don't need to manually batch. For API providers, use the Batch API for all non-real-time workloads to save 50%. Reserve synchronous API calls for user-facing real-time interactions.

Cost Tracking & Budgeting

You can't optimize what you don't measure. Implement cost tracking per request, per user, per model, and per feature. Set up budgets with alerts and automatic model downgrade when approaching limits.

cost-tracker.ts

TypeScript

1	// Cost tracking middleware
2	class CostTracker {
3	private costs: CostEntry[] = [];
4
5	async trackCall(params: {
6	model: string;
7	inputTokens: number;
8	outputTokens: number;
9	userId?: string;
10	feature?: string;
11	}) {
12	const cost = this.calculateCost(
13	params.model,
14	params.inputTokens,
15	params.outputTokens
16	);
17
18	const entry = {
19	...params,
20	cost,
21	timestamp: Date.now()
22	};
23
24	this.costs.push(entry);
25
26	// Check budget threshold
27	await this.checkBudget(entry);
28
29	return cost;
30	}
31
32	private calculateCost(
33	model: string,
34	inputTokens: number,
35	outputTokens: number
36	): number {
37	const rates = {
38	"gpt-4o": { input: 2.50, output: 10.00 },
39	"gpt-4o-mini": { input: 0.15, output: 0.60 },
40	"claude-3-5-sonnet": { input: 3.00, output: 15.00 }
41	};
42
43	const rate = rates[model] \|\| rates["gpt-4o-mini"];
44	return (
45	(inputTokens / 1_000_000) * rate.input +
46	(outputTokens / 1_000_000) * rate.output
47	);
48	}
49
50	async checkBudget(entry: CostEntry) {
51	const dailyTotal = this.costs
52	.filter(c => {
53	const today = new Date();
54	const entryDate = new Date(c.timestamp);
55	return entryDate.toDateString() === today.toDateString();
56	})
57	.reduce((sum, c) => sum + c.cost, 0);
58
59	if (dailyTotal > 100) {
60	console.warn(`Daily cost ${dailyTotal} exceeds $100 threshold`);
61	// Trigger alert, switch to cheaper model, etc.
62	}
63	}
64
65	getStats() {
66	return {
67	totalCost: this.costs.reduce((s, c) => s + c.cost, 0),
68	byModel: this.groupBy("model"),
69	byFeature: this.groupBy("feature"),
70	avgCostPerRequest: this.costs.length > 0
71	? this.costs.reduce((s, c) => s + c.cost, 0) / this.costs.length
72	: 0
73	};
74	}
75
76	private groupBy(key: keyof CostEntry) {
77	return this.costs.reduce((acc, entry) => {
78	const val = String(entry[key]);
79	acc[val] = (acc[val] \|\| 0) + entry.cost;
80	return acc;
81	}, {} as Record<string, number>);
82	}
83	}

$Blueprint — Engineering Documentation·Section ID: AI-COST-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor