|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/deployment
$cat docs/ai-engineering-—-model-deployment.md
updated Recently·40 min read·published

AI Engineering — Model Deployment

AI EngineeringDeploymentIntermediate to Advanced
Introduction

Deploying AI models to production involves choosing between API providers, self-hosting, edge deployment, and serverless options. Each approach has different trade-offs for latency, cost, privacy, scalability, and operational complexity.

Modern deployment strategies combine multiple approaches: use API providers for complex reasoning tasks, self-host for high-volume or sensitive workloads, and edge deployment for latency-critical applications. The right architecture depends on your specific requirements.

Deployment Options

API Providers

The simplest path: use OpenAI, Anthropic, Google, or other providers. Zero infrastructure management, pay per token, automatic scaling, and access to the latest models. Best for startups, low-volume applications, and tasks requiring frontier models.

ProviderModelsStrengths
OpenAIGPT-4o, o1, o3, embeddingsBest ecosystem, tools, Assistants API
AnthropicClaude 3.5 Sonnet, Opus, HaikuLong context, safety, reasoning
Google AIGemini 1.5, 2.0, 2.5Multimodal, 1M+ context
Together AIOpen models (Llama, Mistral, DeepSeek)Open source models as API
GroqLlama, Mixtral, GemmaExtremely fast inference (LPU)

Self-Hosting

Run models on your own infrastructure. Full control over data, no per-token costs at inference time, lower latency for high-volume applications. Requires GPU hardware, operational expertise, and ongoing maintenance.

docker-compose.vllm.yml
YAML
1# Docker Compose for vLLM deployment
2version: "3.8"
3services:
4 vllm:
5 image: vllm/vllm-openai:latest
6 ports:
7 - "8000:8000"
8 volumes:
9 - ~/.cache/huggingface:/root/.cache/huggingface
10 command:
11 - "--model"
12 - "meta-llama/Llama-3.2-8B-Instruct"
13 - "--tensor-parallel-size"
14 - "2"
15 - "--max-model-len"
16 - "8192"
17 - "--gpu-memory-utilization"
18 - "0.95"
19 deploy:
20 resources:
21 reservations:
22 devices:
23 - driver: nvidia
24 count: 2
25 capabilities: [gpu]
26 environment:
27 - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}

info

Start with API providers. Self-host only when you hit cost, latency, or data residency constraints that APIs can't solve. Self-hosting a 70B model at scale requires significant GPU investment and operational expertise — factor in engineering time, not just GPU costs.
Cloud Inference Endpoints

Major cloud providers offer managed inference endpoints: Azure OpenAI Service, AWS Bedrock, and GCP Vertex AI. These combine the convenience of APIs with enterprise features like VPC integration, compliance certifications, and dedicated capacity.

ServiceModels AvailableKey Feature
Azure OpenAIGPT-4o, o1, o3, DALL-E, Whisper, embeddingsMicrosoft integration, private networking, EU data regions
AWS BedrockClaude, Llama, Mistral, Titan, JurassicMulti-model, AWS integration, Guardrails for safety
GCP Vertex AIGemini, Claude, Llama, Gemma, ImagenGoogle Cloud integration, Model Garden, AutoSxS
cloud-deployment.ts
TypeScript
1// Azure OpenAI deployment
2import { AzureOpenAI } from "openai";
3
4const client = new AzureOpenAI({
5 endpoint: process.env.AZURE_OPENAI_ENDPOINT,
6 apiKey: process.env.AZURE_OPENAI_KEY,
7 apiVersion: "2024-10-01-preview",
8 deployment: "gpt-4o" // Your deployment name
9});
10
11const response = await client.chat.completions.create({
12 messages: [
13 { role: "user", content: "Hello, Azure!" }
14 ],
15 model: "" // Model is inferred from deployment
16});
17
18// AWS Bedrock with Claude
19import {
20 BedrockRuntimeClient,
21 InvokeModelCommand
22} from "@aws-sdk/client-bedrock-runtime";
23
24const bedrock = new BedrockRuntimeClient({
25 region: "us-east-1"
26});
27
28const command = new InvokeModelCommand({
29 modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
30 contentType: "application/json",
31 body: JSON.stringify({
32 anthropic_version: "bedrock-2023-05-31",
33 max_tokens: 1024,
34 messages: [
35 { role: "user", content: "Hello from Bedrock!" }
36 ]
37 })
38});
39
40const response = await bedrock.send(command);
41const result = JSON.parse(
42 new TextDecoder().decode(response.body)
43);
Scaling & Load Balancing

Production inference requires horizontal scaling, load balancing, and auto-scaling. Key strategies include: request queuing with priority, circuit breakers for failing endpoints, canary deployments for model updates, and geo-distributed inference for global users.

load-balancer.ts
TypeScript
1// Load-balanced inference client
2interface InferenceEndpoint {
3 url: string;
4 model: string;
5 priority: number;
6 weight: number;
7 maxConcurrency: number;
8}
9
10class LoadBalancedClient {
11 private endpoints: InferenceEndpoint[];
12 private activeRequests = new Map<string, number>();
13
14 constructor(endpoints: InferenceEndpoint[]) {
15 this.endpoints = endpoints;
16 }
17
18 async complete(params: any): Promise<any> {
19 const endpoint = this.selectEndpoint();
20 if (!endpoint) throw new Error("No available endpoints");
21
22 this.activeRequests.set(
23 endpoint.url,
24 (this.activeRequests.get(endpoint.url) || 0) + 1
25 );
26
27 try {
28 return await this.callEndpoint(endpoint, params);
29 } finally {
30 const count = this.activeRequests.get(endpoint.url) || 0;
31 this.activeRequests.set(endpoint.url, count - 1);
32 }
33 }
34
35 private selectEndpoint(): InferenceEndpoint | null {
36 const available = this.endpoints.filter(ep =>
37 (this.activeRequests.get(ep.url) || 0) < ep.maxConcurrency
38 );
39
40 if (available.length === 0) return null;
41
42 // Weighted random selection
43 const totalWeight = available.reduce(
44 (sum, ep) => sum + ep.weight, 0
45 );
46 let random = Math.random() * totalWeight;
47
48 for (const ep of available) {
49 random -= ep.weight;
50 if (random <= 0) return ep;
51 }
52
53 return available[available.length - 1];
54 }
55
56 async callEndpoint(
57 endpoint: InferenceEndpoint,
58 params: any
59 ): Promise<any> {
60 // Circuit breaker pattern
61 const response = await fetch(endpoint.url, {
62 method: "POST",
63 headers: { "Content-Type": "application/json" },
64 body: JSON.stringify({ model: endpoint.model, ...params })
65 });
66
67 if (!response.ok) {
68 throw new Error(`Endpoint ${endpoint.url} failed: ${response.status}`);
69 }
70
71 return response.json();
72 }
73}

best practice

Implement circuit breakers for inference endpoints. If an endpoint returns 5xx errors or times out, mark it as degraded and route traffic to healthy endpoints. After a cooldown period, allow test traffic through. This prevents cascading failures and maintains availability during partial outages.
Auto-Scaling Strategies

Self-hosted deployments need auto-scaling to handle traffic variability. Key metrics: request queue depth, GPU utilization, request latency, and concurrent request count. Use Kubernetes HPA with custom metrics or serverless GPU platforms.

hpa-vllm.yml
YAML
1# Kubernetes HPA for vLLM deployment
2apiVersion: autoscaling/v2
3kind: HorizontalPodAutoscaler
4metadata:
5 name: vllm-hpa
6spec:
7 scaleTargetRef:
8 apiVersion: apps/v1
9 kind: Deployment
10 name: vllm
11 minReplicas: 1
12 maxReplicas: 10
13 metrics:
14 - type: Pods
15 pods:
16 metric:
17 name: vllm_queue_depth
18 target:
19 type: AverageValue
20 averageValue: 10
21 - type: Resource
22 resource:
23 name: nvidia_com_gpu_memory_used
24 target:
25 type: Utilization
26 averageUtilization: 80
27 behavior:
28 scaleDown:
29 stabilizationWindowSeconds: 300
30 policies:
31 - type: Percent
32 value: 50
33 periodSeconds: 60
Kubernetes HPA with custom metrics (GPU, queue depth) for GPU-based inference
Serverless GPU: RunPod, Banana, Replicate, Modal — scale to zero when idle
Request queuing: Redis-backed queues with priority levels for burst handling
Model caching: Pre-warm GPU memory with frequently used models to avoid cold starts
🔥

pro tip

GPU cold starts take 1-5 minutes. Use a minimum replica count of 2 for production, with pre-warmed model weights. Serverless GPU platforms can scale to zero for development but introduce unacceptable cold-start latency for user-facing applications. Consider a hybrid: always-on baseline + burst capacity.
Monitoring & Observability

Production inference requires monitoring: latency percentiles (p50, p95, p99), throughput (tokens/second), error rates, GPU utilization, memory usage, and cost tracking. Use structured logging with OpenTelemetry for distributed tracing.

Latency: Track time-to-first-token (TTFT) and tokens-per-second (TPS) per model and endpoint
Quality: Monitor response length, refusal rates, and user feedback scores
Cost: Track cost per request, cost per user, and daily/weekly burn rate with alerts
Alerting: Set up alerts for error rate spikes, latency degradation, and cost anomalies
$Blueprint — Engineering Documentation·Section ID: AI-DEP-01·Revision: 1.0