AI Engineering — Model Deployment
Deploying AI models to production involves choosing between API providers, self-hosting, edge deployment, and serverless options. Each approach has different trade-offs for latency, cost, privacy, scalability, and operational complexity.
Modern deployment strategies combine multiple approaches: use API providers for complex reasoning tasks, self-host for high-volume or sensitive workloads, and edge deployment for latency-critical applications. The right architecture depends on your specific requirements.
API Providers
The simplest path: use OpenAI, Anthropic, Google, or other providers. Zero infrastructure management, pay per token, automatic scaling, and access to the latest models. Best for startups, low-volume applications, and tasks requiring frontier models.
| Provider | Models | Strengths |
|---|---|---|
| OpenAI | GPT-4o, o1, o3, embeddings | Best ecosystem, tools, Assistants API |
| Anthropic | Claude 3.5 Sonnet, Opus, Haiku | Long context, safety, reasoning |
| Google AI | Gemini 1.5, 2.0, 2.5 | Multimodal, 1M+ context |
| Together AI | Open models (Llama, Mistral, DeepSeek) | Open source models as API |
| Groq | Llama, Mixtral, Gemma | Extremely fast inference (LPU) |
Self-Hosting
Run models on your own infrastructure. Full control over data, no per-token costs at inference time, lower latency for high-volume applications. Requires GPU hardware, operational expertise, and ongoing maintenance.
| 1 | # Docker Compose for vLLM deployment |
| 2 | version: "3.8" |
| 3 | services: |
| 4 | vllm: |
| 5 | image: vllm/vllm-openai:latest |
| 6 | ports: |
| 7 | - "8000:8000" |
| 8 | volumes: |
| 9 | - ~/.cache/huggingface:/root/.cache/huggingface |
| 10 | command: |
| 11 | - "--model" |
| 12 | - "meta-llama/Llama-3.2-8B-Instruct" |
| 13 | - "--tensor-parallel-size" |
| 14 | - "2" |
| 15 | - "--max-model-len" |
| 16 | - "8192" |
| 17 | - "--gpu-memory-utilization" |
| 18 | - "0.95" |
| 19 | deploy: |
| 20 | resources: |
| 21 | reservations: |
| 22 | devices: |
| 23 | - driver: nvidia |
| 24 | count: 2 |
| 25 | capabilities: [gpu] |
| 26 | environment: |
| 27 | - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN} |
info
Major cloud providers offer managed inference endpoints: Azure OpenAI Service, AWS Bedrock, and GCP Vertex AI. These combine the convenience of APIs with enterprise features like VPC integration, compliance certifications, and dedicated capacity.
| Service | Models Available | Key Feature |
|---|---|---|
| Azure OpenAI | GPT-4o, o1, o3, DALL-E, Whisper, embeddings | Microsoft integration, private networking, EU data regions |
| AWS Bedrock | Claude, Llama, Mistral, Titan, Jurassic | Multi-model, AWS integration, Guardrails for safety |
| GCP Vertex AI | Gemini, Claude, Llama, Gemma, Imagen | Google Cloud integration, Model Garden, AutoSxS |
| 1 | // Azure OpenAI deployment |
| 2 | import { AzureOpenAI } from "openai"; |
| 3 | |
| 4 | const client = new AzureOpenAI({ |
| 5 | endpoint: process.env.AZURE_OPENAI_ENDPOINT, |
| 6 | apiKey: process.env.AZURE_OPENAI_KEY, |
| 7 | apiVersion: "2024-10-01-preview", |
| 8 | deployment: "gpt-4o" // Your deployment name |
| 9 | }); |
| 10 | |
| 11 | const response = await client.chat.completions.create({ |
| 12 | messages: [ |
| 13 | { role: "user", content: "Hello, Azure!" } |
| 14 | ], |
| 15 | model: "" // Model is inferred from deployment |
| 16 | }); |
| 17 | |
| 18 | // AWS Bedrock with Claude |
| 19 | import { |
| 20 | BedrockRuntimeClient, |
| 21 | InvokeModelCommand |
| 22 | } from "@aws-sdk/client-bedrock-runtime"; |
| 23 | |
| 24 | const bedrock = new BedrockRuntimeClient({ |
| 25 | region: "us-east-1" |
| 26 | }); |
| 27 | |
| 28 | const command = new InvokeModelCommand({ |
| 29 | modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0", |
| 30 | contentType: "application/json", |
| 31 | body: JSON.stringify({ |
| 32 | anthropic_version: "bedrock-2023-05-31", |
| 33 | max_tokens: 1024, |
| 34 | messages: [ |
| 35 | { role: "user", content: "Hello from Bedrock!" } |
| 36 | ] |
| 37 | }) |
| 38 | }); |
| 39 | |
| 40 | const response = await bedrock.send(command); |
| 41 | const result = JSON.parse( |
| 42 | new TextDecoder().decode(response.body) |
| 43 | ); |
Production inference requires horizontal scaling, load balancing, and auto-scaling. Key strategies include: request queuing with priority, circuit breakers for failing endpoints, canary deployments for model updates, and geo-distributed inference for global users.
| 1 | // Load-balanced inference client |
| 2 | interface InferenceEndpoint { |
| 3 | url: string; |
| 4 | model: string; |
| 5 | priority: number; |
| 6 | weight: number; |
| 7 | maxConcurrency: number; |
| 8 | } |
| 9 | |
| 10 | class LoadBalancedClient { |
| 11 | private endpoints: InferenceEndpoint[]; |
| 12 | private activeRequests = new Map<string, number>(); |
| 13 | |
| 14 | constructor(endpoints: InferenceEndpoint[]) { |
| 15 | this.endpoints = endpoints; |
| 16 | } |
| 17 | |
| 18 | async complete(params: any): Promise<any> { |
| 19 | const endpoint = this.selectEndpoint(); |
| 20 | if (!endpoint) throw new Error("No available endpoints"); |
| 21 | |
| 22 | this.activeRequests.set( |
| 23 | endpoint.url, |
| 24 | (this.activeRequests.get(endpoint.url) || 0) + 1 |
| 25 | ); |
| 26 | |
| 27 | try { |
| 28 | return await this.callEndpoint(endpoint, params); |
| 29 | } finally { |
| 30 | const count = this.activeRequests.get(endpoint.url) || 0; |
| 31 | this.activeRequests.set(endpoint.url, count - 1); |
| 32 | } |
| 33 | } |
| 34 | |
| 35 | private selectEndpoint(): InferenceEndpoint | null { |
| 36 | const available = this.endpoints.filter(ep => |
| 37 | (this.activeRequests.get(ep.url) || 0) < ep.maxConcurrency |
| 38 | ); |
| 39 | |
| 40 | if (available.length === 0) return null; |
| 41 | |
| 42 | // Weighted random selection |
| 43 | const totalWeight = available.reduce( |
| 44 | (sum, ep) => sum + ep.weight, 0 |
| 45 | ); |
| 46 | let random = Math.random() * totalWeight; |
| 47 | |
| 48 | for (const ep of available) { |
| 49 | random -= ep.weight; |
| 50 | if (random <= 0) return ep; |
| 51 | } |
| 52 | |
| 53 | return available[available.length - 1]; |
| 54 | } |
| 55 | |
| 56 | async callEndpoint( |
| 57 | endpoint: InferenceEndpoint, |
| 58 | params: any |
| 59 | ): Promise<any> { |
| 60 | // Circuit breaker pattern |
| 61 | const response = await fetch(endpoint.url, { |
| 62 | method: "POST", |
| 63 | headers: { "Content-Type": "application/json" }, |
| 64 | body: JSON.stringify({ model: endpoint.model, ...params }) |
| 65 | }); |
| 66 | |
| 67 | if (!response.ok) { |
| 68 | throw new Error(`Endpoint ${endpoint.url} failed: ${response.status}`); |
| 69 | } |
| 70 | |
| 71 | return response.json(); |
| 72 | } |
| 73 | } |
best practice
Self-hosted deployments need auto-scaling to handle traffic variability. Key metrics: request queue depth, GPU utilization, request latency, and concurrent request count. Use Kubernetes HPA with custom metrics or serverless GPU platforms.
| 1 | # Kubernetes HPA for vLLM deployment |
| 2 | apiVersion: autoscaling/v2 |
| 3 | kind: HorizontalPodAutoscaler |
| 4 | metadata: |
| 5 | name: vllm-hpa |
| 6 | spec: |
| 7 | scaleTargetRef: |
| 8 | apiVersion: apps/v1 |
| 9 | kind: Deployment |
| 10 | name: vllm |
| 11 | minReplicas: 1 |
| 12 | maxReplicas: 10 |
| 13 | metrics: |
| 14 | - type: Pods |
| 15 | pods: |
| 16 | metric: |
| 17 | name: vllm_queue_depth |
| 18 | target: |
| 19 | type: AverageValue |
| 20 | averageValue: 10 |
| 21 | - type: Resource |
| 22 | resource: |
| 23 | name: nvidia_com_gpu_memory_used |
| 24 | target: |
| 25 | type: Utilization |
| 26 | averageUtilization: 80 |
| 27 | behavior: |
| 28 | scaleDown: |
| 29 | stabilizationWindowSeconds: 300 |
| 30 | policies: |
| 31 | - type: Percent |
| 32 | value: 50 |
| 33 | periodSeconds: 60 |
pro tip
Production inference requires monitoring: latency percentiles (p50, p95, p99), throughput (tokens/second), error rates, GPU utilization, memory usage, and cost tracking. Use structured logging with OpenTelemetry for distributed tracing.