Caching & Latency

◆AI◆Intermediate

Introduction

Latency and cost are the two biggest operational challenges in production LLM deployments. Caching is the most effective technique for reducing both — by reusing previous responses and computations, you can serve a large fraction of requests without invoking the model at all or by reusing intermediate computation.

LLM caching is fundamentally different from traditional web caching. Responses vary with prompt wording, temperature, system instructions, and model version. Effective caching strategies must account for semantic similarity (not just exact match), cache invalidation on model updates, and the trade-off between cache hit rate and computational overhead.

This guide covers semantic caching with embeddings, exact-match response caching, prefix caching for prompt sharing, speculative decoding for token-level speedups, KV cache management, batching strategies, and practical latency benchmarks.

Semantic Caching

Semantic caching stores responses keyed by embedding vectors rather than exact text. When a new query arrives, it is embedded and compared against stored embeddings using cosine similarity. If a sufficiently similar query exists in the cache, the cached response is returned — even if the queries used different words.

This is particularly effective for customer support, FAQ systems, and knowledge-base queries where users ask the same questions with different wording. A well-tuned semantic cache can achieve 30-60% hit rates with a cosine similarity threshold of 0.85-0.95.

semantic-cache.py

Python

1	import numpy as np
2	from typing import List, Optional, Tuple
3	from openai import OpenAI
4
5	class SemanticCache:
6	def __init__(self, threshold: float = 0.92, max_size: int = 10000):
7	self.threshold = threshold
8	self.max_size = max_size
9	self.embeddings: List[np.ndarray] = []
10	self.responses: List[str] = []
11	self.prompts: List[str] = []
12	self.client = OpenAI()
13
14	def _embed(self, text: str) -> np.ndarray:
15	response = self.client.embeddings.create(
16	model="text-embedding-3-small",
17	input=text
18	)
19	return np.array(response.data[0].embedding)
20
21	def get(self, prompt: str) -> Optional[str]:
22	if not self.embeddings:
23	return None
24	query_emb = self._embed(prompt)
25	similarities = [
26	np.dot(query_emb, e) / (np.linalg.norm(query_emb) * np.linalg.norm(e))
27	for e in self.embeddings
28	]
29	best_idx = int(np.argmax(similarities))
30	if similarities[best_idx] >= self.threshold:
31	return self.responses[best_idx]
32	return None
33
34	def put(self, prompt: str, response: str):
35	emb = self._embed(prompt)
36	if len(self.embeddings) >= self.max_size:
37	self.embeddings.pop(0)
38	self.responses.pop(0)
39	self.prompts.pop(0)
40	self.embeddings.append(emb)
41	self.responses.append(response)
42	self.prompts.append(prompt)
43
44	def hit_rate(self) -> float:
45	if not hasattr(self, "_hits") or not hasattr(self, "_misses"):
46	return 0.0
47	total = self._hits + self._misses
48	return self._hits / total if total > 0 else 0.0
49
50	# Usage
51	cache = SemanticCache(threshold=0.92)
52
53	def cached_llm_call(prompt: str) -> str:
54	cached = cache.get(prompt)
55	if cached:
56	cache._hits = getattr(cache, "_hits", 0) + 1
57	return cached
58	cache._misses = getattr(cache, "_misses", 0) + 1
59	response = llm_call(prompt)
60	cache.put(prompt, response)
61	return response

🔥

pro tip

Use a vector database (ChromaDB, Pinecone, Qdrant) instead of in-memory lists for production semantic caches. They provide efficient ANN (approximate nearest neighbor) search, persistence, filtering, and horizontal scaling. Set up TTL-based expiration to handle stale responses when your model is updated.

Exact-Match Caching

Exact-match caching is the simplest and fastest caching strategy. Use it for deterministic requests where exact prompt reuse is common — such as health check endpoints, API documentation lookups, and template-based generations with identical parameters.

exact-cache.py

Python

1	import hashlib
2	import json
3	from functools import lru_cache
4	from typing import Any
5
6	class ExactMatchCache:
7	def __init__(self, max_size: int = 1000, ttl_seconds: int = 3600):
8	self.max_size = max_size
9	self.ttl = ttl_seconds
10	self._cache: dict = {}
11	self._timestamps: dict = {}
12
13	def _make_key(self, args, *kwargs) -> str:
14	serialized = json.dumps({"args": args, "kwargs": kwargs}, sort_keys=True)
15	return hashlib.sha256(serialized.encode()).hexdigest()
16
17	def get(self, key: str) -> Any:
18	if key in self._cache:
19	age = time.time() - self._timestamps[key]
20	if age < self.ttl:
21	return self._cache[key]
22	else:
23	del self._cache[key]
24	del self._timestamps[key]
25	return None
26
27	def put(self, key: str, value: Any):
28	if len(self._cache) >= self.max_size:
29	oldest = min(self._timestamps, key=self._timestamps.get)
30	del self._cache[oldest]
31	del self._timestamps[oldest]
32	self._cache[key] = value
33	self._timestamps[key] = time.time()
34
35	def __call__(self, func):
36	def wrapper(args, *kwargs):
37	key = self._make_key(args, *kwargs)
38	cached = self.get(key)
39	if cached is not None:
40	return cached
41	result = func(args, *kwargs)
42	self.put(key, result)
43	return result
44	return wrapper
45
46	# Usage
47	cache = ExactMatchCache(max_size=5000, ttl_seconds=300)
48
49	@cache
50	def get_completion(prompt: str, temperature: float = 0.0) -> str:
51	# Only safe to cache with temperature=0.0
52	response = client.chat.completions.create(
53	model="gpt-4o-mini",
54	messages=[{"role": "user", "content": prompt}],
55	temperature=0.0
56	)
57	return response.choices[0].message.content

✓

best practice

Exact-match caching is only safe with temperature=0.0 (deterministic generation). With non-zero temperature, the same prompt produces different responses each time, making exact-match caching semantically incorrect. Always invalidate the cache when you update the model, system prompt, or generation parameters.

Prefix / Prompt Caching

Prefix caching stores the KV cache computed for shared prompt prefixes. If many requests share the same system message or prompt prefix (e.g., a long instruction set for a task), the prefix computation can be done once and reused across requests. This technique is supported natively by some inference engines and APIs.

Anthropic's Prompt Caching and OpenAI's Prompt Caching are API-level examples. They automatically cache frequently used prompt prefixes and apply discounts for cache hits. On the self-hosted side, vLLM and TensorRT-LLM support automatic prefix caching (APC) that caches KV blocks across requests.

prefix-caching.py

Python

1	# Anthropic prompt caching (API-level)
2	import anthropic
3
4	client = anthropic.Anthropic()
5
6	response = client.beta.prompt_caching.messages.create(
7	model="claude-3-5-sonnet-20241022",
8	max_tokens=1024,
9	system=[
10	{
11	"type": "text",
12	"text": LONG_SYSTEM_PROMPT,
13	"cache_control": {"type": "ephemeral"}
14	}
15	],
16	messages=[{"role": "user", "content": prompt}]
17	)
18
19	# Check cache metrics
20	print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
21	print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
22
23
24	# Self-hosted with vLLM (automatic prefix caching)
25	from vllm import LLM, SamplingParams
26
27	llm = LLM(
28	model="meta-llama/Llama-3.2-3B",
29	enable_prefix_caching=True, # Enable APC
30	max_num_seqs=256
31	)
32
33	params = SamplingParams(temperature=0.0, max_tokens=256)
34
35	# Shared prefix is cached after first request
36	shared_prefix = "You are an expert Python programmer..."
37	requests = [shared_prefix + q for q in questions]
38	outputs = llm.generate(requests, params)

ℹ

info

Prefix caching is most effective when your application has a large shared prompt (system prompt, task description, few-shot examples) with varying user inputs. The longer the shared prefix, the greater the savings. Monitor your cache hit rate — if it's low, consider restructuring prompts to maximize the shared prefix length.

Speculative Decoding

Speculative decoding speeds up generation by using a fast draft model to propose multiple tokens, which are then verified by the target model in a single forward pass. Because verification can be parallelized and most tokens are accepted, this achieves 2-3x speedup without any quality loss — the output distribution is identical to the target model.

speculative-decoding.py

Python

1	# Speculative decoding with Hugging Face transformers
2	from transformers import (
3	AutoModelForCausalLM,
4	AutoTokenizer,
5	GenerationConfig
6	)
7
8	target_model = AutoModelForCausalLM.from_pretrained(
9	"meta-llama/Llama-3.2-70B",
10	device_map="auto"
11	)
12	draft_model = AutoModelForCausalLM.from_pretrained(
13	"meta-llama/Llama-3.2-3B",
14	device_map="auto"
15	)
16
17	# Configure speculative decoding
18	gen_config = GenerationConfig(
19	assistant_model=draft_model, # Draft model
20	num_assistant_tokens=5, # Tokens to speculate
21	do_sample=False,
22	max_new_tokens=256,
23	)
24
25	# Generate — draft model proposes, target model verifies
26	outputs = target_model.generate(
27	input_ids,
28	generation_config=gen_config
29	)
30
31	# vLLM speculative decoding
32	from vllm import LLM, SamplingParams
33
34	llm = LLM(
35	model="meta-llama/Llama-3.2-70B",
36	speculative_model="meta-llama/Llama-3.2-3B",
37	num_speculative_tokens=5,
38	use_v2_block_manager=True
39	)

Batching Techniques

Batching multiple requests into a single model invocation dramatically improves throughput. Continuous batching (also called dynamic batching or inflight batching) allows the inference engine to add newly arrived requests to a running batch as generation completes.

batching.py

Python

1	from vllm import LLM, SamplingParams
2
3	# vLLM handles continuous batching automatically
4	llm = LLM(
5	model="meta-llama/Llama-3.2-3B",
6	max_num_seqs=256, # Max concurrent sequences
7	max_model_len=4096, # Max total length (prompt + generation)
8	gpu_memory_utilization=0.9,
9	)
10
11	# Multiple prompts — automatically batched
12	prompts = [
13	"Explain quantum computing",
14	"Write a Python function for binary search",
15	"Summarize: The Industrial Revolution...",
16	] * 10 # 30 prompts total
17
18	params = SamplingParams(temperature=0.0, max_tokens=256)
19	outputs = llm.generate(prompts, params)
20
21	# Track throughput
22	import time
23	start = time.time()
24	outputs = llm.generate(prompts, params)
25	elapsed = time.time() - start
26	total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
27	print(f"Throughput: {total_tokens/elapsed:.0f} tokens/second")
28	print(f"Average latency: {elapsed/len(prompts)*1000:.0f}ms per request")

✓

best practice

Use continuous batching wherever possible. It achieves 10-20x higher throughput than static batching (waiting for N requests before processing). The optimal batch size depends on your GPU memory and model size — monitor GPU utilization and increase batch size until you approach memory limits.

Latency Benchmarks

Understanding typical latency ranges helps set realistic expectations and identify optimization opportunities. The following table shows approximate end-to-end latencies for different model sizes and optimization levels.

Model	Optimization	TTFT	Tokens/s	Latency (100 tokens)
GPT-4o	API (no cache)	~300ms	~60	~2s
GPT-4o-mini	API (no cache)	~150ms	~150	~800ms
Llama 3.2 3B	vLLM (A100)	~50ms	~400	~300ms
Llama 3.2 70B	vLLM (4x A100)	~200ms	~80	~1.5s
Semantic Cache Hit	Embedding + similarity	~10ms	N/A	~10ms

$Blueprint — Engineering Documentation·Section ID: AI-CACHE-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor