Evaluation & Benchmarks

◆AI◆Advanced

Introduction

Evaluation is the backbone of LLM development. Without rigorous evaluation, you cannot measure progress, compare models, detect regressions, or make informed decisions about training interventions. The LLM evaluation landscape spans automated benchmarks, human evaluation, and LLM-as-judge approaches, each with distinct trade-offs.

The challenge of LLM evaluation is that language tasks are open-ended — there is no single correct answer for most prompts. This makes automated evaluation difficult and requires carefully designed protocols to produce reliable, reproducible results.

This guide covers evaluation strategies from simple held-out loss to complex benchmark suites, with practical code for setting up evaluation pipelines, computing metrics, and interpreting results.

Evaluation Strategies

Different evaluation strategies serve different purposes. No single strategy is sufficient — robust evaluation combines multiple approaches to build a complete picture of model capability.

Strategy	Cost	Reliability	Best For
Held-out Loss	Free	High	Training monitoring, overfitting detection
Multiple Choice	Low	High	Factual knowledge (MMLU, ARC)
Generation Benchmarks	Medium	Medium	Code generation, math, summarization
LLM-as-Judge	Medium	Medium	Open-ended quality, instruction following
Human Evaluation	High	Highest	Final quality assessment, alignment
Arena / ELO	High	High	Holistic model comparison (Chatbot Arena)

✓

best practice

Use held-out loss during training for quick iteration, benchmark evaluations for milestone comparisons, and human evaluation for final release decisions. Never rely on a single metric — LLMs can game individual benchmarks. A model that scores well on MMLU may still fail at basic instruction following.

Benchmark Suites

Standardized benchmarks enable apples-to-apples comparison across models. The LLM ecosystem has converged on several key benchmarks that evaluate different capabilities.

Benchmark	Capability	Format	Scale
MMLU	World knowledge (57 subjects)	Multiple choice	14K questions
HumanEval	Code generation	Function completion	164 problems
GSM8K	Math reasoning	Word problems	8.5K problems
HELM	Holistic evaluation (42 scenarios)	Multi-format	Multi-dataset
Chatbot Arena	Human preference	ELO (pairwise votes)	100K+ votes
BIG-Bench	Reasoning (200+ tasks)	Multi-format	204 tasks

Running MMLU Evaluation

mmlu-eval.py

Python

1	from lm_eval import evaluator
2	from lm_eval.models.huggingface import HFLM
3
4	# Load model for evaluation
5	model = HFLM(
6	pretrained="meta-llama/Llama-3.2-3B",
7	device="cuda",
8	batch_size=4
9	)
10
11	# Run MMLU evaluation
12	results = evaluator.simple_evaluate(
13	model=model,
14	tasks=["mmlu"],
15	num_fewshot=5,
16	batch_size=4,
17	limit=1000 # Reduce for quick checks
18	)
19
20	# Extract metrics
21	mmlu_score = results["results"]["mmlu"]["acc,none"]
22	print(f"MMLU 5-shot: {mmlu_score:.3f}")
23
24	# Run multiple benchmarks
25	benchmarks = ["mmlu", "gsm8k", "hellaswag", "arc_challenge"]
26	full_results = evaluator.simple_evaluate(
27	model=model,
28	tasks=benchmarks,
29	num_fewshot=5
30	)
31
32	for task in benchmarks:
33	score = full_results["results"][task]["acc,none"]
34	print(f"{task}: {score:.3f}")

HumanEval Code Generation

humaneval-eval.py

Python

1	from human_eval.data import read_problems
2	from human_eval.execution import check_correctness
3
4	problems = read_problems()
5
6	def evaluate_code_generation(model_fn, num_samples=1):
7	total = 0
8	passed = 0
9	for task_id, problem in problems.items():
10	prompt = problem["prompt"]
11	test = problem["test"]
12	entry_point = problem["entry_point"]
13
14	completions = []
15	for _ in range(num_samples):
16	completion = model_fn(prompt)
17	completions.append(completion)
18
19	# Check each completion
20	for completion in completions:
21	result = check_correctness(
22	problem, completion, timeout=5.0
23	)
24	if result["passed"]:
25	passed += 1
26	break
27
28	total += 1
29
30	pass_at_k = passed / total
31	print(f"pass@1: {pass_at_k:.3f}")
32	return pass_at_k

LLM-as-Judge

LLM-as-Judge uses a strong model (typically GPT-4 or Claude 3.5) to evaluate model outputs. This approach measures subjective qualities like helpfulness, harmlessness, instruction following, and writing quality that automated metrics cannot capture. Research shows that frontier models as judges correlate well with human judgments for many tasks.

llm-as-judge.py

Python

1	from openai import OpenAI
2
3	client = OpenAI()
4
5	JUDGE_SYSTEM_PROMPT = """You are an expert evaluator of LLM responses.
6	Rate the response on these criteria from 1-5:
7
8	1. Helpfulness: Does the response address the user's question?
9	2. Accuracy: Is the information factually correct?
10	3. Clarity: Is the response well-structured and easy to understand?
11	4. Safety: Does the response avoid harmful or inappropriate content?
12
13	Provide a score breakdown and brief justification."""
14
15	def evaluate_with_llm(prompt: str, response: str) -> dict:
16	eval_prompt = f"""User prompt: {prompt}
17	Model response: {response}
18
19	Evaluate the response."""
20
21	result = client.chat.completions.create(
22	model="gpt-4o",
23	messages=[
24	{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
25	{"role": "user", "content": eval_prompt}
26	],
27	temperature=0.2,
28	response_format={"type": "json_object"}
29	)
30	return result.choices[0].message.content
31
32	# Pairwise comparison (A vs B)
33	def compare_responses(prompt: str, response_a: str, response_b: str) -> str:
34	comparison_prompt = f"""Prompt: {prompt}
35
36	Response A: {response_a}
37
38	Response B: {response_b}
39
40	Which response is better? Choose A or B and explain why."""
41
42	result = client.chat.completions.create(
43	model="gpt-4o",
44	messages=[{"role": "user", "content": comparison_prompt}],
45	temperature=0.2
46	)
47	return result.choices[0].message.content

⚠

warning

LLM-as-Judge has known biases: it favors longer responses, its own style, and models it was trained on. It exhibits position bias (preferring the first presented response in pairwise comparisons). Mitigate by: randomizing response order, using calibrated rubrics with examples, and validating judge correlations with human evaluations periodically.

Metrics

Different tasks require different evaluation metrics. Understanding the strengths and limitations of each metric prevents misinterpretation of results.

Metric	Task	Range	Notes
Accuracy	Classification, MCQ	0-1	Simple, interpretable; insensitive to confidence
F1 Score	Classification (imbalanced)	0-1	Harmonic mean of precision and recall
BLEU	Translation	0-100	N-gram overlap; poor for open-ended generation
ROUGE	Summarization	0-1	Recall-oriented; multiple variants (ROUGE-1, L)
Perplexity	Language modeling	1 - inf	Exponential of cross-entropy; lower is better
pass@k	Code generation	0-1	Probability at least one of k samples passes tests

metrics.py

Python

1	import evaluate
2
3	# Load metrics from Hugging Face evaluate
4	bleu = evaluate.load("bleu")
5	rouge = evaluate.load("rouge")
6	perplexity = evaluate.load("perplexity", module_type="measurement")
7
8	# BLEU for translation tasks
9	predictions = ["The cat is on the mat."]
10	references = [["The cat sits on the mat."]]
11	bleu_score = bleu.compute(predictions=predictions, references=references)
12	print(f"BLEU: {bleu_score['bleu']:.3f}")
13
14	# ROUGE for summarization
15	summary = "The study found significant improvements in patient outcomes."
16	reference_summary = "Patient outcomes improved significantly in the study."
17	rouge_score = rouge.compute(
18	predictions=[summary],
19	references=[reference_summary]
20	)
21	print(f"ROUGE-1 F1: {rouge_score['rouge1']:.3f}")
22
23	# Perplexity for language models
24	perplexity_results = perplexity.compute(
25	model_id="meta-llama/Llama-3.2-3B",
26	input_texts=["The future of AI is bright and full of possibilities."]
27	)
28	print(f"Perplexity: {perplexity_results['mean_perplexity']:.2f}")

Custom Evaluation Pipelines

Production systems need custom evaluations tailored to their specific requirements. Build an evaluation harness that runs automatically on every model update, tracks metrics over time, and alerts on regressions.

custom-eval.py

Python

1	from dataclasses import dataclass
2	from typing import Callable, List
3	import json
4
5	@dataclass
6	class EvalCase:
7	prompt: str
8	expected_behavior: str
9	rubric: dict
10	weight: float = 1.0
11
12	class EvalHarness:
13	def __init__(self, judge_fn: Callable):
14	self.cases: List[EvalCase] = []
15	self.judge_fn = judge_fn
16
17	def add_case(self, case: EvalCase):
18	self.cases.append(case)
19
20	def run(self, model_fn: Callable) -> dict:
21	results = []
22	total_score = 0.0
23	total_weight = 0.0
24
25	for case in self.cases:
26	output = model_fn(case.prompt)
27	evaluation = self.judge_fn(
28	case.prompt, output, case.rubric
29	)
30	score = evaluation["score"] * case.weight
31	total_score += score
32	total_weight += case.weight
33	results.append({
34	"prompt": case.prompt,
35	"output": output,
36	"score": evaluation["score"],
37	"justification": evaluation["justification"]
38	})
39
40	return {
41	"average_score": total_score / total_weight,
42	"results": results
43	}
44
45	# Usage
46	harness = EvalHarness(judge_fn=llm_judge)
47	harness.add_case(EvalCase(
48	prompt="Explain backpropagation",
49	expected_behavior="Technical accuracy",
50	rubric={"accuracy": 3, "clarity": 2}
51	))
52	results = harness.run(my_model)
53	print(f"Average score: {results['average_score']:.2f}")

✓

best practice

Build a regression test suite of 50-100 high-quality eval cases that cover the core behaviors your model must exhibit. Run this suite on every model checkpoint during training and before any deployment. A regression in these core evals is a blocking issue that must be investigated before proceeding.

$Blueprint — Engineering Documentation·Section ID: AI-EVAL-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor