SFT / DPO / RLHF

Modern LLMs are trained through a multi-stage pipeline that typically begins with pre-training on massive web corpora, followed by supervised fine-tuning (SFT), and then alignment through preference optimization (DPO, RLHF) or reinforcement learning. Each stage serves a distinct purpose in shaping model behavior.

SFT teaches the model to follow instructions and produce coherent completions. DPO directly optimizes for human preferences without a separate reward model. RLHF introduces a learned reward model to guide the policy through reinforcement learning (PPO). Understanding when and how to use each method is critical for building aligned, capable models.

This guide provides a detailed comparison of these training paradigms, with practical code examples for implementing each approach using established libraries and frameworks.

SFT is the first post-training stage for most LLMs. The pre-trained base model is fine-tuned on high-quality instruction-response pairs to learn the format and style of helpful conversation. SFT uses standard autoregressive language modeling with cross-entropy loss computed only on the response tokens.

Key characteristics of SFT include: teacher forcing (the model sees the correct response during training), maximum likelihood estimation (MLE) objective, and label masking to exclude prompt tokens from loss computation. SFT is simple, stable, and computationally efficient compared to RL-based methods.

DPO is a preference optimization method that directly optimizes the policy from pairwise preference data without training a separate reward model. It reformulates the RLHF objective into a simple binary classification loss that compares the likelihood of chosen vs rejected responses under the trained policy, relative to a reference model.

The DPO loss function takes the form: -log(sigmoid(beta * (log(pi(y_w|x)/pi_ref(y_w|x)) - log(pi(y_l|x)/pi_ref(y_l|x))))) where y_w is the preferred response, y_l is the dispreferred response, pi is the trained policy, and pi_ref is the frozen reference model. The beta parameter controls how far the policy can deviate from the reference.

DPO Training Implementation

RLHF is the three-stage pipeline pioneered by InstructGPT that consists of: (1) SFT on human demonstrations, (2) training a reward model on human preferences, and (3) optimizing the policy against the reward model using PPO. This approach allows the model to generate novel responses that are optimized for human preferences, not just imitate the training data.

Stage 1: Reward Model Training

The reward model is trained on comparison data — pairs of responses where one is preferred over the other. It learns to assign higher scores to preferred responses. The reward model is typically initialized from the same base model as the policy but with a scalar output head replacing the language modeling head.

Stage 2: PPO Training

PPO (Proximal Policy Optimization) optimizes the policy model to maximize the reward signal from the reward model while staying close to the reference model via a KL divergence penalty. This prevents the policy from exploiting the reward model by generating nonsensical but high-scoring text.

Choosing between SFT, DPO, and RLHF depends on your data, compute budget, and quality requirements. SFT is the minimum viable alignment method. DPO offers a practical middle ground. RLHF provides the highest potential quality but at significantly greater complexity and cost.

Preference datasets are the fuel for both DPO and RLHF. Each example consists of a prompt, a chosen (preferred) response, and a rejected (dispreferred) response. High-quality preference data is difficult and expensive to collect, often requiring human annotators with domain expertise.

Common sources of preference data include: human annotation (labelers compare model outputs), AI feedback (another LLM judges outputs), implicit feedback (user clicks, dwell time), and synthetic data (using a reward model to rank candidates).