Fine-tuning Models

Fine-tuning adapts a pre-trained language model to a specific task or domain by continuing the training process on a curated dataset. Unlike prompting or RAG, fine-tuning modifies the model weights themselves, resulting in permanent behavioral changes that persist across all inputs.

Fine-tuning is the most effective way to imbue a model with deep domain knowledge, enforce complex formatting rules, or achieve consistent output style. However, it requires careful data curation, significant compute resources, and thorough evaluation to avoid catastrophic forgetting or overfitting.

This guide covers the full fine-tuning lifecycle: deciding when to fine-tune, preparing datasets, choosing between full fine-tuning and parameter-efficient methods, training with SFT, tuning hyperparameters, evaluating results, and deploying the final model.

Choosing the right adaptation strategy depends on your use case, data availability, latency requirements, and desired behavior change.

Full fine-tuning updates all model parameters during training. It achieves the highest quality adaptation but is extremely expensive — training a 7B parameter model requires ~60 GB of GPU memory per batch. Parameter-efficient methods reduce this dramatically.

LoRA (Low-Rank Adaptation)

LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices into attention layers. This reduces trainable parameters by 10,000x while maintaining 90-95% of full fine-tuning quality. The LoRA weights are typically merged into the base model at inference for zero additional latency.

QLoRA — Quantized LoRA

QLoRA extends LoRA by quantizing the base model to 4-bit using NormalFloat4 (NF4) quantization. This enables fine-tuning models like Llama 3.2 70B on a single 48 GB GPU. QLoRA introduces paged optimizers to handle gradient checkpointing memory spikes and double quantization to reduce the memory overhead of quantization constants.

Supervised Fine-Tuning (SFT) trains the model on input-output pairs to learn desired behavior. The training objective is standard autoregressive language modeling — cross-entropy loss on the target tokens. SFT is the most common fine-tuning paradigm and forms the foundation for instruction-tuning, chat fine-tuning, and task-specific adaptation.

The training loop processes batches of examples where each example consists of a prompt (input) and a completion (target). The loss is computed only on the completion tokens, not the prompt tokens, preventing the model from being penalized for not predicting the input.

SFT Training Loop with Label Masking

The quality of your fine-tuning dataset is the single most important factor in the success of fine-tuning. A small, high-quality dataset consistently outperforms a large, noisy one. The canonical recommendation is 500-5,000 high-quality examples for most SFT tasks.

Fine-tuning hyperparameters significantly impact convergence speed, model quality, and training stability. The following table shows recommended starting values for LoRA fine-tuning of 1-13B parameter models.

Evaluating a fine-tuned model requires more than tracking loss curves. You need task-specific metrics, human evaluation, and comparison to the base model. Always evaluate on a held-out test set that was not seen during training.

Fine-tuned models can be deployed via inference endpoints, serverless GPU functions, or embedded in applications. For LoRA adapters, you can merge weights into the base model for zero-overhead inference or keep them separate for modular switching.