Dataset Curation
Dataset curation is the process of collecting, cleaning, filtering, and formatting data for LLM training. The quality of your dataset is the single most important factor determining model performance — a small, clean dataset consistently outperforms a large, noisy one across all training paradigms (pre-training, SFT, preference optimization).
Modern LLM datasets span orders of magnitude: pre-training datasets contain trillions of tokens from web crawls, while SFT datasets are typically 1K-100K high-quality examples. Each stage of training requires different curation strategies, quality thresholds, and data formats.
This guide covers the end-to-end dataset curation pipeline: collection strategies, quality filtering techniques, deduplication algorithms (MinHash, exact match), synthetic data generation, data augmentation, formatting standards (ChatML, ShareGPT), and dataset management best practices.
Data collection depends on the training stage. For pre-training, web crawls (Common Crawl, CCNet) are the primary source. For SFT, you need demonstration data — human-written instruction-response pairs or distilled outputs from stronger models (e.g., GPT-4). For preference optimization, you need pairwise comparisons.
| Source | Training Stage | Quality | Volume |
|---|---|---|---|
| Common Crawl | Pre-training | Low (raw) | Petabytes |
| Human Annotation | SFT / RLHF | Highest | 1K-100K |
| Model Distillation | SFT | High | 10K-1M |
| Open Datasets | All stages | Variable | Various |
| User Logs | Fine-tuning | Medium | 100K-10M |
best practice
Raw data contains low-quality content that degrades model performance if included in training. Filtering removes toxic content, spam, low-information text, and examples with incorrect formatting. Multiple filtering passes with different heuristics are more effective than a single complex filter.
Filtering Pipeline
| 1 | import re |
| 2 | from typing import List, Dict |
| 3 | |
| 4 | class QualityFilter: |
| 5 | def __init__(self): |
| 6 | self.min_length = 50 |
| 7 | self.max_length = 10000 |
| 8 | self.min_words = 10 |
| 9 | self.max_repetition_ratio = 0.3 |
| 10 | self.toxic_patterns = [ |
| 11 | r'(?i)\b(spam|click here|buy now|limited offer)\b', |
| 12 | ] |
| 13 | self.url_threshold = 0.5 # Max fraction of URLs in text |
| 14 | |
| 15 | def filter(self, examples: List[Dict]) -> List[Dict]: |
| 16 | filtered = [] |
| 17 | for ex in examples: |
| 18 | text = ex.get("text", "") |
| 19 | if not text: |
| 20 | continue |
| 21 | if len(text) < self.min_length or len(text) > self.max_length: |
| 22 | continue |
| 23 | words = text.split() |
| 24 | if len(words) < self.min_words: |
| 25 | continue |
| 26 | if self.repetition_ratio(text) > self.max_repetition_ratio: |
| 27 | continue |
| 28 | if self.has_toxic_content(text): |
| 29 | continue |
| 30 | if self.url_ratio(text) > self.url_threshold: |
| 31 | continue |
| 32 | filtered.append(ex) |
| 33 | return filtered |
| 34 | |
| 35 | def repetition_ratio(self, text: str) -> float: |
| 36 | ngrams = set() |
| 37 | total = 0 |
| 38 | words = text.split() |
| 39 | for i in range(len(words) - 4): |
| 40 | ngram = " ".join(words[i:i+5]) |
| 41 | ngrams.add(ngram) |
| 42 | total += 1 |
| 43 | return 1 - len(ngrams) / max(total, 1) |
| 44 | |
| 45 | def has_toxic_content(self, text: str) -> bool: |
| 46 | return any( |
| 47 | re.search(p, text) for p in self.toxic_patterns |
| 48 | ) |
| 49 | |
| 50 | def url_ratio(self, text: str) -> float: |
| 51 | urls = len(re.findall(r'https?://\S+', text)) |
| 52 | return urls / max(len(text.split()), 1) |
Language & Content Filtering
Use fastText language identification to filter for target languages. Remove documents with low perplexity under a language model (these are often repetitive or templated text). For SFT data, verify that every example has a properly formatted instruction and response pair.
| 1 | import fasttext |
| 2 | |
| 3 | # Load language identification model |
| 4 | lid_model = fasttext.load_model("lid.176.bin") |
| 5 | |
| 6 | def filter_language(text: str, target_lang: str = "en", threshold: float = 0.5) -> bool: |
| 7 | predictions = lid_model.predict(text.replace("\n", " ")) |
| 8 | lang = predictions[0][0].replace("__label__", "") |
| 9 | confidence = predictions[1][0] |
| 10 | return lang == target_lang and confidence >= threshold |
| 11 | |
| 12 | # Perplexity-based filtering |
| 13 | from transformers import AutoModelForMaskedLM, AutoTokenizer |
| 14 | |
| 15 | mlm_model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased") |
| 16 | mlm_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") |
| 17 | |
| 18 | def compute_perplexity(text: str) -> float: |
| 19 | inputs = mlm_tokenizer(text, return_tensors="pt", truncation=True, max_length=512) |
| 20 | outputs = mlm_model(**inputs, labels=inputs["input_ids"]) |
| 21 | return torch.exp(outputs.loss).item() |
| 22 | |
| 23 | # Filter out very low perplexity (template) and very high perplexity (gibberish) |
| 24 | filtered = [ |
| 25 | ex for ex in data |
| 26 | if 20 < compute_perplexity(ex["text"]) < 500 |
| 27 | ] |
Duplicate data inflates training metrics without providing new information and can cause the model to over-memorize specific examples. Deduplication operates at multiple levels: exact document match, near-duplicate (fuzzy) match, and paragraph/sentence-level deduplication.
Exact Deduplication
The simplest and fastest approach — remove documents with identical content using hash-based comparison. Always normalize text (lowercase, strip whitespace) before hashing. Use SHA-256 or xxHash for fast hashing at scale.
| 1 | import hashlib |
| 2 | |
| 3 | def normalize(text: str) -> str: |
| 4 | return " ".join(text.lower().strip().split()) |
| 5 | |
| 6 | def exact_dedup(examples: list, key: str = "text") -> list: |
| 7 | seen = set() |
| 8 | unique = [] |
| 9 | for ex in examples: |
| 10 | normalized = normalize(ex[key]) |
| 11 | doc_hash = hashlib.sha256(normalized.encode()).hexdigest() |
| 12 | if doc_hash not in seen: |
| 13 | seen.add(doc_hash) |
| 14 | unique.append(ex) |
| 15 | return unique |
| 16 | |
| 17 | # Fast hash with xxhash (better for large datasets) |
| 18 | import xxhash |
| 19 | |
| 20 | def exact_dedup_fast(examples: list) -> list: |
| 21 | seen = set() |
| 22 | unique = [] |
| 23 | for ex in examples: |
| 24 | h = xxhash.xxh64(ex["text"]).hexdigest() |
| 25 | if h not in seen: |
| 26 | seen.add(h) |
| 27 | unique.append(ex) |
| 28 | return unique |
| 29 | |
| 30 | print(f"Removed {len(examples) - len(exact_dedup_fast(examples))} exact duplicates") |
MinHash — Near-Duplicate Detection
MinHash with Locality-Sensitive Hashing (LSH) detects documents that are similar but not identical — e.g., articles with slightly different headlines, versions of the same page with different timestamps, or paraphrased content. This is critical for web-scale deduplication where exact match catches only a fraction of duplicates.
| 1 | from datasketch import MinHash, MinHashLSH |
| 2 | import re |
| 3 | |
| 4 | def shingle_text(text: str, k: int = 5) -> set: |
| 5 | words = re.findall(r'\w+', text.lower()) |
| 6 | return set( |
| 7 | " ".join(words[i:i+k]) |
| 8 | for i in range(len(words) - k + 1) |
| 9 | ) |
| 10 | |
| 11 | def build_minhash(text: str, num_perm: int = 128) -> MinHash: |
| 12 | m = MinHash(num_perm=num_perm) |
| 13 | for shingle in shingle_text(text): |
| 14 | m.update(shingle.encode()) |
| 15 | return m |
| 16 | |
| 17 | # Create LSH index |
| 18 | lsh = MinHashLSH(threshold=0.7, num_perm=128) |
| 19 | |
| 20 | for i, ex in enumerate(examples): |
| 21 | m = build_minhash(ex["text"]) |
| 22 | lsh.insert(f"doc_{i}", m) |
| 23 | |
| 24 | # Find near-duplicates |
| 25 | near_duplicates = set() |
| 26 | for i, ex in enumerate(examples): |
| 27 | m = build_minhash(ex["text"]) |
| 28 | results = lsh.query(m) |
| 29 | if len(results) > 1: |
| 30 | near_duplicates.add(i) |
| 31 | |
| 32 | print(f"Found {len(near_duplicates)} near-duplicate documents") |
| 33 | filtered = [ex for i, ex in enumerate(examples) if i not in near_duplicates] |
pro tip
Synthetic data generation uses existing models to create training examples. This is valuable for scenarios where human data is scarce, expensive, or impossible to collect. Common approaches include self-instruct, evol-instruct, back-translation, and contradiction generation.
| 1 | from openai import OpenAI |
| 2 | import json |
| 3 | |
| 4 | client = OpenAI() |
| 5 | |
| 6 | def generate_sft_examples(topic: str, n: int = 100) -> list: |
| 7 | prompt = f"""Generate {n} high-quality instruction-response pairs about {topic}. |
| 8 | |
| 9 | Each example should be a JSON object with: |
| 10 | - "instruction": a clear, specific question or task |
| 11 | - "output": a detailed, accurate response |
| 12 | |
| 13 | Make the instructions diverse: factual questions, how-to guides, |
| 14 | explanations, comparisons, and creative tasks. Ensure responses |
| 15 | are technically accurate and at least 3-4 sentences long.""" |
| 16 | |
| 17 | response = client.chat.completions.create( |
| 18 | model="gpt-4o", |
| 19 | messages=[{"role": "user", "content": prompt}], |
| 20 | response_format={"type": "json_object"}, |
| 21 | temperature=0.8 |
| 22 | ) |
| 23 | |
| 24 | data = json.loads(response.choices[0].message.content) |
| 25 | return data["examples"] |
| 26 | |
| 27 | # Evol-Instruct: Iteratively increase difficulty |
| 28 | def evolve_instruction(instruction: str) -> str: |
| 29 | prompt = f"""Rewrite the following instruction to make it more |
| 30 | challenging and specific. Add constraints, edge cases, or depth: |
| 31 | |
| 32 | Original: {instruction} |
| 33 | |
| 34 | Improved:""" |
| 35 | response = client.chat.completions.create( |
| 36 | model="gpt-4o", |
| 37 | messages=[{"role": "user", "content": prompt}], |
| 38 | temperature=0.7 |
| 39 | ) |
| 40 | return response.choices[0].message.content.strip() |
warning
Consistent formatting is essential for training stability and model performance. Most modern LLMs use chat templates to structure conversations. The two dominant formats are ChatML (used by GPT-4, Llama 3, Mistral) and ShareGPT (used by Vicuna, some community models).
| 1 | # ChatML format |
| 2 | chatml_format = { |
| 3 | "messages": [ |
| 4 | {"role": "system", "content": "You are an expert programmer."}, |
| 5 | {"role": "user", "content": "Write a Python function to reverse a linked list."}, |
| 6 | {"role": "assistant", "content": "Here's a recursive solution:\n\ndef reverse(head):\n if not head or not head.next:\n return head\n new_head = reverse(head.next)\n head.next.next = head\n head.next = None\n return new_head"} |
| 7 | ] |
| 8 | } |
| 9 | |
| 10 | # Convert to tokenizer's expected format using apply_chat_template |
| 11 | from transformers import AutoTokenizer |
| 12 | |
| 13 | tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B") |
| 14 | formatted = tokenizer.apply_chat_template( |
| 15 | chatml_format["messages"], |
| 16 | tokenize=False, |
| 17 | add_generation_prompt=True |
| 18 | ) |
| 19 | |
| 20 | # For training: create packed sequences efficiently |
| 21 | def pack_sequences(dataset, tokenizer, max_length=2048): |
| 22 | all_tokens = [] |
| 23 | for example in dataset: |
| 24 | tokens = tokenizer( |
| 25 | example["text"], |
| 26 | truncation=True, |
| 27 | max_length=max_length |
| 28 | )["input_ids"] |
| 29 | all_tokens.extend(tokens) |
| 30 | |
| 31 | packed = [] |
| 32 | for i in range(0, len(all_tokens), max_length): |
| 33 | chunk = all_tokens[i:i + max_length] |
| 34 | packed.append({"input_ids": chunk, "labels": chunk.copy()}) |
| 35 | |
| 36 | return packed |
best practice
Dataset curation carries significant ethical responsibilities. Biases present in training data are learned and amplified by the model. Privacy violations in training data can lead to data leakage at inference. Always document data provenance, obtain consent for personal data, and conduct bias audits before training.