How LLMs Are Trained, Adapted & Aligned
Pretraining data and scaling laws, instruction tuning, preference alignment (RLHF and DPO), parameter-efficient fine-tuning, evaluation and contamination, safety, and how to choose between prompting, RAG and fine-tuning.
Stage 1: pretraining
An LLM starts as a decoder-only transformer trained on next-token prediction over trillions of tokens of web pages, books, code and other text. The data pipeline is most of the work: deduplication, quality filtering, removing personal information and toxic content, balancing languages and domains. Scaling laws show loss falling smoothly as a power law in model size, data and compute; the "Chinchilla" result found that, for a fixed compute budget, many models had been too big and trained on too little data (roughly 20 tokens per parameter is compute-optimal; many current models are trained far longer because smaller models are cheaper to run). The result is a base model: excellent at continuing text, not yet at following instructions.
Watch it: From base model to assistant: pretraining, SFT and preference tuning
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Stage 2: instruction tuning
Supervised fine-tuning (SFT) on thousands to millions of (instruction, good response) pairs, formatted with a chat template (system / user / assistant turns marked by special tokens), turns the base model into an assistant. The loss is computed only on the response tokens. Quality and diversity of examples matter more than quantity: a few thousand excellent examples can change behaviour a lot.
Stage 3: preference alignment (RLHF, DPO)
Many qualities (helpful, honest, harmless, well-formatted) are easier to compare than to demonstrate. People (or AI feedback under written principles) see two responses to the same prompt and pick the better one.
- RLHF: train a reward model to predict which response humans prefer (a Bradley–Terry model: P(A ≻ B) = σ(r(A) − r(B))), then optimise the LLM with reinforcement learning (PPO) to get high reward, with a KL penalty that keeps it close to the SFT model so it does not exploit flaws in the reward model ("reward hacking").
- DPO (direct preference optimisation) skips the separate reward model and RL loop: it trains directly on preference pairs with a simple classification-style loss, using the frozen SFT model as a reference:
where is the preferred and the rejected response. It is simpler and more stable, and widely used.
Alignment also includes safety training (refusing harmful requests without over-refusing) and is evaluated with red-teaming.
Try it: the DPO loss
Move the policy's log-probabilities for the preferred and rejected responses, relative to the reference model, and see the DPO loss and its gradient.
Interactive: this part runs in your browser.
Adapting an LLM: prompt, RAG or fine-tune?
| Need | First choice |
|---|---|
| A task the model can already do with clear instructions | Prompting (with examples), plus an evaluation set |
| Answers grounded in your documents, which change over time, with citations | RAG (see the LLM topic) |
| A consistent format, style or narrow skill; lower cost/latency with a smaller model | Fine-tuning (usually LoRA/QLoRA on a small open model) |
| New factual knowledge | RAG rather than fine-tuning: fine-tuning teaches behaviour better than facts |
Fine-tuning practicalities: a few hundred to a few thousand high-quality examples in the model's chat template; LoRA with learning rate around 1e-4 to 2e-4; hold out an evaluation set; watch for forgetting general abilities; never train on personal data without a lawful basis and de-identification.
Evaluating LLMs honestly
- Benchmarks (MMLU, GSM8K, HumanEval…) measure general abilities, but public benchmarks leak into training data: contamination inflates scores. Prefer held-out, recent or private test sets.
- Your task, your data: a versioned evaluation set with rubrics; measure accuracy, hallucination rate, refusal rate, format errors, latency and cost.
- Variance: sampling makes outputs vary; run several samples or use greedy decoding for evaluation; report confidence intervals.
- Safety and fairness: red-team for harmful outputs, prompt injection and privacy leaks; check quality across languages and user groups.
Code example: preference pairs and the DPO loss
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: LoRA SFT and DPO with TRL
The two most common post-training recipes for small open models (notebook 12 runs the first).
from trl import SFTTrainer, SFTConfig, DPOTrainer, DPOConfig
from peft import LoraConfig
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")
# 1) instruction tuning: dataset rows like {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
sft = SFTTrainer(model="Qwen/Qwen2.5-0.5B-Instruct", train_dataset=sft_ds, peft_config=lora,
args=SFTConfig(output_dir="sft", learning_rate=2e-4, num_train_epochs=2, per_device_train_batch_size=8))
sft.train()
# 2) preference tuning: rows like {"prompt": ..., "chosen": ..., "rejected": ...}
dpo = DPOTrainer(model=sft.model, ref_model=None, train_dataset=pref_ds, processing_class=sft.tokenizer, peft_config=lora,
args=DPOConfig(output_dir="dpo", beta=0.1, learning_rate=5e-6, num_train_epochs=1))
dpo.train()Quiz
1. A base model (after pretraining only) is good at…
2. In SFT, the loss is usually computed on…
3. What does DPO need for each training example?
4. Why does RLHF include a KL penalty to the SFT model?
5. Clinic policies change monthly and answers must cite them. Best approach?
6. Your model scores 92% on a public benchmark released two years ago. A concern is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Training language models to follow instructions (Ouyang et al., 2022) ↗ InstructGPT: SFT + RLHF
- Direct Preference Optimization (Rafailov et al., 2023) ↗ The DPO paper
- Training Compute-Optimal LLMs (Hoffmann et al., 2022) ↗ Chinchilla scaling laws
- TRL documentation ↗ SFT, DPO and RLHF trainers