Natural Language Processing
Topic 7 · NLP · ⏱ about 8 h

How LLMs Are Trained, Adapted & Aligned

Pretraining data and scaling laws, instruction tuning, preference alignment (RLHF and DPO), parameter-efficient fine-tuning, evaluation and contamination, safety, and how to choose between prompting, RAG and fine-tuning.

Show:
In this topic
Beginner

Stage 1: pretraining

An LLM starts as a decoder-only transformer trained on next-token prediction over trillions of tokens of web pages, books, code and other text. The data pipeline is most of the work: deduplication, quality filtering, removing personal information and toxic content, balancing languages and domains. Scaling laws show loss falling smoothly as a power law in model size, data and compute; the "Chinchilla" result found that, for a fixed compute budget, many models had been too big and trained on too little data (roughly 20 tokens per parameter is compute-optimal; many current models are trained far longer because smaller models are cheaper to run). The result is a base model: excellent at continuing text, not yet at following instructions.

Beginner

Watch it: From base model to assistant: pretraining, SFT and preference tuning

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Stage 2: instruction tuning

Supervised fine-tuning (SFT) on thousands to millions of (instruction, good response) pairs, formatted with a chat template (system / user / assistant turns marked by special tokens), turns the base model into an assistant. The loss is computed only on the response tokens. Quality and diversity of examples matter more than quantity: a few thousand excellent examples can change behaviour a lot.

Beginner

Stage 3: preference alignment (RLHF, DPO)

Many qualities (helpful, honest, harmless, well-formatted) are easier to compare than to demonstrate. People (or AI feedback under written principles) see two responses to the same prompt and pick the better one.

  • RLHF: train a reward model to predict which response humans prefer (a Bradley–Terry model: P(A ≻ B) = σ(r(A) − r(B))), then optimise the LLM with reinforcement learning (PPO) to get high reward, with a KL penalty that keeps it close to the SFT model so it does not exploit flaws in the reward model ("reward hacking").
  • DPO (direct preference optimisation) skips the separate reward model and RL loop: it trains directly on preference pairs with a simple classification-style loss, using the frozen SFT model as a reference:
    L=−log⁡σ(β[(log⁡πθ(yw∣x)−log⁡πref(yw∣x))−(log⁡πθ(yl∣x)−log⁡πref(yl∣x))])\mathcal L=-\log\sigma\Big(\beta\big[(\log\pi_\theta(y_w|x)-\log\pi_\text{ref}(y_w|x))-(\log\pi_\theta(y_l|x)-\log\pi_\text{ref}(y_l|x))\big]\Big)
    where ywy_w is the preferred and yly_l the rejected response. It is simpler and more stable, and widely used.

Alignment also includes safety training (refusing harmful requests without over-refusing) and is evaluated with red-teaming.

Try it

Try it: the DPO loss

Move the policy's log-probabilities for the preferred and rejected responses, relative to the reference model, and see the DPO loss and its gradient.

Interactive: this part runs in your browser.

Beginner

Adapting an LLM: prompt, RAG or fine-tune?

NeedFirst choice
A task the model can already do with clear instructionsPrompting (with examples), plus an evaluation set
Answers grounded in your documents, which change over time, with citationsRAG (see the LLM topic)
A consistent format, style or narrow skill; lower cost/latency with a smaller modelFine-tuning (usually LoRA/QLoRA on a small open model)
New factual knowledgeRAG rather than fine-tuning: fine-tuning teaches behaviour better than facts

Fine-tuning practicalities: a few hundred to a few thousand high-quality examples in the model's chat template; LoRA with learning rate around 1e-4 to 2e-4; hold out an evaluation set; watch for forgetting general abilities; never train on personal data without a lawful basis and de-identification.

Beginner

Evaluating LLMs honestly

  • Benchmarks (MMLU, GSM8K, HumanEval…) measure general abilities, but public benchmarks leak into training data: contamination inflates scores. Prefer held-out, recent or private test sets.
  • Your task, your data: a versioned evaluation set with rubrics; measure accuracy, hallucination rate, refusal rate, format errors, latency and cost.
  • Variance: sampling makes outputs vary; run several samples or use greedy decoding for evaluation; report confidence intervals.
  • Safety and fairness: red-team for harmful outputs, prompt injection and privacy leaks; check quality across languages and user groups.
Try it

Code example: preference pairs and the DPO loss

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: LoRA SFT and DPO with TRL

The two most common post-training recipes for small open models (notebook 12 runs the first).

Python · TRL + PEFT
from trl import SFTTrainer, SFTConfig, DPOTrainer, DPOConfig
from peft import LoraConfig
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")
# 1) instruction tuning: dataset rows like {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
sft = SFTTrainer(model="Qwen/Qwen2.5-0.5B-Instruct", train_dataset=sft_ds, peft_config=lora,
                 args=SFTConfig(output_dir="sft", learning_rate=2e-4, num_train_epochs=2, per_device_train_batch_size=8))
sft.train()
# 2) preference tuning: rows like {"prompt": ..., "chosen": ..., "rejected": ...}
dpo = DPOTrainer(model=sft.model, ref_model=None, train_dataset=pref_ds, processing_class=sft.tokenizer, peft_config=lora,
                 args=DPOConfig(output_dir="dpo", beta=0.1, learning_rate=5e-6, num_train_epochs=1))
dpo.train()
This code needs TRL, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. A base model (after pretraining only) is good at…

2. In SFT, the loss is usually computed on…

3. What does DPO need for each training example?

4. Why does RLHF include a KL penalty to the SFT model?

5. Clinic policies change monthly and answers must cite them. Best approach?

6. Your model scores 92% on a public benchmark released two years ago. A concern is…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready