Deep Learning
Topic 6 · Deep Learning · ⏱ about 6 h

Optimisers & Learning-Rate Schedules

Mini-batch SGD, momentum, RMSProp, Adam and AdamW, why the learning rate matters most, warm-up and cosine schedules, and gradient clipping.

Show:
In this topic
Beginner

Mini-batch stochastic gradient descent

Training minimises the average loss over the training set. Computing the exact gradient needs a pass over all the data, which is too slow, so networks use mini-batch SGD: take a random batch of B examples (typically 32 to 1,024), compute the gradient on it, and step θ←θ−η g\theta\leftarrow\theta-\eta\,g. One pass over the data is an epoch; reshuffle every epoch.

The batch gradient is a noisy estimate of the true gradient. That noise is not all bad: it helps escape sharp minima and acts as a mild regulariser. Larger batches give less noisy gradients and use the GPU better, but each epoch has fewer steps, and very large batches often generalise slightly worse unless the learning rate and schedule are adjusted.

Beginner

Momentum: rolling through ravines

Loss surfaces often look like long narrow ravines: steep across, shallow along. Plain SGD bounces between the walls and creeps along the floor. Momentum keeps a running "velocity", an exponentially weighted average of past gradients:

v←βv+g,θ←θ−η v,β≈0.9.v\leftarrow\beta v+g,\qquad \theta\leftarrow\theta-\eta\,v,\qquad\beta\approx0.9.

Components that keep pointing the same way (along the ravine) add up; components that flip sign (across it) cancel. Nesterov momentum evaluates the gradient slightly ahead, at θ−ηβv\theta-\eta\beta v, which is usually a little better. SGD with Nesterov momentum and a good schedule is still the optimiser of choice for many CNNs.

Beginner

Adaptive learning rates: RMSProp, Adam, AdamW

Different parameters need different step sizes: rare features get small, infrequent gradients; others get large ones. RMSProp divides each parameter's step by a running root-mean-square of its recent gradients. Adam combines this with momentum and corrects both averages for starting at zero:

m←β1m+(1−β1)g,s←β2s+(1−β2)g2,m^=m1−β1t, s^=s1−β2t,θ←θ−η m^s^+ϵm\leftarrow\beta_1m+(1-\beta_1)g,\quad s\leftarrow\beta_2s+(1-\beta_2)g^2,\quad \hat m=\frac{m}{1-\beta_1^t},\ \hat s=\frac{s}{1-\beta_2^t},\quad \theta\leftarrow\theta-\eta\,\frac{\hat m}{\sqrt{\hat s}+\epsilon}

with defaults β1=0.9, β2=0.999, ϵ=10−8\beta_1=0.9,\ \beta_2=0.999,\ \epsilon=10^{-8}. Adam is forgiving and works well with little tuning, which makes it the default for transformers and most new projects (a learning rate of 3×10−43\times10^{-4} is a famous first guess).

AdamW fixes how Adam handles weight decay. Adding λθ\lambda\theta to the gradient (classic L2) gets divided by s^\sqrt{\hat s} too, so heavily-updated weights are barely decayed. AdamW instead decouples it: θ←θ−η(m^/(s^+ϵ)+λθ)\theta\leftarrow\theta-\eta(\hat m/(\sqrt{\hat s}+\epsilon)+\lambda\theta). Use AdamW, typically with λ\lambda between 0.01 and 0.1, and don't decay biases and normalisation parameters.

Try it

Try it: an optimiser race

Four optimisers start from the same point on a narrow ravine (the minimum is the cross). Change the learning rate and add gradient noise like a mini-batch would. Which one is robust to the learning rate?

Interactive: this part runs in your browser.

Beginner

The learning rate is the most important hyperparameter

Too high: the loss oscillates, spikes or becomes NaN. Too low: it decreases painfully slowly and may stall. If you only tune one thing, tune the learning rate, on a log scale (1e-4, 3e-4, 1e-3, 3e-3…).

A quick way to find a good range is the learning-rate range test: train for a few hundred steps while increasing the learning rate exponentially from 1e-7 to 1, and plot the loss. Pick a value about ten times smaller than where the loss is lowest, before it blows up.

The rate also should change during training: large early to make progress, small late to settle into a good minimum. Common schedules:

  • Step decay: divide by 10 at fixed epochs (classic for ResNets).
  • Cosine decay: smoothly from the peak to near zero along half a cosine wave.
  • Linear warm-up for the first few percent of steps, then decay. Warm-up avoids huge, unstable early updates while Adam's statistics and normalisation layers settle; it is standard for transformers.
  • One-cycle: warm up to a high rate, then anneal down: often trains fast.
Try it

Try it: learning-rate schedules

Compare the shapes of common schedules over a training run, and the effect of warm-up length.

Interactive: this part runs in your browser.

Beginner

Gradient clipping and accumulation

A single bad batch can produce a huge gradient that throws the parameters far away (loss spikes, NaNs), especially in RNNs and transformers. Gradient clipping by global norm rescales the whole gradient when its norm exceeds a threshold c (often 1.0): g←g⋅min⁡(1, c/∥g∥)g\leftarrow g\cdot\min(1,\ c/\lVert g\rVert). The direction is kept; only the length is capped.

Gradient accumulation simulates a larger batch when it does not fit in memory: run k small batches, add up their gradients (dividing each loss by k), and step once. Remember that batch-normalisation statistics still see only the small batches.

Try it

Code example: write SGD-momentum and Adam

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Try it

Code example: warm-up, cosine decay and clipping

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: optimiser, schedule and clipping in PyTorch

The standard recipe for a transformer-style model.

Python · PyTorch
import torch, math
decay, no_decay = [], []
for n, p in model.named_parameters():
    (no_decay if p.ndim < 2 else decay).append(p)        # no decay on biases / norm weights
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.05},
                         {"params": no_decay, "weight_decay": 0.0}], lr=3e-4, betas=(0.9, 0.95))
total, warm = 10_000, 500
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: s / warm if s < warm
                     else 0.5 * (1 + math.cos(math.pi * (s - warm) / (total - warm))))
for step, (xb, yb) in zip(range(total), loader):
    loss = loss_fn(model(xb), yb)
    opt.zero_grad(set_to_none=True)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    opt.step(); sched.step()
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. What does momentum do in a narrow ravine?

2. Why does Adam divide by √ŝ?

3. What is the difference between Adam with L2 and AdamW?

4. Your loss becomes NaN in the first 50 steps. The most likely fix is…

5. Why use a learning-rate warm-up for transformers?

6. Clipping the gradient by global norm…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready