Machine Learning
Topic 30 ยท Deep learning ยท โฑ about 15 h

Deep Learning in Practice (PyTorch)

Tensors, autograd and the training loop; data loading, GPUs, transfer learning and fine-tuning, with three Colab notebooks.

Show:
In this topic
Beginner

Why a framework, and what PyTorch gives you

Everything in the deep-learning topics (layers, losses, backpropagation, optimisers) can be written in NumPy, as you did for gradient descent in the game. Real networks have millions of parameters, run on GPUs and change shape every week, so teams use a framework. PyTorch is the most widely used in research and increasingly in industry; TensorFlow/Keras and JAX are the main alternatives, and the ideas transfer directly.

PyTorch gives you three things. Tensors: n-dimensional arrays like NumPy's, which can live on a GPU (x.to("cuda")). Autograd: every operation on tensors that require gradients is recorded in a graph, and calling loss.backward() computes the gradient of the loss with respect to every parameter by the chain rule, automatically. Building blocks: layers (nn.Linear, nn.Conv2d), losses, optimisers (torch.optim.AdamW) and data utilities.

import torch w = torch.tensor(2.0, requires_grad=True) x = torch.tensor(3.0) loss = (w * x - 12) ** 2 # forward pass builds the graph loss.backward() # backward pass: d loss / d w print(w.grad) # 2 ยท (wยทx โˆ’ 12) ยท x = 2 ยท (6 โˆ’ 12) ยท 3 = โˆ’36

What PyTorch does not do is write the training loop for you. That is deliberate: the loop is where you control batching, evaluation, early stopping, logging and checkpoints, and where most bugs live.

Beginner

The training loop, line by line

Almost every PyTorch project contains the same loop. For each epoch (one pass over the training data) and each mini-batch:

  1. model.train() switches on training behaviour (dropout active, batch-norm statistics updating).
  2. optimizer.zero_grad() clears the gradients from the previous batch. PyTorch adds new gradients to old ones, so forgetting this line silently corrupts training.
  3. Forward pass: logits = model(xb), then loss = loss_fn(logits, yb).
  4. Backward pass: loss.backward() fills .grad for every parameter.
  5. optimizer.step() updates the parameters using those gradients.

After each epoch, evaluate on the validation set inside model.eval() and with torch.no_grad(): (no dropout, no graph, much less memory). Keep the weights from the best validation epoch, stop when validation stops improving, and save a checkpoint (torch.save(model.state_dict(), path)) together with everything needed to use it: preprocessing statistics, label names, a version.

Two loss details catch many people. For binary classification, output a raw logit and use BCEWithLogitsLoss, which applies the sigmoid internally in a numerically stable way. For multi-class, use CrossEntropyLoss on logits: it already contains the softmax, so adding your own softmax layer is a bug.

Try it

Try it: review a training loop

A teammate's training script has several classic bugs. Click every line you would comment on in code review, then submit.

Interactive: this part runs in your browser.

Beginner

Data loading, GPUs and memory

A Dataset knows how to return one example (__getitem__) and how many there are (__len__). A DataLoader wraps it to produce batches, shuffle the training data each epoch, and load data in parallel with num_workers. Augmentations (random crops, flips, noise) belong in the training dataset's transforms only; validation and test data must be transformed deterministically.

To use a GPU, move both the model and every batch to the same device: model.to(device) and xb.to(device). "Expected all tensors to be on the same device" is the most common PyTorch error. GPU memory is used by the weights, the optimiser state (Adam keeps two extra numbers per parameter), and the activations saved for the backward pass, which grow with batch size. If you run out of memory: reduce the batch size, use mixed precision (torch.autocast, 16-bit activations), or use gradient accumulation (several small batches per optimiser step).

Reproducibility needs care: set seeds (torch.manual_seed, NumPy, Python), log library versions, and expect tiny differences on GPUs unless deterministic algorithms are forced.

Beginner

Transfer learning and fine-tuning

Training large networks from scratch needs huge datasets and compute. In practice, almost every image and text project starts from a pretrained model and adapts it. For images, a CNN or vision transformer pretrained on ImageNet already detects edges, textures and object parts; you replace the final layer with one for your classes. For text, a pretrained transformer such as DistilBERT already understands language; you add a classification head.

A standard recipe: (1) replace the head; (2) freeze the backbone and train only the head for an epoch or two, so the random head does not send large, destructive gradients into good features; (3) unfreeze and fine-tune everything with a small learning rate (around 1e-4 for CNNs, 2e-5 to 5e-5 for transformers), often with warm-up and a decaying schedule; (4) evaluate per class and analyse errors. With a few thousand labelled examples this usually beats training from scratch by a wide margin.

The Hugging Face transformers library provides thousands of pretrained models and tokenizers with a consistent API (AutoTokenizer, AutoModelForSequenceClassification). Before using one, check its licence, what data it was trained on, its size and latency, and whether it can run where your data is allowed to go.

Try it

Hands-on: three Colab notebooks

Deep-learning code needs a GPU and libraries that do not run inside a web page, so these three notebooks run in Google Colab (free, in your browser). Copy a notebook's code with the button and paste it into a new Colab notebook, or upload the .ipynb file you downloaded with the course. Each notebook ends with "Try next" experiments.

Interactive: this part runs in your browser.

Intermediate ยท Maths

The maths: backpropagation through a layer

For a linear layer z=Wx+b\mathbf z = W\mathbf x + \mathbf b followed by a loss LL, let ฮด=โˆ‚L/โˆ‚z\boldsymbol\delta = \partial L/\partial \mathbf z be the gradient arriving from above. The chain rule gives the three gradients autograd computes:

โˆ‚Lโˆ‚W=ฮดโ€‰xโŠค,โˆ‚Lโˆ‚b=ฮด,โˆ‚Lโˆ‚x=WโŠคฮด\frac{\partial L}{\partial W} = \boldsymbol\delta\,\mathbf x^{\top},\qquad \frac{\partial L}{\partial \mathbf b} = \boldsymbol\delta,\qquad \frac{\partial L}{\partial \mathbf x} = W^{\top}\boldsymbol\delta

The last one is passed down to the previous layer, which is why the algorithm is called back-propagation. For a ReLU, โˆ‚L/โˆ‚zi\partial L/\partial z_i is passed through where zi>0z_i>0 and zeroed elsewhere. For a mini-batch XX (one example per row), the weight gradient is ฮ”โŠคX\Delta^{\top}X averaged over the batch.

With softmax and cross-entropy combined, the gradient with respect to the logits is simply pโˆ’y\mathbf p - \mathbf y (predicted probabilities minus the one-hot label). That compact, stable form is why CrossEntropyLoss takes logits rather than probabilities.

AdamW keeps running averages of the gradient mtm_t and its square vtv_t and updates ฮธโ†ฮธโˆ’ฮทโ€‰(m^t/(v^t+ฯต)+ฮปฮธ)\theta \leftarrow \theta - \eta\,\big(\hat m_t/(\sqrt{\hat v_t}+\epsilon) + \lambda\theta\big), where the weight-decay term ฮปฮธ\lambda\theta is applied separately from the adaptive step (the "W" in AdamW).

Advanced

Advanced: making training fast and reliable

Mixed precision (torch.autocast with a gradient scaler, or bfloat16 on recent GPUs) roughly halves memory and speeds up training. Gradient clipping (clip_grad_norm_) prevents rare huge gradients from wrecking training, especially in RNNs and transformers. Learning-rate schedules (warm-up, cosine or linear decay) are standard for transformers. Gradient accumulation simulates large batches on small GPUs.

Debugging recipe: first overfit a single small batch (the loss should go near zero; if not, there is a bug), check that the initial loss matches theory (about lnโก(classes)\ln(\text{classes}) for balanced classification), plot learning curves, and look at predictions on real examples.

Scaling up: DistributedDataParallel trains on several GPUs; checkpoint regularly so long runs can resume; track every run (config, data version, code commit, metrics) with an experiment tracker such as MLflow or Weights & Biases. Export models for serving with TorchScript or ONNX, and measure latency on the target hardware.

Check yourself

Quiz

1. What happens if you forget optimizer.zero_grad()?

2. Which pair belongs in the validation loop?

3. Your model ends with nn.Softmax and you use nn.CrossEntropyLoss. This isโ€ฆ

4. "Expected all tensors to be on the same device" meansโ€ฆ

5. Why freeze the backbone for the first epochs of fine-tuning?

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready