Deep Learning
Topic 3 ยท Deep Learning ยท โฑ about 6 h

Backpropagation & Autograd from Scratch

Computational graphs, the chain rule as message passing, gradients of the operations networks use, gradient checking, and how PyTorch's autograd works.

Show:
In this topic
Beginner

Every computation is a graph

Any formula a network computes can be drawn as a computational graph: inputs and parameters at the bottom, each operation (add, multiply, ReLU, matrix product) a node, the loss at the top. The forward pass computes values from the inputs up to the loss. The backward pass computes, for every node, how much the loss would change if that node's value changed slightly: its gradient โˆ‚L/โˆ‚node\partial L/\partial \text{node}.

The key fact is the chain rule. Each operation only needs to know its own local gradient (how its output changes with its inputs). The backward pass starts with โˆ‚L/โˆ‚L=1\partial L/\partial L = 1 at the top and passes gradients downwards: each node multiplies the gradient arriving from above by its local gradient and sends the result to its inputs.

โˆ‚Lโˆ‚x=โˆ‚Lโˆ‚yโ‹…โˆ‚yโˆ‚x(gradient from above ร— local gradient)\frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\cdot\frac{\partial y}{\partial x}\qquad\text{(gradient from above ร— local gradient)}

Doing this from the top down is called reverse-mode differentiation. It is efficient for networks because there is one output (the loss) and millions of parameters: one backward pass, which costs about the same as one or two forward passes, gives the gradient for all of them.

Beginner

Watch it: Autograd: values flow forward, gradients flow back

A short narrated animation. Press โ–ถ Play, or step through it with โฎ โญ. Tick ๐Ÿ”Š narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

A worked example

Take L=relu(aโ‹…b+c)L=\text{relu}(a\cdot b+c) with a=2, b=โˆ’3, c=10a=2,\ b=-3,\ c=10.

NodeForward valueLocal gradientGradient of L (backward)
d = a ยท bโˆ’6โˆ‚d/โˆ‚a = b = โˆ’3, โˆ‚d/โˆ‚b = a = 2โˆ‚L/โˆ‚a = 1 ยท (โˆ’3) = โˆ’3, โˆ‚L/โˆ‚b = 1 ยท 2 = 2
e = d + c4โˆ‚e/โˆ‚d = 1, โˆ‚e/โˆ‚c = 1โˆ‚L/โˆ‚d = 1, โˆ‚L/โˆ‚c = 1
L = relu(e)41 if e > 0 else 0โˆ‚L/โˆ‚e = 1 (e is positive)

Read it bottom-up for the forward pass and top-down for the backward pass. Three patterns appear everywhere: an add node passes its gradient unchanged to both inputs; a multiply node sends each input the gradient times the other input; a ReLU passes the gradient through where the input was positive and blocks it where it was negative. If c were โˆ’10, e would be โˆ’4, the ReLU would block everything, and every gradient would be 0: a "dead" unit for this example.

Try it

Try it: forward and backward on a graph

Change the inputs. Values (blue) flow up; gradients (orange) flow down. The last row checks each gradient numerically, the way you test an autograd implementation.

Interactive: this part runs in your browser.

Beginner

Gradients of the operations networks use

A real network uses a handful of operations on whole tensors. Their backward rules, with G=โˆ‚L/โˆ‚YG=\partial L/\partial Y arriving from above:

ForwardBackward
Y=XWY=XW (dense layer; X: (B, in), W: (in, out))โˆ‚L/โˆ‚W=XโŠคG\partial L/\partial W=X^\top G, โˆ‚L/โˆ‚X=Gโ€‰WโŠค\partial L/\partial X=G\,W^\top
Y=X+bY=X+b (bias broadcast over the batch)โˆ‚L/โˆ‚b=โˆ‘batchG\partial L/\partial b=\sum_{\text{batch}}G: sum over the axes that were broadcast
Y=relu(X)Y=\text{relu}(X)GโŠ™[X>0]G\odot[X>0]
Y=ฯƒ(X)Y=\sigma(X)GโŠ™Y(1โˆ’Y)G\odot Y(1-Y)
Y=tanhโก(X)Y=\tanh(X)GโŠ™(1โˆ’Y2)G\odot(1-Y^2)
softmax + cross-entropy, probabilities P, one-hot targets T, mean over Bโˆ‚L/โˆ‚logits=(Pโˆ’T)/B\partial L/\partial \text{logits}=(P-T)/B

A useful check: a gradient always has the same shape as the thing it is the gradient of. XโŠคGX^\top G is (in, B)(B, out) = (in, out), the shape of W. If a value is used in two places (for example a residual connection), its gradients from both uses are added; this is why PyTorch accumulates into .grad.

Beginner

Gradient checking

When you write a backward pass yourself, check it numerically. Nudge one parameter by a small hh in each direction and compare the change in the loss with your analytic gradient:

โˆ‚Lโˆ‚ฮธโ‰ˆL(ฮธ+h)โˆ’L(ฮธโˆ’h)2h,hโ‰ˆ10โˆ’5,rel. error=โˆฃgaโˆ’gnโˆฃmaxโก(โˆฃgaโˆฃ+โˆฃgnโˆฃ,โ€‰10โˆ’12).\frac{\partial L}{\partial \theta}\approx\frac{L(\theta+h)-L(\theta-h)}{2h},\qquad h\approx 10^{-5},\qquad \text{rel. error}=\frac{|g_a-g_n|}{\max(|g_a|+|g_n|,\,10^{-12})}.

Use float64 for the check. A relative error below about 10โˆ’610^{-6} means the gradient is right; around 10โˆ’210^{-2} or worse means a bug. Check a few random entries of each parameter rather than all of them, and turn off randomness (dropout) while checking. Frameworks provide this as a test utility (torch.autograd.gradcheck).

Try it

Code example: build a tiny autograd engine

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Advanced

How PyTorch's autograd works

PyTorch builds the graph dynamically as your Python code runs (define-by-run): every operation on a tensor with requires_grad=True records a node with its backward function (y.grad_fn). loss.backward() walks the graph in reverse topological order, exactly like the exercise, and adds the result into .grad of every leaf tensor (parameters). Then the graph is freed (retain_graph=True keeps it).

  • with torch.no_grad(): records nothing: use it for evaluation and inference (less memory, faster).
  • x.detach() returns a tensor cut off from the graph: gradients will not flow back through it (used for targets, logging, stopping gradients deliberately).
  • Because gradients accumulate, call optimizer.zero_grad() each step, or accumulate over several batches deliberately (gradient accumulation).
  • Saved activations for the backward pass are the main memory cost of training; gradient checkpointing recomputes them instead of storing them.
Code

Code: the same example in PyTorch

Autograd reproduces the worked example, and gradcheck tests a custom function.

Python ยท PyTorch
import torch
a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(-3.0, requires_grad=True)
c = torch.tensor(10.0, requires_grad=True)
L = torch.relu(a * b + c)
L.backward()
print(L.item(), a.grad.item(), b.grad.item(), c.grad.item())   # 4.0 -3.0 2.0 1.0

# numerical check of a function (needs float64)
f = lambda x: (x.sin() * x).sum()
x = torch.randn(5, dtype=torch.float64, requires_grad=True)
print(torch.autograd.gradcheck(f, (x,)))                        # True
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab โ†— (free; Runtime โ†’ Change runtime type โ†’ GPU for deep learning).
Check yourself

Quiz

1. Why is reverse-mode differentiation used to train networks?

2. In the backward pass, a multiply node z = x ยท y sends x the gradientโ€ฆ

3. For a dense layer Y = XW with X: (32, 10) and W: (10, 4), what is the shape of โˆ‚L/โˆ‚W?

4. A bias b of shape (4,) is broadcast over a batch of 32. Its gradient isโ€ฆ

5. Your analytic and numerical gradients differ with relative error 0.03. This most likely meansโ€ฆ

6. Why must you call optimizer.zero_grad() each step in PyTorch?

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready