Maths for AI
Topic 10 · Calculus & optimisation · ⏱ about 3 h

Partial Derivatives, Gradients & the Hessian

Calculus with many weights at once.

Show:
In this topic
Beginner

Partial derivatives and the gradient

Real models have many weights, not just one, so their loss is a function of many variables at once, written L(w1,w2,…,wd)L(w_1, w_2, \dots, w_d). We still want to know which way to adjust each weight, so we need a version of the derivative that works with several inputs.

∇f (uphill)−∇f (downhill)minimum
Contour lines join points of equal height. The gradient is perpendicular to them and points uphill; gradient descent steps the opposite way.

The idea is to change one variable at a time. The partial derivative ∂L∂w1\frac{\partial L}{\partial w_1} (the curly ∂\partial is read "partial") measures how the loss changes when we nudge only w1w_1 and hold every other weight fixed. In practice this means you differentiate as usual, but treat all the other variables as if they were constants.

Let's work through an example with two variables,

f(w1,w2)=w12+3w1w2+w22. f(w_1, w_2) = w_1^2 + 3w_1w_2 + w_2^2.

To find ∂f∂w1\frac{\partial f}{\partial w_1}, we pretend w2w_2 is just a fixed number. The first term w12w_1^2 gives 2w12w_1. The middle term 3w1w23w_1w_2 is "3w23w_2 times w1w_1", and since 3w23w_2 is now a constant, its derivative is 3w23w_2. The last term w22w_2^2 doesn't involve w1w_1 at all, so it is a constant and contributes 0. Putting these together, and then doing the same with the roles swapped,

∂f∂w1=2w1+3w2,∂f∂w2=3w1+2w2. \frac{\partial f}{\partial w_1} = 2w_1 + 3w_2, \qquad \frac{\partial f}{\partial w_2} = 3w_1 + 2w_2.

The gradient, written ∇f\nabla f (the upside-down triangle is called "nabla"), simply collects all the partial derivatives into one vector. At the point (w1,w2)=(1,2)(w_1, w_2) = (1, 2):

∇f=[2w1+3w23w1+2w2],∇f(1,2)=[2+63+4]=[87]. \nabla f = \begin{bmatrix}2w_1 + 3w_2\\ 3w_1 + 2w_2\end{bmatrix}, \qquad \nabla f(1, 2) = \begin{bmatrix}2 + 6\\ 3 + 4\end{bmatrix} = \begin{bmatrix}8\\7\end{bmatrix}.

The gradient has a beautiful geometric meaning. Picture the loss as a landscape of hills and valleys, drawn on a map with contour lines (lines of equal height). At any point, the gradient vector points in the direction of steepest ascent, straight uphill, and it is always perpendicular to the contour line through that point. Its length tells you how steep the hill is there. So the negative gradient, −∇f-\nabla f, points straight downhill, which is exactly where we want to go when minimising a loss.

That gives us gradient descent in many dimensions: subtract a small multiple of the gradient from the current weights. Starting from w=(1,2)w = (1, 2) with a learning rate η=0.1\eta = 0.1,

wnew=[12]−0.1[87]=[1−0.82−0.7]=[0.21.3]. w_{\text{new}} = \begin{bmatrix}1\\2\end{bmatrix} - 0.1\begin{bmatrix}8\\7\end{bmatrix} = \begin{bmatrix}1 - 0.8\\2 - 0.7\end{bmatrix} = \begin{bmatrix}0.2\\1.3\end{bmatrix}.

Let's check that this step actually helped. At the starting point, f(1,2)=1+6+4=11f(1, 2) = 1 + 6 + 4 = 11. At the new point, f(0.2,1.3)=0.04+0.78+1.69=2.51f(0.2, 1.3) = 0.04 + 0.78 + 1.69 = 2.51. One step reduced the function from 11 to about 2.5. Training a neural network does exactly this, except that the gradient has millions of entries instead of two.

Try it

Try it: gradient descent on a 2-D landscape

The shading shows the height of the function (darker = lower). Arrows show the downhill direction everywhere. Click to choose a starting point, then step. Increase the stretch κ to make a narrow valley and watch the zig-zag; tick momentum to smooth it out.

Interactive: this part runs in your browser.

Intermediate · Maths

Matrix calculus: the three formulas behind every derivation

Several derivations later in the course (linear regression, ridge regression, logistic regression) use three formulas from matrix calculus. They can look intimidating, but each one is just a familiar single-variable rule written for vectors. Here we derive each one and check it with a small 2-D example, so that you can trust it when it appears later.

Formula 1: the gradient of a linear function. For a fixed vector aa,

∇w(a⊤w)=a. \nabla_w\big(a^\top w\big) = a.

To see why, write it out in two dimensions: a⊤w=a1w1+a2w2a^\top w = a_1w_1 + a_2w_2. The partial derivative with respect to w1w_1 is a1a_1 (the other term is a constant), and with respect to w2w_2 it is a2a_2. Stacking these gives the vector [a1,a2][a_1, a_2], which is just aa. This is the vector version of the familiar fact that the derivative of awaw is aa.

Formula 2: the gradient of a quadratic form. For a square matrix AA,

∇w(w⊤Aw)=(A+A⊤) w,which equals 2Aw when A is symmetric. \nabla_w\big(w^\top A w\big) = (A + A^\top)\,w, \quad\text{which equals } 2Aw \text{ when } A \text{ is symmetric.}

Let's check it with the symmetric matrix A=[1332]A = \begin{bmatrix}1&3\\3&2\end{bmatrix}. Multiplying out, w⊤Aw=w12+6w1w2+2w22w^\top A w = w_1^2 + 6w_1w_2 + 2w_2^2. The partial derivatives are 2w1+6w22w_1 + 6w_2 and 6w1+4w26w_1 + 4w_2. Meanwhile 2Aw=2[w1+3w23w1+2w2]=[2w1+6w26w1+4w2]2Aw = 2\begin{bmatrix}w_1 + 3w_2\\3w_1 + 2w_2\end{bmatrix} = \begin{bmatrix}2w_1 + 6w_2\\6w_1 + 4w_2\end{bmatrix}, which is exactly the same. This is the vector version of "the derivative of aw2aw^2 is 2aw2aw".

Formula 3: the gradient of the squared error. This is the one that matters most. We first expand the squared length using ∥v∥2=v⊤v\lVert v\rVert^2 = v^\top v and the transpose rules:

∥y−Xw∥2=(y−Xw)⊤(y−Xw)=y⊤y−2(X⊤y)⊤w+w⊤(X⊤X)w. \lVert y - Xw\rVert^2 = (y - Xw)^\top(y - Xw) = y^\top y - 2(X^\top y)^\top w + w^\top(X^\top X)w.

Now differentiate each term with respect to ww. The first term, y⊤yy^\top y, doesn't contain ww, so its gradient is zero. The middle term is linear in ww with a=−2X⊤ya = -2X^\top y, so Formula 1 gives −2X⊤y-2X^\top y. The last term is a quadratic form with the symmetric matrix A=X⊤XA = X^\top X, so Formula 2 gives 2X⊤Xw2X^\top Xw. Adding them,

∇w∥y−Xw∥2=−2X⊤y+2X⊤Xw=−2X⊤(y−Xw). \nabla_w\lVert y - Xw\rVert^2 = -2X^\top y + 2X^\top Xw = -2X^\top(y - Xw).

It helps to read this result in words: "minus two, times the features transposed, times the errors". It has exactly the same shape as the single-example result 2(wx−y)x2(wx - y)x from the chain-rule section, just applied to all the examples at once.

Two more objects complete the toolkit. When a function produces a vector rather than a single number (as a layer of a neural network does), its derivative is a whole matrix of partial derivatives, called the Jacobian, with entries Jij=∂hi/∂xjJ_{ij} = \partial h_i/\partial x_j. The chain rule for such functions multiplies Jacobians together, and that is precisely what backpropagation computes.

Finally, the Hessian is the matrix of second derivatives, Hij=∂2f∂wi ∂wjH_{ij} = \frac{\partial^2 f}{\partial w_i\,\partial w_j}. Where the gradient describes slope, the Hessian describes curvature: how the slope itself is changing. For our earlier function f=w12+3w1w2+w22f = w_1^2 + 3w_1w_2 + w_2^2, differentiating the gradient once more gives

H=[2332]. H = \begin{bmatrix}2&3\\3&2\end{bmatrix}.

If the Hessian is "positive semi-definite" everywhere (a property explained in the eigenvector topic), the function is shaped like a bowl and has no misleading local minima. This particular function is not a bowl: it curves upwards in one direction and downwards in another, which makes it a saddle. The optimisation topic explains why that distinction matters so much.

Code

Code: verify the gradient of the squared error

Python · NumPy
import numpy as np

f = lambda w: w[0] ** 2 + 3 * w[0] * w[1] + w[1] ** 2
grad_f = lambda w: np.array([2 * w[0] + 3 * w[1], 3 * w[0] + 2 * w[1]])
w = np.array([1.0, 2.0])
print("gradient at (1,2):", grad_f(w))                      # [8, 7]
w_new = w - 0.1 * grad_f(w)
print("after one step:", w_new, " f:", f(w), "->", round(f(w_new), 3))   # 11 -> 2.5

def numerical_grad(F, w, h=1e-6):
    g = np.zeros_like(w)
    for j in range(len(w)):
        e = np.zeros_like(w); e[j] = h
        g[j] = (F(w + e) - F(w - e)) / (2 * h)              # nudge one coordinate at a time
    return g

rng = np.random.default_rng(0)
X, y, w = rng.normal(size=(20, 3)), rng.normal(size=20), rng.normal(size=3)
loss = lambda w: np.sum((y - X @ w) ** 2)
analytic = -2 * X.T @ (y - X @ w)                           # identity 3
print("analytic: ", analytic.round(5))
print("numerical:", numerical_grad(loss, w).round(5))
print("Hessian eigenvalues of f:", np.linalg.eigvalsh(np.array([[2, 3], [3, 2]])))   # [-1, 5]: a saddle
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
For f(x,y)=x2+3xyf(x, y) = x^2 + 3xy, what is ∂f/∂y\partial f / \partial y at (2,5)(2, 5)?
Question 2.
Gradient descent: w=4w = 4, gradient =10= 10, learning rate =0.1= 0.1. What is the new ww?
Question 3.
For f(x,y)=x2+y2f(x, y) = x^2 + y^2, what is the length of the gradient at (3,4)(3, 4)?
Check yourself

Quiz

1. For f(a, b) = a²b, ∂f/∂a is…

2. The gradient points in the direction of…

3. ∇_w ‖y − Xw‖² equals…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready