Maths for AI
Topic 7 · Linear algebra · ⏱ about 4 h

Matrices & Matrix Multiplication

A dataset is a matrix; a layer of a neural net is a matrix multiply.

Show:
In this topic
Beginner

Matrices and the matrix–vector product

A matrix is a rectangular table of numbers. A matrix with mm rows and nn columns is called an "mm by nn" matrix, written m×nm\times n. In machine learning, the most important matrix is the data matrix XX, which stacks all the examples on top of each other: each row is one example and each column is one feature. Here is a tiny dataset of three houses, each described by its floor area and its number of bedrooms, together with a weight vector:

X (3 × 2)w (2 × 1)Xw (3 × 1)120380220040.5109060140×=row 1 · w = 120×0.5 + 3×10 = 60 + 30 = 90 (one prediction per row)
A matrix–vector product is the dot product of each row (one example) with the weight vector: all predictions at once.
X=[12038022004],w=[0.510]. X = \begin{bmatrix} 120 & 3 \\ 80 & 2 \\ 200 & 4\end{bmatrix}, \qquad w = \begin{bmatrix}0.5\\10\end{bmatrix}.

XX is a 3×23\times 2 matrix (three houses, two features) and ww is a column vector with two entries, one weight per feature.

The most useful thing we can do with these is the matrix–vector product XwXw. The rule is simple: take the dot product of each row of XX with the vector ww, and stack the answers into a new column. Because each row is one house, each dot product is the prediction for that house:

Xw=[(120)(0.5)+(3)(10)(80)(0.5)+(2)(10)(200)(0.5)+(4)(10)]=[60+3040+20100+40]=[9060140]. Xw = \begin{bmatrix} (120)(0.5) + (3)(10) \\ (80)(0.5) + (2)(10) \\ (200)(0.5) + (4)(10)\end{bmatrix} = \begin{bmatrix} 60 + 30 \\ 40 + 20 \\ 100 + 40\end{bmatrix} = \begin{bmatrix} 90 \\ 60 \\ 140\end{bmatrix}.

A single matrix–vector product therefore computes the predictions for every example at once. This is why linear regression is written compactly as y^=Xw\hat y = Xw, and it is also why GPUs, which are extremely fast at multiplying matrices, made deep learning practical.

Matrix multiplication only works when the shapes fit together, and the rule for this is the most useful debugging tool you will have when writing ML code. An (m×n)(m\times n) matrix can multiply an (n×p)(n\times p) matrix, and the result is (m×p)(m\times p):

(m×n)⋅(n×p)=(m×p). (m\times n)\cdot(n\times p) = (m\times p).

The two "inner" numbers must match, because each row of the first matrix (length nn) has to be dotted with each column of the second (also length nn). The "outer" numbers give the shape of the answer. In our example, (3×2)⋅(2×1)=(3×1)(3\times2)\cdot(2\times1) = (3\times1): three predictions, one per house. Whenever your code crashes with a "shape mismatch" error, writing out the shapes like this almost always shows you what went wrong.

Beginner

Matrix × matrix, transpose, identity and inverse

Multiplying two matrices extends the same idea. The entry in row ii and column jj of the product ABAB is the dot product of row ii of AA with column jj of BB. Let's work through a 2×22\times2 example carefully:

unit square: area 1det = 1.62: area × 1.62det = 0: squashed flatno inverse!
The determinant is the factor by which a matrix scales area. When it is zero, the plane is flattened onto a line and cannot be "unflattened": there is no inverse.
[1234][5678]=[(1)(5)+(2)(7)(1)(6)+(2)(8)(3)(5)+(4)(7)(3)(6)+(4)(8)]=[19224350]. \begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}5&6\\7&8\end{bmatrix} = \begin{bmatrix}(1)(5)+(2)(7) & (1)(6)+(2)(8)\\ (3)(5)+(4)(7) & (3)(6)+(4)(8)\end{bmatrix} = \begin{bmatrix}19&22\\43&50\end{bmatrix}.

The top-left entry, 19, comes from the first row [1,2][1, 2] dotted with the first column [5,7][5, 7]; the top-right entry, 22, comes from the same row dotted with the second column [6,8][6, 8]; and so on. An important difference from ordinary numbers is that order matters: in general AB≠BAAB \ne BA. If you multiply these two matrices the other way round you get a completely different answer, which you can check in the code section.

The transpose of a matrix, written A⊤A^\top, is obtained by flipping it so that its rows become columns:

[1234]⊤=[1324]. \begin{bmatrix}1&2\\3&4\end{bmatrix}^\top = \begin{bmatrix}1&3\\2&4\end{bmatrix}.

Transposing twice gets you back where you started, (A⊤)⊤=A(A^\top)^\top = A, and the transpose of a product reverses the order, (AB)⊤=B⊤A⊤(AB)^\top = B^\top A^\top; this second rule is used in nearly every derivation in the course. Transposes also explain the notation a⊤ba^\top b for the dot product: if aa and bb are column vectors, then a⊤a^\top is a row, and a row times a column is exactly the dot product. In particular, x⊤x=∥x∥2x^\top x = \lVert x\rVert^2.

The identity matrix II has 1s along its diagonal and 0s everywhere else. Multiplying by it changes nothing, IA=AI=AIA = AI = A, so it plays the role that the number 1 plays in ordinary arithmetic.

That leads to the inverse. For ordinary numbers, dividing by 5 undoes multiplying by 5. For a square matrix AA, the inverse A−1A^{-1} is the matrix that undoes AA, in the sense that A−1A=IA^{-1}A = I. For a 2×22\times2 matrix there is a simple formula:

[abcd]−1=1ad−bc[d−b−ca]. \begin{bmatrix}a&b\\c&d\end{bmatrix}^{-1} = \frac{1}{ad - bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}.

The number ad−bcad - bc in the denominator is called the determinant. If the determinant is zero, the formula would divide by zero, so the inverse does not exist. Geometrically, a zero determinant means the matrix squashes the plane flat onto a line, and once two different points have been squashed onto the same point, there is no way to tell them apart again. In linear regression, this happens when two features are perfectly correlated (for example, a temperature recorded in both Celsius and Fahrenheit).

Inverses let us solve systems of linear equations. Consider the pair of equations 2x+y=52x + y = 5 and x+3y=10x + 3y = 10. We can write them as one matrix equation,

[2113][xy]=[510]. \begin{bmatrix}2&1\\1&3\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix} = \begin{bmatrix}5\\10\end{bmatrix}.

The determinant of the coefficient matrix is (2)(3)−(1)(1)=5(2)(3) - (1)(1) = 5, which is not zero, so an inverse exists. Using the formula and multiplying both sides by the inverse:

[xy]=15[3−1−12][510]=15[15−10−5+20]=15[515]=[13]. \begin{bmatrix}x\\y\end{bmatrix} = \frac{1}{5}\begin{bmatrix}3&-1\\-1&2\end{bmatrix}\begin{bmatrix}5\\10\end{bmatrix} = \frac15\begin{bmatrix}15 - 10\\ -5 + 20\end{bmatrix} = \frac15\begin{bmatrix}5\\15\end{bmatrix} = \begin{bmatrix}1\\3\end{bmatrix}.

We can check the answer by substituting back: 2(1)+3=52(1) + 3 = 5 and 1+3(3)=101 + 3(3) = 10, so both equations are satisfied. The famous normal equation of linear regression, w^=(X⊤X)−1X⊤y\hat w = (X^\top X)^{-1}X^\top y, is exactly this idea applied to a larger system. (In real code we ask the computer to solve the system directly rather than computing an inverse, which is faster and more accurate, but the mathematics is the same.)

Try it

Try it: step through a matrix multiplication

Press Next entry to compute the product one entry at a time. Each entry is one row of A dotted with one column of B.

Interactive: this part runs in your browser.

Try it

Try it: a matrix is a transformation

A matrix moves every point in space. The columns of AA are where the basis arrows e1=[1,0]e_1 = [1,0] and e2=[0,1]e_2 = [0,1] land. Try the presets, and type your own numbers.

Interactive: this part runs in your browser.

Code

Code: matrices in NumPy

Python · NumPy
import numpy as np

X = np.array([[120, 3], [80, 2], [200, 4]], dtype=float)   # 3 x 2
w = np.array([0.5, 10.0])                                  # length 2
print("X shape:", X.shape, " predictions Xw:", X @ w)      # [90, 60, 140]

A, B = np.array([[1, 2], [3, 4]]), np.array([[5, 6], [7, 8]])
print("AB =\n", A @ B, "\nBA =\n", B @ A)                  # different!
print("A transpose:\n", A.T)
print("(AB)^T == B^T A^T ?", np.array_equal((A @ B).T, B.T @ A.T))

M = np.array([[2.0, 1.0], [1.0, 3.0]])
print("det:", np.linalg.det(M), "\ninverse:\n", np.linalg.inv(M))
print("solve 2x+y=5, x+3y=10 ->", np.linalg.solve(M, [5, 10]))   # [1, 3]
# Prefer np.linalg.solve over inv(M) @ b: faster and more accurate.

try:
    np.ones((3, 2)) @ np.ones((3, 1))      # inner dims 2 and 3 don't match
except ValueError as e:
    print("shape error:", e)
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
For A=(1234)A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} and x=(5,6)x = (5, 6), what is the first entry of AxAx?
Question 2.
A is 32×78432 \times 784 and B is 784×128784 \times 128. How many numbers are in ABAB?
Question 3.
What is the determinant of (4213)\begin{pmatrix} 4 & 2 \\ 1 & 3 \end{pmatrix}?
Check yourself

Quiz

1. A (4×3) matrix times a (3×2) matrix gives a…

2. (AB)ᵀ equals…

3. A matrix with determinant 0…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready