Maths for AI
Topic 6 · Linear algebra · ⏱ about 3 h

Vectors, Norms & the Dot Product

Data points are vectors; predictions are dot products.

Show:
In this topic
Beginner

Vectors: lists of numbers that are also arrows

A vector is an ordered list of numbers. The word "ordered" matters: [3,4][3, 4] and [4,3][4, 3] are different vectors. In machine learning, every example in a dataset is stored as a vector of its features. A house might become x=[120, 3, 15]x = [120,\ 3,\ 15], recording its floor area, number of bedrooms and age, and a greyscale image of 28 by 28 pixels becomes a vector of 784 brightness values.

a = [4, 1]b = [1, 3]a + b = [5, 4]L2: √(3² + 4²) = 5L1: 3 + 4 = 7
Left: adding vectors head to tail. Right: the L2 norm is the straight-line length; the L1 norm is the "taxi" distance along the axes.

It helps to picture vectors geometrically. A vector with two entries, such as [3,4][3, 4], can be drawn as a point on a graph, or as an arrow from the origin (0,0)(0, 0) to that point. Vectors with more entries live in higher-dimensional spaces that we can't draw, but all the same rules still apply, so the 2-D picture is a reliable guide to your intuition.

Vectors can be added and scaled. To add two vectors of the same length, you add their matching entries; to scale a vector by a number, you multiply every entry by that number:

[1,2]+[3,−1]=[1+3, 2+(−1)]=[4,1],3⋅[1,2]=[3,6]. [1, 2] + [3, -1] = [1 + 3,\ 2 + (-1)] = [4, 1], \qquad 3\cdot[1, 2] = [3, 6].

Geometrically, adding means placing the arrows head to tail, and scaling stretches or shrinks the arrow (a negative number also flips its direction).

The length of a vector, which mathematicians call its norm, comes straight from Pythagoras' theorem. The arrow to [3,4][3, 4] is the hypotenuse of a right-angled triangle with sides 3 and 4, so its length is 32+42=25=5\sqrt{3^2 + 4^2} = \sqrt{25} = 5. The same calculation works in any number of dimensions:

∥x∥2=x12+x22+⋯+xd2. \lVert x\rVert_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_d^2}.

This is called the L2 norm or Euclidean length. There is a second common way to measure size, the L1 norm, which simply adds up the absolute values of the entries:

∥x∥1=∣x1∣+∣x2∣+⋯+∣xd∣. \lVert x\rVert_1 = |x_1| + |x_2| + \dots + |x_d|.

For [3,4][3, 4] the L1 norm is 3+4=73 + 4 = 7. You can think of it as the distance a taxi would drive on a grid of streets, rather than the straight-line distance a bird would fly. These two norms are exactly the penalties used by Ridge regression (L2) and Lasso (L1). Note also that the squared L2 norm, ∥x∥22=x12+⋯+xd2\lVert x\rVert_2^2 = x_1^2 + \dots + x_d^2, has no square root, which makes it much easier to differentiate; that is why it appears so often in loss functions.

Finally, the distance between two points is the length of the vector that joins them, which is their difference. For a=[1,1]a = [1, 1] and b=[4,5]b = [4, 5], the difference is b−a=[3,4]b - a = [3, 4], so the distance is ∥b−a∥=5\lVert b - a\rVert = 5. This is precisely how the k-nearest-neighbours algorithm decides which training points are "close" to a new point.

Interactive: this part runs in your browser.

Interactive. Drag the tips of a and b, then switch between adding, subtracting, scaling and projecting.
Beginner

The dot product: the most important operation in ML

The dot product is the single most important operation in machine learning, so it is worth understanding from two different angles. Algebraically, it takes two vectors of the same length, multiplies their matching entries, and adds up the results to give a single number:

a·b > 0 (same way)angle 34°a·b = 0 (orthogonal)angle 90°a·b < 0 (opposite)angle 143°
The sign of the dot product tells you whether two vectors point the same way, at right angles, or opposite ways.
a⋅b=a⊤b=∑j=1dajbj. a\cdot b = a^\top b = \sum_{j=1}^{d} a_j b_j.

For example, [1,2,3]⋅[4,5,6]=(1)(4)+(2)(5)+(3)(6)=4+10+18=32[1, 2, 3]\cdot[4, 5, 6] = (1)(4) + (2)(5) + (3)(6) = 4 + 10 + 18 = 32. The notation a⊤ba^\top b means the same thing; it comes from matrix notation, which the next topic explains.

The reason the dot product matters so much is that every linear model's prediction is a dot product. Suppose a model predicts house prices (in thousands of dollars) using the weights w=[0.5, 10]w = [0.5,\ 10], meaning 0.5 thousand dollars per square metre and 10 thousand dollars per bedroom, plus a base price b=20b = 20. For a house with x=[120, 3]x = [120,\ 3], the prediction is

y^=w⋅x+b=(0.5)(120)+(10)(3)+20=60+30+20=110. \hat y = w\cdot x + b = (0.5)(120) + (10)(3) + 20 = 60 + 30 + 20 = 110.

So the model predicts $110,000. Each feature contributes "its value times how much the model cares about it", and the dot product adds those contributions up. Neural networks do exactly the same thing millions of times, once for every connection between neurons.

The second, geometric, view explains what the dot product measures. It can be shown that

a⋅b=∥a∥ ∥b∥cos⁡θ, a\cdot b = \lVert a\rVert\,\lVert b\rVert\cos\theta,

where θ\theta is the angle between the two arrows. Because cos⁡θ\cos\theta is positive for angles under 90°, zero at exactly 90°, and negative beyond 90°, the sign of the dot product tells you how the vectors are oriented. A positive dot product means they point in broadly the same direction. A dot product of exactly zero means they are at right angles, which we call orthogonal. A negative dot product means they point in broadly opposite directions.

If we divide the dot product by both lengths, we are left with just cos⁡θ\cos\theta, a number between −1 and 1 called the cosine similarity:

cos⁡θ=a⋅b∥a∥ ∥b∥. \cos\theta = \frac{a\cdot b}{\lVert a\rVert\,\lVert b\rVert}.

Cosine similarity compares only the directions of two vectors and ignores their lengths. Search engines and large language models represent words and documents as vectors, called embeddings, and use cosine similarity to decide whether two pieces of text mean similar things.

The dot product also gives us projections. If uu is a unit vector (a vector of length 1), then b⋅ub\cdot u tells you how far bb reaches in the direction of uu. You can picture it as the length of the shadow bb casts onto the line through uu. Principal component analysis is built entirely from projections of this kind.

One last useful identity: a vector dotted with itself gives its squared length, x⋅x=x12+⋯+xd2=∥x∥2x\cdot x = x_1^2 + \dots + x_d^2 = \lVert x\rVert^2. This connects the dot product back to the norm from the previous section.

Try it

Try it: dot product and projection

Rotate the two arrows. Watch the dot product computed both ways, by components and by lengths × cos θ, give the same number. Make them perpendicular to get 0.

Interactive: this part runs in your browser.

Code

Code: vectors in NumPy

Python · NumPy
import numpy as np

a, b = np.array([1.0, 2.0, 3.0]), np.array([4.0, 5.0, 6.0])
print("a + b =", a + b, "  3a =", 3 * a)
print("dot:", a @ b, "=", np.dot(a, b), "=", np.sum(a * b))      # 32
print("L2 norm of [3,4]:", np.linalg.norm([3, 4]), "  L1:", np.linalg.norm([3, 4], ord=1))

# Linear model prediction = dot product + intercept
w, x, bias = np.array([0.5, 10.0]), np.array([120.0, 3.0]), 20.0
print("prediction:", w @ x + bias)                                # 110

# Cosine similarity and projection
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
u = a / np.linalg.norm(a)                     # unit vector in a's direction
print("cosine similarity:", round(cos, 4), "  projection length of b on a:", round(b @ u, 4))
print("orthogonal?", np.isclose(np.array([1, 2]) @ np.array([-2, 1]), 0))
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
Compute the dot product (1,2,3)⋅(4,−5,6)(1, 2, 3) \cdot (4, -5, 6).
Question 2.
What is the length (L2 norm) of (3,4)(3, 4)?
Question 3.
What is the cosine similarity of (1,0)(1, 0) and (1,1)(1, 1)? Give 3 decimal places.
Check yourself

Quiz

1. [2, 0, 1] · [3, 5, 4] =

2. The L2 norm of [6, 8] is…

3. If a · b = 0 (both non-zero), the vectors are…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready