Vectors, Norms & the Dot Product
Data points are vectors; predictions are dot products.
Vectors: lists of numbers that are also arrows
A vector is an ordered list of numbers. The word "ordered" matters: and are different vectors. In machine learning, every example in a dataset is stored as a vector of its features. A house might become , recording its floor area, number of bedrooms and age, and a greyscale image of 28 by 28 pixels becomes a vector of 784 brightness values.
It helps to picture vectors geometrically. A vector with two entries, such as , can be drawn as a point on a graph, or as an arrow from the origin to that point. Vectors with more entries live in higher-dimensional spaces that we can't draw, but all the same rules still apply, so the 2-D picture is a reliable guide to your intuition.
Vectors can be added and scaled. To add two vectors of the same length, you add their matching entries; to scale a vector by a number, you multiply every entry by that number:
Geometrically, adding means placing the arrows head to tail, and scaling stretches or shrinks the arrow (a negative number also flips its direction).
The length of a vector, which mathematicians call its norm, comes straight from Pythagoras' theorem. The arrow to is the hypotenuse of a right-angled triangle with sides 3 and 4, so its length is . The same calculation works in any number of dimensions:
This is called the L2 norm or Euclidean length. There is a second common way to measure size, the L1 norm, which simply adds up the absolute values of the entries:
For the L1 norm is . You can think of it as the distance a taxi would drive on a grid of streets, rather than the straight-line distance a bird would fly. These two norms are exactly the penalties used by Ridge regression (L2) and Lasso (L1). Note also that the squared L2 norm, , has no square root, which makes it much easier to differentiate; that is why it appears so often in loss functions.
Finally, the distance between two points is the length of the vector that joins them, which is their difference. For and , the difference is , so the distance is . This is precisely how the k-nearest-neighbours algorithm decides which training points are "close" to a new point.
Interactive: this part runs in your browser.
The dot product: the most important operation in ML
The dot product is the single most important operation in machine learning, so it is worth understanding from two different angles. Algebraically, it takes two vectors of the same length, multiplies their matching entries, and adds up the results to give a single number:
For example, . The notation means the same thing; it comes from matrix notation, which the next topic explains.
The reason the dot product matters so much is that every linear model's prediction is a dot product. Suppose a model predicts house prices (in thousands of dollars) using the weights , meaning 0.5 thousand dollars per square metre and 10 thousand dollars per bedroom, plus a base price . For a house with , the prediction is
So the model predicts $110,000. Each feature contributes "its value times how much the model cares about it", and the dot product adds those contributions up. Neural networks do exactly the same thing millions of times, once for every connection between neurons.
The second, geometric, view explains what the dot product measures. It can be shown that
where is the angle between the two arrows. Because is positive for angles under 90°, zero at exactly 90°, and negative beyond 90°, the sign of the dot product tells you how the vectors are oriented. A positive dot product means they point in broadly the same direction. A dot product of exactly zero means they are at right angles, which we call orthogonal. A negative dot product means they point in broadly opposite directions.
If we divide the dot product by both lengths, we are left with just , a number between −1 and 1 called the cosine similarity:
Cosine similarity compares only the directions of two vectors and ignores their lengths. Search engines and large language models represent words and documents as vectors, called embeddings, and use cosine similarity to decide whether two pieces of text mean similar things.
The dot product also gives us projections. If is a unit vector (a vector of length 1), then tells you how far reaches in the direction of . You can picture it as the length of the shadow casts onto the line through . Principal component analysis is built entirely from projections of this kind.
One last useful identity: a vector dotted with itself gives its squared length, . This connects the dot product back to the norm from the previous section.
Try it: dot product and projection
Rotate the two arrows. Watch the dot product computed both ways, by components and by lengths × cos θ, give the same number. Make them perpendicular to get 0.
Interactive: this part runs in your browser.
Code: vectors in NumPy
import numpy as np
a, b = np.array([1.0, 2.0, 3.0]), np.array([4.0, 5.0, 6.0])
print("a + b =", a + b, " 3a =", 3 * a)
print("dot:", a @ b, "=", np.dot(a, b), "=", np.sum(a * b)) # 32
print("L2 norm of [3,4]:", np.linalg.norm([3, 4]), " L1:", np.linalg.norm([3, 4], ord=1))
# Linear model prediction = dot product + intercept
w, x, bias = np.array([0.5, 10.0]), np.array([120.0, 3.0]), 20.0
print("prediction:", w @ x + bias) # 110
# Cosine similarity and projection
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
u = a / np.linalg.norm(a) # unit vector in a's direction
print("cosine similarity:", round(cos, 4), " projection length of b on a:", round(b @ u, 4))
print("orthogonal?", np.isclose(np.array([1, 2]) @ np.array([-2, 1]), 0))Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. [2, 0, 1] · [3, 5, 4] =
2. The L2 norm of [6, 8] is…
3. If a · b = 0 (both non-zero), the vectors are…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 3Blue1Brown: Essence of linear algebra ↗ Visual intuition
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML