Maths for AI
Topic 5 · Notation & functions · ⏱ about 2 h

Reading ML Maths: Notation, exp & log

The symbols, functions and log rules that every ML formula uses.

Show:
In this topic
Beginner

Why learn the notation first?

When people first open a machine learning paper, the formulas often look far harder than they really are. Most of that difficulty comes from notation: a compact shorthand that experienced readers decode without thinking. Once you know the shorthand, a formula such as y^=∑j=1dwjxj+b\hat y = \sum_{j=1}^{d} w_j x_j + b reads as an ordinary sentence: "to make a prediction, multiply each feature by its weight, add all of those products together, and then add the number bb." The aim of this section is to give you that shorthand, so that later topics feel like reading rather than code-breaking.

The first thing to notice is that letters come in three "sizes". A plain lower-case letter such as xx, yy or ww usually stands for a single number, which mathematicians call a scalar; for example, x=3.2x = 3.2 could be the size of a room in some unit. A vector is an ordered list of numbers. Some books print it in bold (x\mathbf x), but in machine learning it is very common to write it as an ordinary xx and let the context make it clear. A house could be described by the vector x=[120, 3, 15]x = [120,\ 3,\ 15], meaning 120 square metres, 3 bedrooms and 15 years old. Finally, a capital letter such as XX usually means a matrix, which is a table of numbers. In machine learning, XX is almost always the whole dataset, with one row for each example and one column for each feature.

Small subscripts tell you which number you are talking about. If xx is the house vector above, then x2=3x_2 = 3 is its second entry, the number of bedrooms. When we have many houses, we need a second index to say which house; you will see this written either as xix_i or as x(i)x^{(i)}, so x(5)x^{(5)} means "the fifth house in the dataset". The brackets in x(i)x^{(i)} are there to show that the ii is a label, not a power. Two further counting letters appear constantly: nn is usually the number of examples (for instance n=1000n = 1000 houses) and dd is the number of features (here d=3d = 3).

Decorations on top of a letter change its meaning in a consistent way. A hat, as in y^\hat y (read "y-hat"), means a prediction or estimate: if the real price is y=400y = 400 and our model says 410, we write y^=410\hat y = 410. A star, as in w∗w^*, means the true or best possible value, the value we would love to know. A bar, as in xˉ\bar x, means an average, so xˉ=1n∑ixi\bar x = \frac{1}{n}\sum_i x_i.

A few more symbols describe sets and relationships. The symbol R\mathbb R stands for all real numbers, and Rd\mathbb R^d stands for all lists of dd real numbers, so writing x∈R3x \in \mathbb R^3 (read "x is in R-three") simply says that xx is a list of three numbers. The upside-down A, ∀\forall, means "for all": the statement wj≥0 ∀jw_j \ge 0\ \forall j says that every weight is zero or positive. Finally, ≈\approx means "approximately equal to", :=:= means "is defined to be", and ∝\propto means "is proportional to", so f(x)∝x2f(x) \propto x^2 says that f(x)=c x2f(x) = c\,x^2 for some constant cc that we don't care about at the moment.

The table below collects all of these symbols in one place so that you can come back to it whenever a formula looks unfamiliar.

SymbolHow to read itExample
x, y, wx,\ y,\ wa single number (scalar)x=3.2x = 3.2
xx or x\mathbf xa vector (list of numbers)x=[120, 3, 15]x = [120,\ 3,\ 15]
XXa matrix; usually the datasetone row per example
xjx_jentry jj of a vectorx2=3x_2 = 3
xix_i or x(i)x^{(i)}example number iix(5)x^{(5)} = the fifth house
y^, w∗, xˉ\hat y,\ w^*,\ \bar xprediction, true/best value, averagey^=410\hat y = 410
n, dn,\ dnumber of examples, number of featuresn=1000, d=3n = 1000,\ d = 3
∈, Rd, ∀\in,\ \mathbb R^d,\ \forall"is in", lists of dd numbers, "for all"x∈R3x\in\mathbb R^3
≈, :=, ∝\approx,\ :=,\ \proptoapproximately, defined as, proportional tof(x)∝x2f(x)\propto x^2
Beginner

Sums, products, min and argmin

Three operations appear in almost every formula in this course: adding things up, multiplying things together, and asking "which value is best?". Each has its own symbol.

argmin = 3 (the input)min = 1 (the lowest output)f(w) = (w − 3)² + 1
The minimum is the lowest value of the function; the argmin is the input that produces it.

The capital Greek letter sigma, Σ\Sigma, means add up. The expression under it says where a counter starts, the number on top says where it stops, and the formula to its right says what to add at each step. For example,

∑i=14i2 \sum_{i=1}^{4} i^2

tells us to let ii run through 1, 2, 3 and 4, square each value, and add the results. Writing it out in full gives 12+22+32+42=1+4+9+16=301^2 + 2^2 + 3^2 + 4^2 = 1 + 4 + 9 + 16 = 30. Nothing more mysterious than that is going on.

Here is a sum you will meet constantly. Suppose yiy_i is the true value for example ii and y^i\hat y_i is the model's prediction. Then

1n∑i=1n(yi−y^i)2 \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat y_i)^2

is read from the inside out. First, for each example, compute the error yi−y^iy_i - \hat y_i. Next, square it, which makes every error positive and makes large errors count much more than small ones. Then add up these squared errors over all nn examples, and finally divide by nn to get an average. This quantity is the mean squared error, the standard way to measure how wrong a regression model is.

The capital Greek letter pi, Π\Pi, works in exactly the same way but means multiply together: ∏i=13pi=p1×p2×p3\prod_{i=1}^{3} p_i = p_1 \times p_2 \times p_3. It appears whenever we combine the probabilities of independent events, because the probability that several independent things all happen is the product of their individual probabilities.

The last idea, the difference between min and argmin, confuses many students, so let's take it slowly with an example. Consider the function f(w)=(w−3)2+1f(w) = (w - 3)^2 + 1. The squared term (w−3)2(w-3)^2 can never be negative, and it equals zero exactly when w=3w = 3. So the smallest value ff can ever take is 0+1=10 + 1 = 1, and it takes that value at w=3w = 3. We describe these two facts separately:

min⁡wf(w)=1andarg⁡min⁡wf(w)=3. \min_w f(w) = 1 \qquad \text{and} \qquad \arg\min_w f(w) = 3.

The min is the lowest output, while the argmin is the input that produces it ("arg" is short for argument, which is another word for input). When we train a model we write w^=arg⁡min⁡wL(w)\hat w = \arg\min_w L(w), meaning "the weights w^\hat w are whichever weights make the loss LL as small as possible". We almost always care about the argmin, because the weights are the model; the loss value itself is just a score.

Finally, vertical bars mean absolute value, so ∣−5∣=5|{-5}| = 5, and double bars around a vector, as in ∥x∥\lVert x \rVert, mean the length of that vector, which the next topic explains.

Beginner

Functions, exponentials and logarithms

A function is best thought of as a machine: you feed in an input and it hands back exactly one output. The function f(x)=2x+1f(x) = 2x + 1 takes the input 3 and returns 2×3+1=72\times3 + 1 = 7. A machine learning model is simply a function with some adjustable knobs, called parameters. For example, fw(x)=wx+bf_w(x) = wx + b is a whole family of straight lines, and training means turning the knobs ww and bb until the line fits the data well.

eˣ: always > 0log x: only for x > 0σ(x): squashed into (0, 1)
The exponential, the logarithm (its inverse) and the sigmoid built from them.

Two particular functions appear everywhere in machine learning. The first is the exponential function, exe^x, where e≈2.718e \approx 2.718 is a special constant. Three facts about it are worth remembering. It is always positive, whatever xx is. It equals 1 when x=0x = 0. And it grows extremely fast as xx increases, while shrinking towards zero (without ever reaching it) as xx becomes more negative. Because it always produces positive numbers, we use it whenever we need to turn arbitrary scores into something that can behave like a probability.

The second is the natural logarithm, written log⁡x\log x (some books write ln⁡x\ln x). The logarithm undoes the exponential: log⁡(ea)=a\log(e^a) = a. In words, log⁡x\log x answers the question "to what power must I raise ee to get xx?". Since exe^x is always positive, the logarithm is only defined for positive inputs. Because e0=1e^0 = 1, we have log⁡1=0\log 1 = 0, and numbers between 0 and 1 have negative logarithms. This last fact matters, because probabilities live between 0 and 1, so their logarithms are negative.

Logarithms obey three rules that we will use again and again. The first is that the log of a product is the sum of the logs:

log⁡(ab)=log⁡a+log⁡b. \log(ab) = \log a + \log b.

For example, log⁡(0.5×0.25)=log⁡0.5+log⁡0.25=−0.693+(−1.386)=−2.079\log(0.5 \times 0.25) = \log 0.5 + \log 0.25 = -0.693 + (-1.386) = -2.079. The second rule says the log of a ratio is a difference, log⁡(a/b)=log⁡a−log⁡b\log(a/b) = \log a - \log b; the quantity log⁡(p/(1−p))\log\big(p/(1-p)\big), called the log-odds, is built this way. The third rule says that a power comes down in front: log⁡(ak)=klog⁡a\log(a^k) = k\log a, so log⁡(p7)=7log⁡p\log(p^7) = 7\log p.

Why does machine learning use logarithms so much? There are three reasons. First, they turn products into sums. The probability of a thousand independent observations is a product of a thousand numbers, which is awkward to work with, but its logarithm is a sum of a thousand terms, which is easy to differentiate and to compute. Second, they avoid numerical underflow. A computer cannot represent a number as small as 0.012000.01^{200}, so it silently rounds it to zero; but its logarithm, 200log⁡0.01=−921200 \log 0.01 = -921, is a perfectly ordinary number. Third, the logarithm is an increasing function, which means that whatever input makes ff largest also makes log⁡f\log f largest. So we can maximise log⁡f\log f instead of ff and get exactly the same answer.

We can now build one of the most important functions in the course, the sigmoid:

σ(z)=11+e−z. \sigma(z) = \frac{1}{1 + e^{-z}}.

Let's evaluate it at a few points to see what it does. At z=0z = 0 we have e0=1e^{0} = 1, so σ(0)=11+1=0.5\sigma(0) = \frac{1}{1 + 1} = 0.5. At z=2z = 2, e−2≈0.135e^{-2} \approx 0.135, so σ(2)=11.135≈0.88\sigma(2) = \frac{1}{1.135} \approx 0.88. At z=−2z = -2, e2≈7.39e^{2} \approx 7.39, so σ(−2)=18.39≈0.12\sigma(-2) = \frac{1}{8.39} \approx 0.12. As zz becomes very large, e−ze^{-z} shrinks towards zero and σ(z)\sigma(z) approaches 1; as zz becomes very negative, e−ze^{-z} becomes huge and σ(z)\sigma(z) approaches 0. So the sigmoid takes any real number and squashes it smoothly into the range between 0 and 1, which is exactly what we need to turn a model's raw score into a probability.

Try it

Try it: explore eˣ, log x and the sigmoid

Slide xx and switch between the three functions. Check that log undoes exp, that log is negative between 0 and 1 and undefined below 0, and that the sigmoid never leaves the range (0, 1).

Interactive: this part runs in your browser.

Code

Code: the notation in NumPy

Every symbol above has a one-line NumPy equivalent. Run this and compare with the worked examples.

Python · NumPy
import numpy as np

i = np.arange(1, 5)                 # [1, 2, 3, 4]
print("sum of i^2:", np.sum(i ** 2))          # 30
print("product:", np.prod([0.5, 0.2, 0.9]))   # 0.09

w = np.linspace(-2, 8, 1001)
f = (w - 3) ** 2 + 1
print("min f =", f.min(), " argmin w =", w[f.argmin()])   # 1.0 and 3.0

print("log(0.5*0.25) =", np.log(0.5 * 0.25), "=", np.log(0.5) + np.log(0.25))
print("0.01**200 =", 0.01 ** 200, "(underflow!)  but 200*log(0.01) =", 200 * np.log(0.01))

sigmoid = lambda z: 1 / (1 + np.exp(-z))
print("sigmoid(0, 2, -2):", sigmoid(np.array([0, 2, -2])).round(3))

# Mean squared error written exactly like the formula (1/n) * sum (y - y_hat)^2
y, y_hat = np.array([3.0, 5.0, 2.0]), np.array([2.5, 5.5, 2.0])
print("MSE:", np.mean((y - y_hat) ** 2))
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
Compute ∑i=14i2\sum_{i=1}^{4} i^2.
Question 2.
Compute ∏i=14i\prod_{i=1}^{4} i.
Question 3.
What is ln⁡(e3)\ln(e^3)?
Question 4.
For f(x)=(x−3)2f(x) = (x - 3)^2, what is arg⁡min⁡xf(x)\arg\min_x f(x)?
Check yourself

Quiz

1. ∑i=132i\sum_{i=1}^{3} 2i equals…

2. For f(w) = (w − 3)² + 1, argmin f is…

3. log(a·b) equals…

4. σ(0) equals…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready