Maths for AI
Topic 12 · Probability & statistics · ⏱ about 4 h

Probability & Distributions

The language of uncertainty, noise and predictions.

Show:
In this topic
Beginner

Random variables, expectation and variance

Machine learning is full of uncertainty: noisy measurements, unpredictable labels, randomly chosen training batches. Probability is the language we use to reason about it. Its central idea is the random variable, a quantity whose value we don't know in advance, such as the result of rolling a die. A random variable comes with a distribution, which tells us how likely each possible value is. Probabilities always lie between 0 and 1, and the probabilities of all the possible values add up to exactly 1.

11/621/631/641/651/661/6mean = 3.5σ ≈ 1.71
A fair die: every face has probability 1/6. The mean is the balance point (3.5); the standard deviation measures the typical distance from it.

The first summary of a distribution is its expectation, or mean, written E[X]\mathbb E[X]. It is the average value you would see over many repetitions, computed as a probability-weighted average of the possible values:

E[X]=∑xx P(X=x). \mathbb E[X] = \sum_x x\,P(X = x).

For a fair die, each face has probability 16\frac16, so E[X]=(1+2+3+4+5+6)×16=216=3.5\mathbb E[X] = (1 + 2 + 3 + 4 + 5 + 6)\times\frac16 = \frac{21}{6} = 3.5. Notice that the expected value doesn't have to be a value you can actually roll; it is a long-run average.

The second summary is the variance, which measures spread: how far values typically land from the mean. It is defined as the average squared distance from the mean, Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X) = \mathbb E\big[(X - \mu)^2\big], where μ=E[X]\mu = \mathbb E[X]. For the die, the distances from 3.5 are ±2.5, ±1.5, ±0.5\pm2.5,\ \pm1.5,\ \pm0.5, so

Var⁡(X)=16[(2.5)2+(1.5)2+(0.5)2+(0.5)2+(1.5)2+(2.5)2]=17.56≈2.92. \operatorname{Var}(X) = \frac16\Big[(2.5)^2 + (1.5)^2 + (0.5)^2 + (0.5)^2 + (1.5)^2 + (2.5)^2\Big] = \frac{17.5}{6} \approx 2.92.

Because the variance is in squared units, we often take its square root to get the standard deviation, σ=Var⁡(X)≈1.71\sigma = \sqrt{\operatorname{Var}(X)} \approx 1.71, which is in the same units as the original quantity.

A handful of rules let us work with expectations and variances without recalculating from scratch. Expectation is linear: E[aX+b]=a E[X]+b\mathbb E[aX + b] = a\,\mathbb E[X] + b, and E[X+Y]=E[X]+E[Y]\mathbb E[X + Y] = \mathbb E[X] + \mathbb E[Y] for any two random variables. Variance behaves differently. Adding a constant shifts every value by the same amount and doesn't change the spread, but multiplying by aa stretches every distance by aa, and therefore the squared distances by a2a^2: Var⁡(aX+b)=a2Var⁡(X)\operatorname{Var}(aX + b) = a^2\operatorname{Var}(X). When two random variables are independent, meaning one tells you nothing about the other, their variances add: Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y).

Putting those rules together gives one of the most important facts in the whole course. Take nn independent measurements X1,…,XnX_1, \dots, X_n, each with variance σ2\sigma^2, and average them. The sum has variance nσ2n\sigma^2 (the variances add), and dividing by nn multiplies the variance by 1n2\frac{1}{n^2}, so

Var⁡(1n∑i=1nXi)=1n2⋅nσ2=σ2n. \operatorname{Var}\Big(\frac{1}{n}\sum_{i=1}^n X_i\Big) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}.

Averaging reduces noise, and it does so in a predictable way. This is the mathematical reason why more data gives more reliable estimates, and why ensembles that average many models are more stable than any single model.

Finally, the simplest distribution of all is the Bernoulli distribution, which describes a biased coin: it takes the value 1 with probability pp and 0 with probability 1−p1 - p. Its mean is pp and its variance is p(1−p)p(1-p). Every yes/no label in machine learning (spam or not, fraud or not) is modelled as a Bernoulli random variable.

Beginner

The Gaussian (normal) distribution

The most important continuous distribution is the Gaussian, also called the normal distribution or the bell curve. It has two parameters: the mean μ\mu, which sets where the centre of the bell is, and the standard deviation σ\sigma, which sets how wide it is. We write X∼N(μ,σ2)X\sim\mathcal N(\mu, \sigma^2). Its density is

μ − 3σμ − 2σμ − 1σμμ + 1σμ + 2σμ + 3σ68%95% within ±2σ
The Gaussian bell curve: about 68% of values lie within one standard deviation of the mean and 95% within two.
p(x)=12πσ2exp⁡ ⁣(−(x−μ)22σ2). p(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\Big(-\frac{(x - \mu)^2}{2\sigma^2}\Big).

This formula is easier to understand if you read it from the inside out. The term (x−μ)2(x - \mu)^2 is the squared distance of xx from the centre. Dividing by 2σ22\sigma^2 measures that distance relative to the spread of the distribution, so being 2 units from the centre counts as "far" when σ\sigma is small but "close" when σ\sigma is large. Taking exp⁡\exp of the negative of this quantity turns small distances into values near 1 and large distances into values near 0, which produces the bell shape. The fraction in front is just a scaling constant that makes the total area under the curve equal to 1, as every probability distribution must.

A very practical rule of thumb follows from the shape of the bell: about 68% of values fall within one standard deviation of the mean (between μ−σ\mu - \sigma and μ+σ\mu + \sigma), about 95% fall within two, and about 99.7% fall within three. So a value more than three standard deviations from the mean is genuinely unusual.

Any Gaussian can be converted into the standard normal N(0,1)\mathcal N(0, 1) by subtracting its mean and dividing by its standard deviation, z=(x−μ)/σz = (x - \mu)/\sigma. The result, called a z-score, says how many standard deviations a value lies from the mean. This is exactly what scikit-learn's StandardScaler does to each feature.

Why is the Gaussian so common? The central limit theorem says that when a quantity is the sum or average of many small, independent random effects, its distribution is approximately Gaussian, whatever the individual effects look like. Measurement noise is usually the combined result of many small disturbances, so it is natural to model it as Gaussian. As you'll see in the likelihood topic, that single modelling assumption leads directly to the least-squares method of linear regression.

One subtle point is worth stating clearly. For a continuous variable, the height p(x)p(x) of the curve is a density, not a probability. The probability of any single exact value is zero; probabilities come from areas under the curve over a range of values. Because of this, a density can be larger than 1 (a very narrow Gaussian has a very tall peak) without breaking any rules.

Interactive: this part runs in your browser.

Interactive. Pick a population that is not bell-shaped at all, then increase n: the averages still form a bell curve.
Try it

Try it: samples vs the bell curve

Draw samples and compare the histogram with the curve. With 20 samples the histogram is rough; with 1000 it matches the curve, and about 68% land within ±1σ.

Interactive: this part runs in your browser.

Intermediate · Maths

Conditional probability and Bayes' theorem

Often we want the probability of something given that we already know something else. The conditional probability of AA given BB is written P(A∣B)P(A\mid B) and defined as

1000 people10 ill (1%)990 healthy9 test +1 test −49.5 test +940.5 test −P(ill | +) =9 / (9 + 49.5)≈ 15%
Bayes' theorem as a tree of natural frequencies: most positive tests come from the large healthy group.
P(A∣B)=P(A,B)P(B), P(A\mid B) = \frac{P(A, B)}{P(B)},

where P(A,B)P(A, B) is the probability that both happen. The idea is that once we know BB has happened, we restrict our attention to the situations where BB is true, and ask what fraction of those also have AA.

Rearranging the definition gives the product rule, P(A,B)=P(A∣B) P(B)P(A, B) = P(A\mid B)\,P(B). A second basic rule, the sum rule, says that we can recover the probability of AA alone by adding up the joint probabilities over every possible value of BB: P(A)=∑bP(A,B=b)P(A) = \sum_b P(A, B = b). Two events are independent when P(A,B)=P(A) P(B)P(A, B) = P(A)\,P(B), which is the same as saying that knowing BB doesn't change the probability of AA.

The product rule can be written two ways, because "A and B" is the same event as "B and A": P(A∣B) P(B)=P(B∣A) P(A)P(A\mid B)\,P(B) = P(B\mid A)\,P(A). Dividing both sides by P(B)P(B) gives Bayes' theorem:

P(A∣B)=P(B∣A) P(A)P(B). P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}.

Bayes' theorem tells us how to update a belief when we see new evidence. P(A)P(A) is the prior, what we believed before seeing the evidence. P(B∣A)P(B\mid A) is the likelihood, how probable the evidence is if AA is true. P(B)P(B) is the overall probability of the evidence, and P(A∣B)P(A\mid B) is the posterior, our updated belief.

A classic example shows why this matters. Suppose a disease affects 1% of people. A test for it correctly detects the disease 90% of the time, but it also wrongly flags 5% of healthy people. You test positive. How likely is it that you actually have the disease? Many people guess about 90%, but let's calculate. First, we need the overall probability of a positive test. A positive can come from an ill person who is correctly flagged, or from a healthy person who is wrongly flagged:

P(+)=0.90×0.01⏟ill and flagged+0.05×0.99⏟healthy and flagged=0.009+0.0495=0.0585. P(+) = \underbrace{0.90\times0.01}_{\text{ill and flagged}} + \underbrace{0.05\times0.99}_{\text{healthy and flagged}} = 0.009 + 0.0495 = 0.0585.

Now Bayes' theorem gives the probability of being ill given a positive test:

P(ill∣+)=P(+∣ill) P(ill)P(+)=0.0090.0585≈0.154. P(\text{ill}\mid +) = \frac{P(+\mid\text{ill})\,P(\text{ill})}{P(+)} = \frac{0.009}{0.0585} \approx 0.154.

The answer is only about 15%. The reason is that the disease is rare: the healthy group is so much larger than the ill group that even a small false-positive rate among the healthy produces far more positive tests than the disease itself does. This is exactly the same effect that makes precision low when a classifier looks for rare events. Bayes' theorem will reappear when we combine data with prior beliefs in MAP estimation.

Interactive: this part runs in your browser.

Breakdown. Bayes' theorem with 1000 people: why a positive test can still mean you are probably healthy.
Try it

Try it: Bayes' theorem with 1000 people

Each dot is a person. Adjust how common the disease is and how good the test is, and count how many of the positive tests are real.

Interactive: this part runs in your browser.

Code

Code: simulate to check the maths

Simulation is a great way to build intuition and to check a calculation.

Python · NumPy
import numpy as np
rng = np.random.default_rng(0)

rolls = rng.integers(1, 7, size=100_000)
print("die mean:", rolls.mean().round(3), "(theory 3.5)  variance:", rolls.var().round(3), "(theory 2.917)")

# Averaging reduces variance: Var(mean of n) = sigma^2 / n
for n in [1, 10, 100]:
    means = rng.integers(1, 7, size=(20_000, n)).mean(axis=1)
    print(f"n={n:3d}: variance of the average = {means.var():.4f}   theory {35/12/n:.4f}")

# Gaussian: 68-95-99.7 rule
z = rng.normal(0, 1, 100_000)
print("within 1, 2, 3 sigma:", [round(float(np.mean(np.abs(z) < k)), 3) for k in (1, 2, 3)])

# Bayes' theorem by simulation (medical test)
N = 1_000_000
ill = rng.random(N) < 0.01
positive = np.where(ill, rng.random(N) < 0.90, rng.random(N) < 0.05)
print("P(ill | positive) simulated:", round(ill[positive].mean(), 3), "  formula: 0.154")
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
A fair die is rolled twice. What is the probability of two sixes? Give a decimal to 4 places.
Question 2.
X takes values 1, 2, 3 with probabilities 0.2, 0.5, 0.3. What is E[X]E[X]?
Question 3.
A test detects 90% of sick people and wrongly flags 5% of healthy people. 2% of people are sick. If someone tests positive, what is the probability they are sick? Give 3 decimal places.
Check yourself

Quiz

1. E[2X + 3] when E[X] = 5 is…

2. Var(3X) equals…

3. The variance of the average of n independent samples (each variance σ²) is…

4. In the medical-test example, why is P(ill | +) so low?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready