Maths for AI
Topic 13 · Probability & statistics · ⏱ about 3 h

Likelihood, MLE & MAP

How "fit the data" becomes a precise mathematical recipe.

Show:
In this topic
Beginner

Likelihood: how plausible is a parameter?

So far we have used probability to describe data when we already know the parameters, such as the bias of a coin. In machine learning the situation is reversed: we have the data, and we want to work out the parameters. The key concept for doing this is the likelihood.

p = 0.3p = 0.5p = 0.7p = 0.9maximum likelihood: p̂ = 0.7L(p) = p⁷(1 − p)³ for 7 heads, 3 tails
The likelihood of each possible coin bias given 7 heads in 10 flips. It peaks at 0.7, the maximum likelihood estimate.

Suppose a model has an unknown parameter θ\theta (the Greek letter theta). The likelihood of a particular value of θ\theta is the probability that the model would have produced exactly the data we observed, if θ\theta were the true value:

L(θ)=P(data∣θ). L(\theta) = P(\text{data}\mid\theta).

The data are held fixed, and we look at how this probability changes as we try different values of θ\theta. It is important to be clear about what the likelihood is not. It is not the probability that θ\theta is correct; it measures how well each candidate value of θ\theta explains what we saw.

Let's make this concrete with a coin. We flip it 10 times and see 7 heads and 3 tails. Our model is that each flip lands heads with some unknown probability pp, independently of the others. Because the flips are independent, the probability of our particular sequence is a product: pp for each head and 1−p1 - p for each tail. So

L(p)=p7(1−p)3. L(p) = p^7(1 - p)^3.

Now we can ask how plausible different values of pp are. If the coin were fair, with p=0.5p = 0.5, the likelihood would be 0.510≈0.000980.5^{10} \approx 0.00098. If p=0.3p = 0.3, it would be 0.37×0.73≈0.000080.3^7\times0.7^3 \approx 0.00008, much smaller, because a coin that mostly lands tails would rarely produce 7 heads. If p=0.7p = 0.7, it would be 0.77×0.33≈0.002220.7^7\times0.3^3 \approx 0.00222, the largest of the values we've tried. And if p=0.9p = 0.9, it would drop again to about 0.000480.00048, because such a coin would usually produce even more heads than 7.

pp0.30.50.70.9
L(p)L(p)0.000080.000980.002220.00048

The value p=0.7p = 0.7 makes the observed data more probable than any other value we tried. The maximum likelihood estimate (MLE) formalises this: it is the parameter value that makes the data most probable,

θ^MLE=arg⁡max⁡θL(θ). \hat\theta_{\text{MLE}} = \arg\max_\theta L(\theta).

In the next section we'll find it exactly using calculus rather than by trying values one at a time.

Intermediate · Maths

Deriving the MLE with calculus, and its link to loss functions

To find the maximum likelihood estimate exactly, we use the recipe from the calculus topic: differentiate and set the derivative to zero. Products are awkward to differentiate, so we first take the logarithm. Because the logarithm is an increasing function, the value of pp that maximises log⁡L\log L is the same one that maximises LL. Using the log rules, the product becomes a sum and the powers come down in front:

ℓ(p)=log⁡L(p)=log⁡(p7(1−p)3)=7log⁡p+3log⁡(1−p). \ell(p) = \log L(p) = \log\big(p^7(1-p)^3\big) = 7\log p + 3\log(1 - p).

This is called the log-likelihood. Now we differentiate. The derivative of log⁡p\log p is 1p\frac1p. For log⁡(1−p)\log(1 - p) we need the chain rule: the outer derivative is 11−p\frac{1}{1-p} and the inner derivative of 1−p1 - p is −1-1, giving −11−p-\frac{1}{1-p}. So

ℓ′(p)=7p−31−p. \ell'(p) = \frac{7}{p} - \frac{3}{1 - p}.

Setting this equal to zero and solving: 7p=31−p\frac{7}{p} = \frac{3}{1-p}, so 7(1−p)=3p7(1 - p) = 3p, which gives 7=10p7 = 10p and therefore p^=0.7\hat p = 0.7. If we repeat the calculation with hh heads out of nn flips, the same steps give p^=h/n\hat p = h/n. The maximum likelihood estimate is the obvious answer, the fraction of heads, but now we have a principled reason for it, and the same principle works for models where the answer isn't obvious at all.

This idea connects directly to the loss functions used in machine learning. Software libraries are designed to minimise, so instead of maximising the log-likelihood we minimise its negative, the negative log-likelihood (NLL). Two cases are especially important.

First, suppose the labels are yes/no values yi∈{0,1}y_i\in\{0, 1\}, and the model predicts a probability pip_i that each label is 1. Each label is a Bernoulli outcome, whose probability is pip_i if yi=1y_i = 1 and 1−pi1 - p_i if yi=0y_i = 0. A compact way to write both cases is piyi(1−pi)1−yip_i^{y_i}(1 - p_i)^{1-y_i}. Taking the negative log and adding over all examples gives

NLL=−∑i[yilog⁡pi+(1−yi)log⁡(1−pi)], \text{NLL} = -\sum_i\Big[y_i\log p_i + (1 - y_i)\log(1 - p_i)\Big],

which is exactly the cross-entropy loss used to train logistic regression and classification networks.

Second, suppose the labels are numbers generated as yi=f(xi)+εiy_i = f(x_i) + \varepsilon_i, where the noise εi\varepsilon_i is Gaussian with variance σ2\sigma^2. The Gaussian density gives −log⁡p(yi)=(yi−f(xi))22σ2+constant-\log p(y_i) = \frac{(y_i - f(x_i))^2}{2\sigma^2} + \text{constant}. The constant and the factor 12σ2\frac{1}{2\sigma^2} don't change which ff is best, so minimising the NLL is the same as minimising the sum of squared errors, the loss used in linear regression. In other words, the common loss functions are not arbitrary choices: each one is maximum likelihood under a particular assumption about the data.

Maximum likelihood uses only the data. Sometimes we also have prior beliefs about the parameter, for example that a coin is probably close to fair. Bayes' theorem combines the two: the posterior is proportional to the likelihood times the prior, p(θ∣data)∝L(θ) p(θ)p(\theta\mid\text{data})\propto L(\theta)\,p(\theta). The maximum a posteriori (MAP) estimate is the value that maximises this posterior, or equivalently its logarithm:

θ^MAP=arg⁡max⁡θ[log⁡L(θ)+log⁡p(θ)]. \hat\theta_{\text{MAP}} = \arg\max_\theta\big[\log L(\theta) + \log p(\theta)\big].

For a coin, a convenient prior is the Beta(a,b)(a, b) distribution, and the MAP estimate works out to p^=h+a−1n+a+b−2\hat p = \frac{h + a - 1}{n + a + b - 2}. With a Beta(5, 5) prior, which expresses a belief that the coin is probably near fair, our 7 heads out of 10 give p^=7+410+8=1118≈0.61\hat p = \frac{7 + 4}{10 + 8} = \frac{11}{18} \approx 0.61. The estimate has been pulled from 0.7 towards 0.5 by the prior. But with 700 heads out of 1000 we would get 7041008≈0.698\frac{704}{1008} \approx 0.698: with plenty of data, the evidence overwhelms the prior. In machine learning, a prior belief that the weights should be small becomes a regularisation penalty, and that is exactly what ridge and lasso regression are.

One last point: estimators themselves can be biased. The maximum likelihood estimate of a Gaussian's variance, 1n∑i(xi−xˉ)2\frac{1}{n}\sum_i(x_i - \bar x)^2, comes out slightly too small on average, because the sample mean xˉ\bar x is always a little closer to the data than the true mean is. Dividing by n−1n - 1 instead of nn corrects this, which is why you will see both versions in software.

Try it

Try it: flip a coin and watch the likelihood

Flip the coin. The curve shows the likelihood of every possible pp. Watch it get narrower (more certain) as data accumulate. Tick the prior to see MAP pull small-sample estimates toward 0.5.

Interactive: this part runs in your browser.

Code

Code: maximum likelihood by formula and by optimisation

Real models have no neat formula, so we minimise the NLL numerically, exactly like training.

Python · NumPy + SciPy
import numpy as np
from scipy.optimize import minimize_scalar, minimize

h, n = 7, 10
p_grid = np.linspace(0.01, 0.99, 99)
loglik = h * np.log(p_grid) + (n - h) * np.log(1 - p_grid)
print("grid search MLE:", p_grid[loglik.argmax()])                  # 0.7

nll = lambda p: -(h * np.log(p) + (n - h) * np.log(1 - p))
print("optimiser MLE:", round(minimize_scalar(nll, bounds=(1e-6, 1 - 1e-6), method="bounded").x, 4))

a, b = 5, 5                                                          # Beta prior
print("MAP:", (h + a - 1) / (n + a + b - 2))                         # 0.611

# Gaussian MLE: fit mu and sigma by minimising the NLL
rng = np.random.default_rng(0)
data = rng.normal(loc=2.0, scale=1.5, size=500)
def gauss_nll(params):
    mu, log_sigma = params                    # optimise log(sigma) so sigma stays positive
    s = np.exp(log_sigma)
    return np.sum(0.5 * np.log(2 * np.pi * s ** 2) + (data - mu) ** 2 / (2 * s ** 2))
res = minimize(gauss_nll, x0=[0.0, 0.0])
print("MLE mu, sigma:", res.x[0].round(3), np.exp(res.x[1]).round(3))
print("closed form:  ", data.mean().round(3), data.std(ddof=0).round(3), " (unbiased std:", data.std(ddof=1).round(3), ")")
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
A sample of 100 has standard deviation 20. What is the standard error of the mean?
Question 2.
A sample mean is 50 with standard error 2. What is the upper end of the 95% confidence interval (use 1.96)? Give 2 decimal places.
Question 3.
To halve the standard error, by what factor must the sample size grow?
Check yourself

Quiz

1. The likelihood L(θ) is…

2. 12 heads in 20 flips. The MLE of p is…

3. Minimising cross-entropy is the same as…

4. MAP differs from MLE by adding…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready