Likelihood, MLE & MAP
How "fit the data" becomes a precise mathematical recipe.
Likelihood: how plausible is a parameter?
So far we have used probability to describe data when we already know the parameters, such as the bias of a coin. In machine learning the situation is reversed: we have the data, and we want to work out the parameters. The key concept for doing this is the likelihood.
Suppose a model has an unknown parameter (the Greek letter theta). The likelihood of a particular value of is the probability that the model would have produced exactly the data we observed, if were the true value:
The data are held fixed, and we look at how this probability changes as we try different values of . It is important to be clear about what the likelihood is not. It is not the probability that is correct; it measures how well each candidate value of explains what we saw.
Let's make this concrete with a coin. We flip it 10 times and see 7 heads and 3 tails. Our model is that each flip lands heads with some unknown probability , independently of the others. Because the flips are independent, the probability of our particular sequence is a product: for each head and for each tail. So
Now we can ask how plausible different values of are. If the coin were fair, with , the likelihood would be . If , it would be , much smaller, because a coin that mostly lands tails would rarely produce 7 heads. If , it would be , the largest of the values we've tried. And if , it would drop again to about , because such a coin would usually produce even more heads than 7.
| 0.3 | 0.5 | 0.7 | 0.9 | |
|---|---|---|---|---|
| 0.00008 | 0.00098 | 0.00222 | 0.00048 |
The value makes the observed data more probable than any other value we tried. The maximum likelihood estimate (MLE) formalises this: it is the parameter value that makes the data most probable,
In the next section we'll find it exactly using calculus rather than by trying values one at a time.
Deriving the MLE with calculus, and its link to loss functions
To find the maximum likelihood estimate exactly, we use the recipe from the calculus topic: differentiate and set the derivative to zero. Products are awkward to differentiate, so we first take the logarithm. Because the logarithm is an increasing function, the value of that maximises is the same one that maximises . Using the log rules, the product becomes a sum and the powers come down in front:
This is called the log-likelihood. Now we differentiate. The derivative of is . For we need the chain rule: the outer derivative is and the inner derivative of is , giving . So
Setting this equal to zero and solving: , so , which gives and therefore . If we repeat the calculation with heads out of flips, the same steps give . The maximum likelihood estimate is the obvious answer, the fraction of heads, but now we have a principled reason for it, and the same principle works for models where the answer isn't obvious at all.
This idea connects directly to the loss functions used in machine learning. Software libraries are designed to minimise, so instead of maximising the log-likelihood we minimise its negative, the negative log-likelihood (NLL). Two cases are especially important.
First, suppose the labels are yes/no values , and the model predicts a probability that each label is 1. Each label is a Bernoulli outcome, whose probability is if and if . A compact way to write both cases is . Taking the negative log and adding over all examples gives
which is exactly the cross-entropy loss used to train logistic regression and classification networks.
Second, suppose the labels are numbers generated as , where the noise is Gaussian with variance . The Gaussian density gives . The constant and the factor don't change which is best, so minimising the NLL is the same as minimising the sum of squared errors, the loss used in linear regression. In other words, the common loss functions are not arbitrary choices: each one is maximum likelihood under a particular assumption about the data.
Maximum likelihood uses only the data. Sometimes we also have prior beliefs about the parameter, for example that a coin is probably close to fair. Bayes' theorem combines the two: the posterior is proportional to the likelihood times the prior, . The maximum a posteriori (MAP) estimate is the value that maximises this posterior, or equivalently its logarithm:
For a coin, a convenient prior is the Beta distribution, and the MAP estimate works out to . With a Beta(5, 5) prior, which expresses a belief that the coin is probably near fair, our 7 heads out of 10 give . The estimate has been pulled from 0.7 towards 0.5 by the prior. But with 700 heads out of 1000 we would get : with plenty of data, the evidence overwhelms the prior. In machine learning, a prior belief that the weights should be small becomes a regularisation penalty, and that is exactly what ridge and lasso regression are.
One last point: estimators themselves can be biased. The maximum likelihood estimate of a Gaussian's variance, , comes out slightly too small on average, because the sample mean is always a little closer to the data than the true mean is. Dividing by instead of corrects this, which is why you will see both versions in software.
Try it: flip a coin and watch the likelihood
Flip the coin. The curve shows the likelihood of every possible . Watch it get narrower (more certain) as data accumulate. Tick the prior to see MAP pull small-sample estimates toward 0.5.
Interactive: this part runs in your browser.
Code: maximum likelihood by formula and by optimisation
Real models have no neat formula, so we minimise the NLL numerically, exactly like training.
import numpy as np
from scipy.optimize import minimize_scalar, minimize
h, n = 7, 10
p_grid = np.linspace(0.01, 0.99, 99)
loglik = h * np.log(p_grid) + (n - h) * np.log(1 - p_grid)
print("grid search MLE:", p_grid[loglik.argmax()]) # 0.7
nll = lambda p: -(h * np.log(p) + (n - h) * np.log(1 - p))
print("optimiser MLE:", round(minimize_scalar(nll, bounds=(1e-6, 1 - 1e-6), method="bounded").x, 4))
a, b = 5, 5 # Beta prior
print("MAP:", (h + a - 1) / (n + a + b - 2)) # 0.611
# Gaussian MLE: fit mu and sigma by minimising the NLL
rng = np.random.default_rng(0)
data = rng.normal(loc=2.0, scale=1.5, size=500)
def gauss_nll(params):
mu, log_sigma = params # optimise log(sigma) so sigma stays positive
s = np.exp(log_sigma)
return np.sum(0.5 * np.log(2 * np.pi * s ** 2) + (data - mu) ** 2 / (2 * s ** 2))
res = minimize(gauss_nll, x0=[0.0, 0.0])
print("MLE mu, sigma:", res.x[0].round(3), np.exp(res.x[1]).round(3))
print("closed form: ", data.mean().round(3), data.std(ddof=0).round(3), " (unbiased std:", data.std(ddof=1).round(3), ")")Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. The likelihood L(θ) is…
2. 12 heads in 20 flips. The MLE of p is…
3. Minimising cross-entropy is the same as…
4. MAP differs from MLE by adding…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML
- An Introduction to Statistical Learning (free book) ↗ James, Witten, Hastie, Tibshirani (Python edition available)