Machine Learning
Topic 11 · Model quality · ⏱ about 3 h

Bias, Variance & Overfitting

The central trade-off in all of machine learning.

Show:
In this topic
Beginner

Underfitting vs overfitting

Every model sits somewhere on a scale from too simple to too complex, and both ends of that scale cause problems. Understanding the two failure modes, and how to tell them apart, is probably the single most useful skill in practical machine learning.

underfit (high bias)good fitoverfit (high variance)
The same data fitted by a model that is too simple, about right, and too complex.
low bias, low variancelow bias, high variancehigh bias, low variancehigh bias, high variance
Bias is how far the average throw lands from the centre; variance is how scattered the throws are.

A model that is too simple for the problem cannot capture the real pattern in the data. Imagine fitting a straight line to points that clearly follow a curve: however you place the line, it misses systematically, so the error is high on the training data and equally high on new data. This is called underfitting, and the model is said to have high bias. Bias is error that comes from wrong assumptions built into the model, in this case the assumption that the relationship is a straight line.

A model that is too complex has the opposite problem. It is flexible enough to bend through every training point, including the random noise, so its training error is tiny. But the noise in new data is different, so on new data its predictions are poor. This is called overfitting, and the model is said to have high variance. Variance is error that comes from being too sensitive to the particular sample of data you happened to train on: train the same model on a slightly different sample and you would get a very different fit.

UnderfittingGood fitOverfitting
Modeltoo simpleabout righttoo complex
Training errorhighlowvery low
Test errorhighlowhigh
Main problemhigh bias(balanced)high variance

These two sources of error, together with the noise that no model can predict, add up to the total expected error on new data:

Expected test error=Bias2+Variance+Irreducible noise. \text{Expected test error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible noise}.

Making a model more flexible usually lowers its bias but raises its variance, and making it simpler does the reverse. The best model is the one that balances the two, and finding that balance is what model selection, regularisation and cross-validation are all about. (The mathematical derivation of this decomposition is in the Intermediate section below.)

Interactive: this part runs in your browser.

Breakdown. Expected test error split into bias², variance and noise for every model complexity.
Beginner

Watch it: Underfitting, a good fit, and overfitting

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Try it

Try it: polynomial degree

Increase the degree. Low degree = straight-ish line (underfit). Very high degree = wild wiggles (overfit). The lower chart shows the classic U-shaped test error.

Interactive: this part runs in your browser.

Beginner

How to diagnose and fix

To fix a model, you first need to know which problem it has, and the key evidence comes from comparing its error on the training data with its error on validation data.

high bias: both high, closetraining-set size →more data won't helphigh variance: big gaptraining-set size →gap narrows with more data— training error — validation error
Learning curves tell you whether a model suffers from high bias or high variance.

A very useful diagnostic is the learning curve, which plots both errors as you train on more and more data. If the model has high bias, both curves level off quickly at a high error, close to each other: the model has already learned all it can, and more data won't help because it simply isn't flexible enough. If the model has high variance, the training error is low but the validation error is much higher, and there is a large gap between the curves; as more data is added, the gap slowly narrows, because more examples make it harder to memorise noise.

Each diagnosis calls for different remedies, and applying the wrong one wastes time. For high bias, make the model more capable: use a more flexible model (for example, add polynomial features or move from a linear model to a tree ensemble), engineer more informative features, reduce the amount of regularisation, or train for longer. Collecting more data will not fix high bias. For high variance, make the model less able to memorise: collect more data, add regularisation, choose a simpler model, use dropout or early stopping in neural networks, or average many models with an ensemble method such as bagging.

What you observeDiagnosisWhat to try
Training and validation error both high and close togetherhigh biasmore flexible model, better features, less regularisation, train longer
Low training error, much higher validation errorhigh variancemore data, regularisation, simpler model, dropout, early stopping, bagging

One modern twist is worth knowing about. Very large neural networks sometimes show double descent: as the model grows, test error first follows the classic U shape, peaks around the point where the model has just enough capacity to fit the training data perfectly, and then falls again as the model becomes even larger. This doesn't contradict the bias–variance trade-off, but it shows that for huge models trained with gradient descent, "more parameters" doesn't automatically mean "more overfitting". For the models you'll use most of the time, the classic U-shaped picture remains the right guide.

Interactive: this part runs in your browser.

Interactive. Learning curves: how training and validation error change as you add data.
Intermediate · Maths

Deriving the bias–variance decomposition

📐 Maths you'll need: m-prob
💡 The idea in plain words

Imagine training the same kind of model many times, each on a fresh dataset. Bias is how far the average of those models is from the truth (a systematic mistake). Variance is how much the individual models disagree with each other (sensitivity to the particular sample). Noise is the randomness no model can predict. Expected error is exactly the sum of the three.

🔤 Symbols in this section
f(x)f(x)the true function
f^D(x)\hat f_D(x)model trained on dataset D
fˉ(x)=ED[f^D(x)]\bar f(x) = \mathbb E_D[\hat f_D(x)]average prediction over many datasets
σ2\sigma^2noise variance
ED,ε\mathbb E_{D,\varepsilon}average over datasets and noise

This section proves the bias–variance decomposition. The proof is short, and seeing it explains exactly what "bias" and "variance" mean.

We assume the data come from a true function plus noise: y=f(x)+εy = f(x) + \varepsilon, where the noise ε\varepsilon has mean zero and variance σ2\sigma^2. We train a model on a training set DD, which is itself random (a different sample would give different data), and call the trained model f^D\hat f_D. Now fix a single test input xx and imagine repeating the whole experiment many times with fresh training sets. The average prediction over all those training sets is

fˉ(x)=ED[f^D(x)]. \bar f(x) = \mathbb E_D\big[\hat f_D(x)\big].

Using this, the expected squared error at xx, averaged over both the training sets and the noise, splits into exactly three parts:

ED,ε[(y−f^D(x))2]=σ2⏟noise+(f(x)−fˉ(x))2⏟bias2+ED[(f^D(x)−fˉ(x))2]⏟variance. \mathbb E_{D,\varepsilon}\big[(y - \hat f_D(x))^2\big] = \underbrace{\sigma^2}_{\text{noise}} + \underbrace{\big(f(x) - \bar f(x)\big)^2}_{\text{bias}^2} + \underbrace{\mathbb E_D\big[(\hat f_D(x) - \bar f(x))^2\big]}_{\text{variance}}.

The derivation uses one trick: add and subtract the same quantities so that the error splits into pieces whose cross terms vanish.

Step 1: split the error. Write y=f+εy = f + \varepsilon and insert fˉ\bar f by adding and subtracting it: y−f^D=ε+(f−fˉ)+(fˉ−f^D)y - \hat f_D = \varepsilon + (f - \bar f) + (\bar f - \hat f_D). (We drop "(x)(x)" for brevity.)
Step 2: square and average. Squaring a sum of three terms gives the three squares plus three cross terms: E[ε2]+(f−fˉ)2+ED[(fˉ−f^D)2]+cross terms\mathbb E[\varepsilon^2] + (f - \bar f)^2 + \mathbb E_D[(\bar f - \hat f_D)^2] + \text{cross terms}.
Step 3: the cross terms vanish. Any cross term containing ε\varepsilon averages to zero, because the noise has mean zero and is independent of the training set. The remaining cross term is 2(f−fˉ) ED[fˉ−f^D]2(f - \bar f)\,\mathbb E_D[\bar f - \hat f_D], and ED[f^D]=fˉ\mathbb E_D[\hat f_D] = \bar f by definition, so the bracket averages to zero too.
Step 4: read off the three terms. What remains is E[ε2]=σ2\mathbb E[\varepsilon^2] = \sigma^2 (noise), (f−fˉ)2(f - \bar f)^2 (bias squared) and ED[(f^D−fˉ)2]\mathbb E_D[(\hat f_D - \bar f)^2] (variance). ∎

This makes the two concepts precise. Bias is the distance between the average model and the truth: even if we could average away all the randomness of the training data, a simple model would still be wrong by this much. Variance measures how far an individual trained model typically strays from that average, which is caused by the particular training sample. The noise term is a floor that no model can get below. The decomposition is exact for squared loss; for classification error there are analogous but messier versions, and the intuition carries over.

Key takeaway. Expected squared error = noise + bias² + variance, exactly; simple models err through bias, flexible ones through variance.
Advanced

Variance of ensembles, capacity and double descent

📐 Maths you'll need: m-prob
💡 The idea in plain words

Averaging many models cancels out their individual mistakes, but only the mistakes they don't share. Separately, theory measures how flexible a model class is (its capacity) to predict how much it can overfit. Modern huge networks break the simple story: past a certain size, making them even bigger can reduce test error again.

🔤 Symbols in this section
BBnumber of models averaged
ρ\rhocorrelation between the models' errors
dVCd_{VC}VC dimension (capacity)
δ\deltafailure probability of the bound

This section develops three ideas that build on the decomposition: why averaging models reduces variance, how the flexibility of a model class can be measured, and why very large networks behave in a surprising way.

Averaging reduces variance, up to a limit. Suppose we average the predictions of BB models, each with variance σ2\sigma^2, and suppose any two of them have correlation ρ\rho (they tend to make similar mistakes to that degree). The variance of the average works out to be

Var⁡(1B∑b=1Bf^b)=ρ σ2+1−ρB σ2. \operatorname{Var}\Big(\frac1B\sum_{b=1}^B \hat f_b\Big) = \rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2 .

As we add more models, the second term shrinks towards zero, but the first term, ρσ2\rho\sigma^2, stays no matter how many models we average. If the models were independent (ρ=0\rho = 0), averaging could remove all the variance; if they were identical (ρ=1\rho = 1), averaging would achieve nothing. This is exactly why random forests deliberately make their trees different from each other: lowering ρ\rho lowers the floor. Note that averaging does not change bias, because the average of many similarly biased models is just as biased.

Measuring flexibility. The tendency to overfit depends on how flexible the model class is, and learning theory measures this with quantities such as the VC dimension. The VC dimension of a class is the largest number of points that the class can shatter, meaning that for every possible way of labelling those points, some model in the class classifies all of them correctly. Straight-line classifiers in two dimensions can shatter 3 points (in general position) but not 4, and in general linear classifiers in dd dimensions have VC dimension d+1d + 1. Theory then shows that, with high probability,

R(f)≤R^n(f)+O ⁣(dVClog⁡(n/dVC)+log⁡(1/δ)n), R(f) \le \hat R_n(f) + O\!\left(\sqrt{\frac{d_{VC}\log(n/d_{VC}) + \log(1/\delta)}{n}}\right),

where RR is the true error, R^n\hat R_n the training error, and δ\delta a small failure probability. In words, the gap between training and test error grows with the flexibility of the model class and shrinks as the amount of data grows.

Double descent. Classical theory predicts that test error should keep rising once a model is flexible enough to fit its training data perfectly (the "interpolation threshold", roughly where the number of parameters matches the number of data points). Modern experiments show that test error does peak there, but then falls again as the model grows even larger. The explanation is that a hugely over-parameterised model has infinitely many ways to fit the training data exactly, and gradient descent, starting from small random weights, tends to find a particularly simple one, with small weights. This preference, which nobody explicitly programmed, is called implicit regularisation, and it is part of the reason why modern neural networks generalise well despite having far more parameters than training examples.

Key takeaway. Averaging leaves ρσ² of variance, so ensembles need diverse members; capacity controls overfitting; and very over-parameterised models can show double descent.
Code

Code: measure bias² and variance by simulation

Train many models on fresh datasets and directly estimate each term of the decomposition. Then use scikit-learn's validation and learning curves.

Python · NumPy simulation
import numpy as np

rng = np.random.default_rng(0)
f = lambda x: np.sin(np.pi * x)            # true function
sigma = 0.3                                # noise std
x_test = np.linspace(-1, 1, 50)

def fit_predict(deg, n=25):
    x = rng.uniform(-1, 1, n)
    y = f(x) + rng.normal(0, sigma, n)
    coef = np.polyfit(x, y, deg)
    return np.polyval(coef, x_test)

print(" deg   bias^2   variance   total(+noise)")
for deg in [1, 3, 5, 9, 12]:
    preds = np.array([fit_predict(deg) for _ in range(300)])   # 300 training sets
    f_bar = preds.mean(axis=0)
    bias2 = np.mean((f_bar - f(x_test)) ** 2)
    var = np.mean(preds.var(axis=0))
    print(f"{deg:4d} {bias2:8.4f} {var:10.4f} {bias2 + var + sigma**2:10.4f}")
Open in playground ↗
Python · scikit-learn diagnostics
from sklearn.model_selection import validation_curve, learning_curve
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_friedman1

X, y = make_friedman1(n_samples=600, noise=1.0, random_state=0)

# Validation curve: error vs model complexity (tree depth)
depths = range(1, 15)
tr, va = validation_curve(DecisionTreeRegressor(random_state=0), X, y,
                          param_name="max_depth", param_range=depths,
                          cv=5, scoring="neg_mean_squared_error")
for d_, a, b in zip(depths, -tr.mean(1), -va.mean(1)):
    print(f"depth {d_:2d}: train MSE {a:6.2f}   val MSE {b:6.2f}")

# Learning curve: error vs training-set size (big gap = variance, both high = bias)
sizes, tr, va = learning_curve(DecisionTreeRegressor(max_depth=5), X, y, cv=5,
                               train_sizes=[0.1, 0.3, 0.6, 1.0],
                               scoring="neg_mean_squared_error")
print(sizes, (-tr.mean(1)).round(2), (-va.mean(1)).round(2))
Open in playground ↗
Check yourself

Quiz

1. Train error 2%, validation error 25%. The problem is…

2. Train error 30%, validation error 31%. Best next step?

3. Which is part of the error decomposition?

4. Averaging B models with correlation ρ leaves, as B → ∞, a variance of…

5. The VC dimension of linear classifiers in ℝᵈ is…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready