Bias, Variance & Overfitting
The central trade-off in all of machine learning.
Underfitting vs overfitting
Every model sits somewhere on a scale from too simple to too complex, and both ends of that scale cause problems. Understanding the two failure modes, and how to tell them apart, is probably the single most useful skill in practical machine learning.
A model that is too simple for the problem cannot capture the real pattern in the data. Imagine fitting a straight line to points that clearly follow a curve: however you place the line, it misses systematically, so the error is high on the training data and equally high on new data. This is called underfitting, and the model is said to have high bias. Bias is error that comes from wrong assumptions built into the model, in this case the assumption that the relationship is a straight line.
A model that is too complex has the opposite problem. It is flexible enough to bend through every training point, including the random noise, so its training error is tiny. But the noise in new data is different, so on new data its predictions are poor. This is called overfitting, and the model is said to have high variance. Variance is error that comes from being too sensitive to the particular sample of data you happened to train on: train the same model on a slightly different sample and you would get a very different fit.
| Underfitting | Good fit | Overfitting | |
|---|---|---|---|
| Model | too simple | about right | too complex |
| Training error | high | low | very low |
| Test error | high | low | high |
| Main problem | high bias | (balanced) | high variance |
These two sources of error, together with the noise that no model can predict, add up to the total expected error on new data:
Making a model more flexible usually lowers its bias but raises its variance, and making it simpler does the reverse. The best model is the one that balances the two, and finding that balance is what model selection, regularisation and cross-validation are all about. (The mathematical derivation of this decomposition is in the Intermediate section below.)
Interactive: this part runs in your browser.
Watch it: Underfitting, a good fit, and overfitting
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Try it: polynomial degree
Increase the degree. Low degree = straight-ish line (underfit). Very high degree = wild wiggles (overfit). The lower chart shows the classic U-shaped test error.
Interactive: this part runs in your browser.
How to diagnose and fix
To fix a model, you first need to know which problem it has, and the key evidence comes from comparing its error on the training data with its error on validation data.
A very useful diagnostic is the learning curve, which plots both errors as you train on more and more data. If the model has high bias, both curves level off quickly at a high error, close to each other: the model has already learned all it can, and more data won't help because it simply isn't flexible enough. If the model has high variance, the training error is low but the validation error is much higher, and there is a large gap between the curves; as more data is added, the gap slowly narrows, because more examples make it harder to memorise noise.
Each diagnosis calls for different remedies, and applying the wrong one wastes time. For high bias, make the model more capable: use a more flexible model (for example, add polynomial features or move from a linear model to a tree ensemble), engineer more informative features, reduce the amount of regularisation, or train for longer. Collecting more data will not fix high bias. For high variance, make the model less able to memorise: collect more data, add regularisation, choose a simpler model, use dropout or early stopping in neural networks, or average many models with an ensemble method such as bagging.
| What you observe | Diagnosis | What to try |
|---|---|---|
| Training and validation error both high and close together | high bias | more flexible model, better features, less regularisation, train longer |
| Low training error, much higher validation error | high variance | more data, regularisation, simpler model, dropout, early stopping, bagging |
One modern twist is worth knowing about. Very large neural networks sometimes show double descent: as the model grows, test error first follows the classic U shape, peaks around the point where the model has just enough capacity to fit the training data perfectly, and then falls again as the model becomes even larger. This doesn't contradict the bias–variance trade-off, but it shows that for huge models trained with gradient descent, "more parameters" doesn't automatically mean "more overfitting". For the models you'll use most of the time, the classic U-shaped picture remains the right guide.
Interactive: this part runs in your browser.
Deriving the bias–variance decomposition
Imagine training the same kind of model many times, each on a fresh dataset. Bias is how far the average of those models is from the truth (a systematic mistake). Variance is how much the individual models disagree with each other (sensitivity to the particular sample). Noise is the randomness no model can predict. Expected error is exactly the sum of the three.
🔤 Symbols in this section
| the true function | |
| model trained on dataset D | |
| average prediction over many datasets | |
| noise variance | |
| average over datasets and noise |
This section proves the bias–variance decomposition. The proof is short, and seeing it explains exactly what "bias" and "variance" mean.
We assume the data come from a true function plus noise: , where the noise has mean zero and variance . We train a model on a training set , which is itself random (a different sample would give different data), and call the trained model . Now fix a single test input and imagine repeating the whole experiment many times with fresh training sets. The average prediction over all those training sets is
Using this, the expected squared error at , averaged over both the training sets and the noise, splits into exactly three parts:
The derivation uses one trick: add and subtract the same quantities so that the error splits into pieces whose cross terms vanish.
This makes the two concepts precise. Bias is the distance between the average model and the truth: even if we could average away all the randomness of the training data, a simple model would still be wrong by this much. Variance measures how far an individual trained model typically strays from that average, which is caused by the particular training sample. The noise term is a floor that no model can get below. The decomposition is exact for squared loss; for classification error there are analogous but messier versions, and the intuition carries over.
Variance of ensembles, capacity and double descent
Averaging many models cancels out their individual mistakes, but only the mistakes they don't share. Separately, theory measures how flexible a model class is (its capacity) to predict how much it can overfit. Modern huge networks break the simple story: past a certain size, making them even bigger can reduce test error again.
🔤 Symbols in this section
| number of models averaged | |
| correlation between the models' errors | |
| VC dimension (capacity) | |
| failure probability of the bound |
This section develops three ideas that build on the decomposition: why averaging models reduces variance, how the flexibility of a model class can be measured, and why very large networks behave in a surprising way.
Averaging reduces variance, up to a limit. Suppose we average the predictions of models, each with variance , and suppose any two of them have correlation (they tend to make similar mistakes to that degree). The variance of the average works out to be
As we add more models, the second term shrinks towards zero, but the first term, , stays no matter how many models we average. If the models were independent (), averaging could remove all the variance; if they were identical (), averaging would achieve nothing. This is exactly why random forests deliberately make their trees different from each other: lowering lowers the floor. Note that averaging does not change bias, because the average of many similarly biased models is just as biased.
Measuring flexibility. The tendency to overfit depends on how flexible the model class is, and learning theory measures this with quantities such as the VC dimension. The VC dimension of a class is the largest number of points that the class can shatter, meaning that for every possible way of labelling those points, some model in the class classifies all of them correctly. Straight-line classifiers in two dimensions can shatter 3 points (in general position) but not 4, and in general linear classifiers in dimensions have VC dimension . Theory then shows that, with high probability,
where is the true error, the training error, and a small failure probability. In words, the gap between training and test error grows with the flexibility of the model class and shrinks as the amount of data grows.
Double descent. Classical theory predicts that test error should keep rising once a model is flexible enough to fit its training data perfectly (the "interpolation threshold", roughly where the number of parameters matches the number of data points). Modern experiments show that test error does peak there, but then falls again as the model grows even larger. The explanation is that a hugely over-parameterised model has infinitely many ways to fit the training data exactly, and gradient descent, starting from small random weights, tends to find a particularly simple one, with small weights. This preference, which nobody explicitly programmed, is called implicit regularisation, and it is part of the reason why modern neural networks generalise well despite having far more parameters than training examples.
Code: measure bias² and variance by simulation
Train many models on fresh datasets and directly estimate each term of the decomposition. Then use scikit-learn's validation and learning curves.
import numpy as np
rng = np.random.default_rng(0)
f = lambda x: np.sin(np.pi * x) # true function
sigma = 0.3 # noise std
x_test = np.linspace(-1, 1, 50)
def fit_predict(deg, n=25):
x = rng.uniform(-1, 1, n)
y = f(x) + rng.normal(0, sigma, n)
coef = np.polyfit(x, y, deg)
return np.polyval(coef, x_test)
print(" deg bias^2 variance total(+noise)")
for deg in [1, 3, 5, 9, 12]:
preds = np.array([fit_predict(deg) for _ in range(300)]) # 300 training sets
f_bar = preds.mean(axis=0)
bias2 = np.mean((f_bar - f(x_test)) ** 2)
var = np.mean(preds.var(axis=0))
print(f"{deg:4d} {bias2:8.4f} {var:10.4f} {bias2 + var + sigma**2:10.4f}")from sklearn.model_selection import validation_curve, learning_curve
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_friedman1
X, y = make_friedman1(n_samples=600, noise=1.0, random_state=0)
# Validation curve: error vs model complexity (tree depth)
depths = range(1, 15)
tr, va = validation_curve(DecisionTreeRegressor(random_state=0), X, y,
param_name="max_depth", param_range=depths,
cv=5, scoring="neg_mean_squared_error")
for d_, a, b in zip(depths, -tr.mean(1), -va.mean(1)):
print(f"depth {d_:2d}: train MSE {a:6.2f} val MSE {b:6.2f}")
# Learning curve: error vs training-set size (big gap = variance, both high = bias)
sizes, tr, va = learning_curve(DecisionTreeRegressor(max_depth=5), X, y, cv=5,
train_sizes=[0.1, 0.3, 0.6, 1.0],
scoring="neg_mean_squared_error")
print(sizes, (-tr.mean(1)).round(2), (-va.mean(1)).round(2))Quiz
1. Train error 2%, validation error 25%. The problem is…
2. Train error 30%, validation error 31%. Best next step?
3. Which is part of the error decomposition?
4. Averaging B models with correlation ρ leaves, as B → ∞, a variance of…
5. The VC dimension of linear classifiers in ℝᵈ is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- An Introduction to Statistical Learning (free book) ↗ James, Witten, Hastie, Tibshirani (Python edition available)