Statistics for AI Research
Seed variance, confidence intervals, paired comparisons (bootstrap, permutation, McNemar), effect sizes, multiple comparisons, and how to report results honestly.
Results are random variables
Training involves randomness: initialisation, data order, dropout, augmentation, even GPU non-determinism. Run the same configuration with five seeds and you get five different numbers; on small datasets they can differ by several points. The test set is also a sample: a different sample of test examples would give a different score. So a single number answers "what happened once?", not "how good is this method?".
Minimum good practice: run 3–5 seeds (more for small or noisy settings), report mean ± standard deviation or a 95% confidence interval, and make the comparison between methods with an appropriate test. When differences are smaller than seed-to-seed variation, say "no clear difference", which is also a finding.
Try it: is the improvement real?
Two methods, a true difference you control, and noisy runs. See how often a single run, 3 seeds or 10 seeds would lead you to the right conclusion.
Interactive: this part runs in your browser.
Comparing two methods properly
- Paired comparisons are more powerful: evaluate both methods on the same test examples (and, where possible, the same seeds), and analyse the per-example differences.
- Paired bootstrap: resample test examples with replacement many times, recompute the difference in the metric each time; the 2.5th–97.5th percentiles give a 95% CI for the difference. Works for any metric (F1, BLEU, AUC).
- Permutation (randomisation) test: randomly swap the two systems' predictions per example many times to see how often a difference as large as the observed one appears by chance: a p-value without distributional assumptions.
- McNemar's test for two classifiers' correct/incorrect decisions on the same examples: uses only the examples where they disagree.
- Across seeds or datasets: a paired t-test or Wilcoxon signed-rank test on the per-seed (or per-dataset) differences.
Report the effect size (the difference itself, with its CI) not just the p-value: a "significant" 0.1-point gain on a huge test set may be practically meaningless.
Multiple comparisons and p-hacking
If you compare 20 variants with no real effect at p < 0.05, you expect about one "significant" result by chance. Testing many datasets, metrics or hyperparameter settings and highlighting the winners is how false findings get published. Remedies: decide the main comparisons in advance; correct for the number of tests (Bonferroni: divide 0.05 by the number of tests; Holm–Bonferroni: a more powerful step-down version; or control the false discovery rate with Benjamini–Hochberg); and report all the comparisons you made, not only the significant ones.
Try it: the garden of forking paths
Run many comparisons where nothing is really different, and watch "significant" results appear. Then apply a correction.
Interactive: this part runs in your browser.
Code example: seeds and confidence intervals
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: paired bootstrap and permutation tests
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: many comparisons, corrected
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: tests in SciPy
Standard implementations to cross-check your own.
from scipy import stats
import numpy as np
a = np.array([84.1, 83.2, 84.8, 83.9, 84.4]); b = np.array([83.5, 83.0, 84.1, 83.3, 84.2]) # same seeds
print(stats.ttest_rel(a, b)) # paired t-test across seeds
print(stats.wilcoxon(a - b)) # non-parametric alternative
lo, hi = stats.t.interval(0.95, len(a) - 1, loc=a.mean(), scale=stats.sem(a)); print(lo, hi)
from statsmodels.stats.contingency_tables import mcnemar
print(mcnemar([[520, 31], [14, 35]], exact=True).pvalue) # [[both right, only A right], [only B right, both wrong]]
from statsmodels.stats.multitest import multipletests
print(multipletests([0.01, 0.04, 0.03, 0.2], method="holm"))Quiz
1. Why run several seeds?
2. Two models are evaluated on the same 1,000 test examples. The most appropriate comparison is…
3. McNemar's test uses…
4. You test 20 datasets at p < 0.05 with Bonferroni. The per-test threshold is…
5. A difference is statistically significant but 0.05 points. You should…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dror et al., The Hitchhiker's Guide to Testing Statistical Significance in NLP (ACL 2018) ↗ which test to use when
- Agarwal et al., Deep RL at the Edge of the Statistical Precipice (NeurIPS 2021) ↗ reporting uncertainty with few runs