Becoming an AI Researcher
Topic 8 · Part 2 · Ideas & experiments · ⏱ about 10 h

Statistics for AI Research

Seed variance, confidence intervals, paired comparisons (bootstrap, permutation, McNemar), effect sizes, multiple comparisons, and how to report results honestly.

Show:
In this topic
Beginner

Results are random variables

Training involves randomness: initialisation, data order, dropout, augmentation, even GPU non-determinism. Run the same configuration with five seeds and you get five different numbers; on small datasets they can differ by several points. The test set is also a sample: a different sample of test examples would give a different score. So a single number answers "what happened once?", not "how good is this method?".

Minimum good practice: run 3–5 seeds (more for small or noisy settings), report mean ± standard deviation or a 95% confidence interval, and make the comparison between methods with an appropriate test. When differences are smaller than seed-to-seed variation, say "no clear difference", which is also a finding.

Try it

Try it: is the improvement real?

Two methods, a true difference you control, and noisy runs. See how often a single run, 3 seeds or 10 seeds would lead you to the right conclusion.

Interactive: this part runs in your browser.

Beginner

Comparing two methods properly

  • Paired comparisons are more powerful: evaluate both methods on the same test examples (and, where possible, the same seeds), and analyse the per-example differences.
  • Paired bootstrap: resample test examples with replacement many times, recompute the difference in the metric each time; the 2.5th–97.5th percentiles give a 95% CI for the difference. Works for any metric (F1, BLEU, AUC).
  • Permutation (randomisation) test: randomly swap the two systems' predictions per example many times to see how often a difference as large as the observed one appears by chance: a p-value without distributional assumptions.
  • McNemar's test for two classifiers' correct/incorrect decisions on the same examples: uses only the examples where they disagree.
  • Across seeds or datasets: a paired t-test or Wilcoxon signed-rank test on the per-seed (or per-dataset) differences.

Report the effect size (the difference itself, with its CI) not just the p-value: a "significant" 0.1-point gain on a huge test set may be practically meaningless.

Beginner

Multiple comparisons and p-hacking

If you compare 20 variants with no real effect at p < 0.05, you expect about one "significant" result by chance. Testing many datasets, metrics or hyperparameter settings and highlighting the winners is how false findings get published. Remedies: decide the main comparisons in advance; correct for the number of tests (Bonferroni: divide 0.05 by the number of tests; Holm–Bonferroni: a more powerful step-down version; or control the false discovery rate with Benjamini–Hochberg); and report all the comparisons you made, not only the significant ones.

Try it

Try it: the garden of forking paths

Run many comparisons where nothing is really different, and watch "significant" results appear. Then apply a correction.

Interactive: this part runs in your browser.

Try it

Code example: seeds and confidence intervals

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Try it

Code example: paired bootstrap and permutation tests

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Try it

Code example: many comparisons, corrected

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: tests in SciPy

Standard implementations to cross-check your own.

Python · scipy / statsmodels
from scipy import stats
import numpy as np
a = np.array([84.1, 83.2, 84.8, 83.9, 84.4]); b = np.array([83.5, 83.0, 84.1, 83.3, 84.2])   # same seeds
print(stats.ttest_rel(a, b))                 # paired t-test across seeds
print(stats.wilcoxon(a - b))                 # non-parametric alternative
lo, hi = stats.t.interval(0.95, len(a) - 1, loc=a.mean(), scale=stats.sem(a)); print(lo, hi)
from statsmodels.stats.contingency_tables import mcnemar
print(mcnemar([[520, 31], [14, 35]], exact=True).pvalue)    # [[both right, only A right], [only B right, both wrong]]
from statsmodels.stats.multitest import multipletests
print(multipletests([0.01, 0.04, 0.03, 0.2], method="holm"))
Open in playground ↗
Check yourself

Quiz

1. Why run several seeds?

2. Two models are evaluated on the same 1,000 test examples. The most appropriate comparison is…

3. McNemar's test uses…

4. You test 20 datasets at p < 0.05 with Bonferroni. The per-test threshold is…

5. A difference is statistically significant but 0.05 points. You should…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready