What α actually buys
The significance level α is a budget for false alarms: test a true null at α = 0.05 and you will wrongly reject it 5% of the time in the long run. That calibration holds per test. The trap this page exists to expose: budgets do not stay per-test when you run many tests. Twenty independent tests at 0.05, all nulls true, give a 1 − 0.9520 = 0.6415 chance of at least one false alarm somewhere, and an expected 1.0 false positives in the batch. A "significant" result pulled from a wide scan is often just the budget being spent.
One test: family-wise error 5%. Ten tests: 1 − 0.9510 = 0.4013. A hundred: 0.9941, near certainty, with 5 false positives expected. Genomics and A/B testing platforms live in the right-hand column of that table, which is why multiple-testing corrections are not pedantry: without them, large honest studies would manufacture findings on schedule.
The corrections, and their price
Bonferroni spends the budget evenly: test each of m hypotheses at α/m and the family rate stays under α. Šidák does slightly better under independence: 1 − (1 − α)1/m per test. Both work; both cost power, since real effects must now clear a higher bar (at m = 20, a per-test 0.0025). The modern compromise in large-scale work is controlling the false discovery rate instead, accepting a managed share of false hits among the rejections rather than near-zero family risk. Choosing between them is a decision about which error hurts more, and it belongs in the methods section, written before the data arrive.
Treating p < 0.05 as "95% likely the effect is real". α is a frequency of false alarms among true nulls, not a probability that this particular hypothesis is true or false. The confusion has a name, the base-rate fallacy, and the correction machinery above does not fix it.
Frequently asked questions
What is the family-wise error rate?
The probability that a batch of tests produces at least one false positive: 1 - (1 - alpha)^m for m independent tests, which the calculator above computes for your numbers.
When must I correct for multiple testing?
Whenever one conclusion draws on several tests: many outcomes, many subgroups, many pairwise comparisons. If any single false hit would be reported as a finding, correct.
Is 0.05 a law?
No, it is a convention Fisher popularised. Fields pick their own: particle physics demands 5 sigma, some journals push for 0.005, exploratory work often uses 0.10 with the choice stated openly.