The two ways a test can be wrong
A hypothesis test makes a decision under uncertainty, and there are exactly two error modes. A Type I error rejects a true null hypothesis: a false alarm, declaring an effect that is not there. A Type II error fails to reject a false null: a miss, walking past a real effect. The courtroom analogy is standard because it is exact: Type I convicts the innocent; Type II frees the guilty.
| H0 is true | H0 is false | |
|---|---|---|
| Reject H0 | Type I error (probability α) | Correct: detection (probability = power) |
| Do not reject H0 | Correct: quiet | Type II error (probability β) |
Alpha: the error rate you choose
The significance level α is the Type I rate you accept in advance, conventionally 0.05. It has a frequency meaning worth stating plainly: if the null were true and you ran this test on endless fresh samples, you would falsely reject about 5% of the time. The p-value does not set this rate; your chosen α does, before the data arrive.
A fair coin flipped 100 times lands 64 heads. Testing fairness: z = 2.8, two-tailed p = 0.0051. At α = 0.05 you reject fairness. If the coin is secretly fair, you have committed a Type I error, one of the unlucky 1-in-200-ish runs this extreme. At α = 0.001 you would not reject, protecting harder against false alarms, at the price of missing more genuinely biased coins: Type II risk rises whenever α tightens, unless sample size compensates.
Beta, power, and the sample-size lever
Power = 1 − β is the probability of catching a real effect of a stated size. It rises with sample size, with the true effect's size, and with α. A study with power 0.807 (for example, n = 32 against a medium effect d = 0.5 in a one-sample t design) still misses about one real effect in five. Underpowered studies are not harmless: their "no significant difference" is close to no information, and their significant results are disproportionately exaggerations.
Reading "not significant" as "no effect". It often means only "this study could not tell". The Type II side of the table is invisible in most reports, which is why effect sizes and confidence intervals belong next to every p-value.
Frequently asked questions
Which error is worse, Type I or Type II?
It depends on the stakes. Approving a harmful drug is a Type I disaster; missing a real treatment is a Type II one. Screening tests tolerate false alarms (Type I) to avoid misses (Type II); confirmatory trials reverse the priority.
Does a p-value of 0.03 mean a 3% chance my rejection is a Type I error?
No. Alpha is a rate across hypothetical repetitions, fixed in advance. A single result either is or is not an error; the p-value measures extremeness under the null, not the probability that this particular decision is wrong.
How do I reduce both errors at once?
More data. Tightening alpha trades one error for the other at fixed n; increasing the sample size (or reducing measurement noise) lowers both. There is no free lunch, only the lunch you pay for with observations.