§ 01What Is a Hypothesis Test?
A hypothesis test is a formal procedure for using sample data to decide between two competing claims about a population parameter. One claim is assumed true until evidence forces us to reconsider it; the other is what we suspect or wish to detect.
Suppose a factory claims its machines fill bags to a mean weight of 500 g. You weigh a random sample of 40 bags and find a sample mean of 496 g. Is this discrepancy just random variation, or is there real evidence that the machine is under-filling? Hypothesis testing gives you a principled, quantitative answer.
The key insight is that we never prove anything with a sample — we only assess the strength of the evidence against an assumed baseline. This asymmetry shapes every part of the procedure.
2. Choose significance level α
3. Compute the test statistic
4. Find the p-value (or critical value)
5. Reject or fail to reject H0
§ 02The Null and Alternative Hypotheses
The null hypothesis H0 is the default claim: typically one of "no effect," "no difference," or "equal to a specific value." It is the statement we assume true and attempt to disprove. The alternative hypothesis H1 (also written Ha) is what we believe is true if H0 is false.
Null Hypothesis H0
Always contains an equality. Expressed as:
- μ = μ0 (population mean equals a value)
- p = p0 (population proportion equals a value)
- μ1 = μ2 (two means are equal)
Alternative Hypothesis H1
Specifies the direction of the claimed effect:
- μ ≠ μ0 (two-tailed: any difference)
- μ > μ0 (right-tailed: mean has increased)
- μ < μ0 (left-tailed: mean has decreased)
The direction of H1 determines the tail of the test. A two-tailed test splits the significance level equally between both tails; a one-tailed test concentrates it in one tail and is more powerful for detecting the specific direction stated in H1.
§ 03Significance Level α and the p-Value
The significance level α is the probability threshold below which we consider the evidence against H0 sufficient to reject it. Common choices are α = 0.05 (5%), α = 0.01 (1%), and α = 0.10 (10%). It is set in advance, before computing any test statistic.
The p-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming H0 is true. A small p-value means the observed data would be very unlikely under H0, which is evidence against it.
If p-value > α → Fail to reject H0
"Fail to reject" is not the same as "accept." We never prove H0 is true; we only lack sufficient evidence to reject it.
The Critical Value Approach
An equivalent method avoids computing an exact p-value. Instead, find the critical value zα (or tα,ν) that cuts off the rejection region. If the test statistic falls in the rejection region, reject H0. The two approaches always give the same decision.
§ 04Type I and Type II Errors
Hypothesis testing involves two kinds of mistakes. Understanding them is essential for choosing α and interpreting results responsibly.
| Error Type | What Happened | Probability | Also Called |
|---|---|---|---|
| Type I (α) | Rejected H0 when H0 was actually true (false positive) | P(Type I) = α | False positive; significance level |
| Type II (β) | Failed to reject H0 when H1 was actually true (false negative) | P(Type II) = β | False negative; missed detection |
| Power (1−β) | Correctly rejected H0 when H1 was true | 1 − β | Sensitivity; true positive rate |
There is an inherent trade-off: reducing α (making Type I errors less likely) automatically increases β (making Type II errors more likely), for a fixed sample size. The only way to reduce both simultaneously is to increase the sample size n.
§ 05The z-Test for a Population Mean
When the population standard deviation σ is known and either the population is normal or n ≥ 30 (by the Central Limit Theorem), use the z-test. The test statistic measures how many standard errors the sample mean x̄ is from the hypothesised mean μ0.
Under H0, this statistic follows a standard normal distribution Z ∼ N(0,1). You then find the p-value from the z-table and compare to α.
A machine is set to fill bags to μ = 500 g with σ = 12 g. A sample of n = 36 bags gives x̄ = 495 g. At α = 0.05, is there evidence the machine is mis-calibrated?
p = 0.0124 < α = 0.05, so we reject H0.
A courier claims mean delivery time μ = 3.2 days. Customers suspect it is higher. A sample of n = 50 deliveries gives x̄ = 3.5 days. Population σ = 1.1 days. Test at α = 0.01.
z = 1.93 < 2.326, so we fail to reject H0.
§ 06The t-Test When σ Is Unknown
In practice, the population standard deviation σ is almost never known. When it must be estimated from the sample using s, the test statistic follows a t-distribution with ν = n − 1 degrees of freedom, not the standard normal. The t-distribution has heavier tails, reflecting the additional uncertainty introduced by estimating σ.
As n grows, the t-distribution approaches the standard normal: for n ≥ 30, the difference is practically negligible. For small samples (n < 30), the t-distribution matters considerably.
A university claims the mean exam score is μ = 68. A lecturer samples n = 16 students and finds x̄ = 72, s = 8. Is there evidence the class mean is higher than claimed? Test at α = 0.05.
t = 2.00 > 1.753, so we reject H0.
Group A (n1 = 12): x̄1 = 75, s1 = 6. Group B (n2 = 15): x̄2 = 70, s2 = 7. Test H0: μ1 = μ2 vs H1: μ1 ≠ μ2 at α = 0.05, assuming equal variances.
§ 07Hypothesis Test for a Population Proportion
When the parameter of interest is a proportion p rather than a mean, the test statistic is still a z-statistic, derived from the normal approximation to the binomial. The conditions np0 ≥ 10 and n(1−p0) ≥ 10 must hold.
A manufacturer claims at most 4% of products are defective. In a sample of n = 200, 12 are defective (p̂ = 0.06). Test H0: p = 0.04 vs H1: p > 0.04 at α = 0.05.
A candidate needs more than 50% of the vote to win. A poll of n = 400 voters finds 210 in favour (p̂ = 0.525). Is there significant evidence the candidate will win? Test at α = 0.05.
§ 08One-Tailed vs Two-Tailed Tests
Choosing between one-tailed and two-tailed is a conceptual decision that must be made before the data is collected, based on what effect the researcher is actually interested in detecting.
A new drug is tested for its effect on blood pressure. Researchers want to know if it lowers blood pressure (a decrease from μ = 120 mmHg). They observe x̄ = 115 in n = 25 patients with s = 10. Test at α = 0.05.
§ 09Further Worked Examples
10 employees complete a typing speed test before and after a training course. The mean difference (after − before) is d̄ = 8.5 wpm with sd = 4.2 wpm. Test H0: μd = 0 vs H1: μd > 0 at α = 0.05.
A website historically converts 8% of visitors. After a redesign, 56 out of 600 visitors converted (p̂ = 0.0933). Did the redesign change the rate? Test H0: p = 0.08 vs H1: p ≠ 0.08 at α = 0.05.
Two researchers test the same hypothesis. One gets p = 0.048, the other gets p = 0.052. With α = 0.05, the first rejects H0 and the second does not. This illustrates an important point about the threshold.
The difference between p = 0.048 and p = 0.052 is trivially small, no meaningful distinction in the strength of evidence. Yet the binary reject/fail-to-reject outcome differs. This is a known weakness of rigid α-thresholds.
Always report the exact p-value alongside the decision. "p = 0.052, fail to reject at α = 0.05" conveys far more than simply "not significant." Effect size and confidence intervals should accompany hypothesis tests.
§ 10Common Mistakes and Misconceptions
§ 11The Duality with Confidence Intervals
Hypothesis testing and confidence intervals are two sides of the same coin. A two-tailed test at significance level α rejects H0: μ = μ0 if and only if μ0 falls outside the corresponding (1−α)×100% confidence interval.
Both approaches yield identical decisions for two-tailed tests. Confidence intervals carry more information, since they show the plausible range of parameter values, not just a binary decision.
Continue the Statistics Series
§ 12Quiz — Test Your Understanding
Ten questions covering all aspects of hypothesis testing. Immediate feedback on each answer.