You can never measure everyone — but a well-chosen sample, analysed with the right tools,
tells you a remarkable amount about the whole. Understand how samples behave, why the
Central Limit Theorem is the most powerful result in statistics, and how to build trustworthy
estimates of population parameters.
CLTKey Theorem
10Worked Examples
3Estimator Properties
10Quiz Questions
Share this page
§ 01Population vs Sample
Statistical inference begins with a fundamental distinction: the population is every
individual we care about, and the sample is the subset we actually measure. Almost
every real-world study works with a sample, because measuring an entire population is
impossible, too expensive, or destructive.
A parameter is a numerical summary of a population (e.g. the true mean μ
or proportion p). A statistic is the corresponding numerical summary computed
from a sample (e.g. x̄ or p̂). Because samples vary, so do statistics — and
understanding that variability is the entire point of sampling theory.
Quantity
Population Parameter
Sample Statistic
Mean
μ — fixed, unknown
x̄ = (1/n)∑xi — varies by sample
Variance
σ² — fixed, unknown
s² = ∑(xi−x̄)²/(n−1) — varies
Standard Deviation
σ
s
Proportion
p — fixed, unknown
p̂ = x/n — varies by sample
Size
N (often infinite or very large)
n (what we choose)
The Core Problem
We want to know μ (or p or σ), but we can only observe x̄ (or p̂ or s).
The question is: how reliably does the sample statistic reflect the population parameter,
and how do we quantify the uncertainty?
§ 02Random Sampling and Sampling Methods
For a sample statistic to be a useful estimate of a population parameter, the sample must be
collected in a way that gives every member of the population a fair chance of being selected.
Non-random samples introduce bias — a systematic tendency to over- or
under-represent certain parts of the population — which no amount of statistical
analysis can correct.
Simple Random Sample (SRS)
Every subset of size n equally likelyThe gold standard. Each individual is chosen independently with equal probability. Requires a complete list (sampling frame) of the population. Forms the basis of most statistical theory.
Stratified Sampling
SRS within each subgroupDivide the population into homogeneous strata (e.g. age groups), then take an SRS from each stratum. More precise than SRS when strata differ substantially. Guarantees representation.
Cluster Sampling
Sample whole groups randomlyDivide the population into clusters (e.g. schools), randomly select clusters, then sample everyone in selected clusters. Practical and cost-effective for geographically spread populations.
Convenience Sampling
NOT random — avoidSelecting whoever is easiest to reach (e.g. surveying friends, online volunteers). Highly prone to selection bias. Results cannot be generalised to the population with any statistical validity.
Example 01Identifying bias in a sampling design
A school wants to estimate the average number of hours students spend on homework per week. They survey students who stay late in the library. Identify the problem.
Problem
Library-staying students are likely to spend more time on homework than the average student. This is selection bias — the sample systematically over-represents diligent students.
Fix
Use a simple random sample from the complete class register, giving every student an equal chance of being selected regardless of study habits.
Key Point
Bias in collection cannot be undone by statistical analysis. Get the design right first.
§ 03Point Estimation
A point estimator is a statistic — a function of the sample —
used to estimate a single unknown population parameter. The number it produces for a
particular sample is called a point estimate.
Common Point Estimators
x̄ estimates μ (population mean)
s² estimates σ² (population variance)
p̂ = x/n estimates p (population proportion)
Median estimates the population median
Sample range estimates population range (poorly)
Estimator vs Estimate
The estimator is the rule or formula (a random variable). The estimate is the specific number computed from one sample.
Example: x̄ = (x1+…+xn)/n is the estimator. For a sample {4, 7, 9, 6}, the estimate is x̄ = 6.5.
Three Desirable Properties of Estimators
Unbiasedness
E[θ̂] = θOn average, across all possible samples, the estimator hits the true parameter exactly. No systematic over- or under-estimation. The sample mean x̄ is unbiased for μ. The sample variance s² = ∑(xᵢ−x̄)²/(n−1) is unbiased for σ² (the divisor n−1 is crucial).
Consistency
θ̂ → θ as n → ∞As the sample size grows, the estimator converges in probability to the true parameter. Larger samples give more accurate estimates. The sample mean is consistent for μ (by the Law of Large Numbers).
Efficiency
Smallest Var(θ̂) among unbiased estimatorsAmong all unbiased estimators, an efficient estimator has the smallest variance. The sample mean is more efficient than the sample median for estimating μ in a normal population (it has smaller sampling variance).
Example 02Why divide by n − 1 for the sample variance?
Show why the unbiased sample variance uses n−1 rather than n in the denominator.
Intuition
The deviations (xi − x̄) are computed around the sample mean x̄, not the true mean μ. Since x̄ is chosen to minimise ∑(xi−x̄)², the sum is always smaller than it would be around μ. Dividing by n−1 corrects for this underestimation.
Formal Result
E[∑(xi−x̄)²] = (n−1)σ²
Therefore E[s²] = E[∑(xi−x̄)² / (n−1)] = σ² ✓
Key Takeaway
Dividing by n gives a biased (too small) estimate of σ². Always use n−1 for s².
Example 03Computing point estimates from a sample
A sample of n = 6 battery lifetimes (hours): 48, 52, 55, 47, 51, 53. Compute x̄, s², and s.
x̄ = 51 h (estimates μ) s² = 9.2 h² (estimates σ²) s ≈ 3.03 h (estimates σ)
§ 04The Sampling Distribution of x̄
If we drew many different samples of size n from the same population and computed x̄ each
time, those values would form a distribution — the sampling distribution of
x̄. This distribution describes how x̄ varies across samples, and its
properties are the foundation of all inferential statistics.
Sampling Distribution of the Sample Mean
E[x̄] = μ (mean of x̄ equals population mean)
Var(x̄) = σ²/n (variance decreases as n grows)
SE(x̄) = σ/√n (standard error of the mean)
These results hold regardless of the population distribution, for any sample size n.
The standard error SE = σ/√n is perhaps the single most important
quantity in sampling theory. It measures the typical distance between x̄ and μ.
Notice it decreases as n increases — larger samples produce more precise estimates,
but with diminishing returns: doubling precision requires quadrupling the sample size.
Example 04Computing the standard error and its dependence on n
A population has μ = 80 and σ = 20. Find SE(x̄) for n = 4, 16, 25, 100.
Formula
SE = σ/√n = 20/√n
Results
n=4: SE = 20/2 = 10 n=16: SE = 20/4 = 5n=25: SE = 20/5 = 4 n=100: SE = 20/10 = 2
Observation
To halve the SE from 10 to 5, n must quadruple (4 → 16). Precision improves with √n, not n.
Example 05Exact sampling distribution (normal population)
If the population is normal: X ∼ N(120, 15²), find the distribution of x̄ for n = 9. What is P(x̄ > 125)?
There is a 15.87% chance that a sample of n = 9 gives x̄ > 125.
§ 05The Central Limit Theorem
When the population is normal, x̄ is exactly normal for any n. But what if the population
is skewed, discrete, or has an unusual shape? The Central Limit Theorem (CLT) provides a
remarkable answer.
The Central Limit Theorem
If X1, X2, …, Xn are i.i.d. with mean μ and finite variance σ², then as n → ∞:
x̄ ∼approx N(μ, σ²/n)
Equivalently: Z = (x̄ − μ) / (σ/√n) → N(0,1)
The approximation is typically excellent for n ≥ 30, and often for much smaller n if the population is not highly skewed.
The CLT is the reason the normal distribution appears everywhere in statistics. It means that
regardless of how a population is distributed — whether it is skewed, bimodal, uniform,
or exponential — sample means from large enough samples behave approximately normally.
This is what makes it possible to construct confidence intervals and perform hypothesis tests
without knowing the population distribution.
Why the CLT Is So Powerful
Every probability result about x̄ derived from the normal distribution is applicable to
any population, as long as n is large enough. Polling, quality control, clinical trials,
and financial risk models all depend on this single theorem.
Example 06CLT applied to a skewed population
Customer service call durations follow an exponential distribution with mean μ = 4 minutes and σ = 4 minutes (exponential: σ = μ). A sample of n = 64 calls is taken. Find P(x̄ < 3.5).
CLT Setup
n = 64 ≥ 30, so by the CLT: x̄ ∼ N(4, 4²/64) = N(4, 0.25)
SE = 4/√64 = 4/8 = 0.5
Standardise
Z = (3.5 − 4) / 0.5 = −0.5/0.5 = −1.00
Probability
P(x̄ < 3.5) = P(Z < −1.00) = 1 − Φ(1.00) = 0.1587
Result
Despite the exponential (skewed) population, P(x̄ < 3.5) ≈ 15.87% via the CLT.
Example 07CLT applied to a sum of random variables
A factory machine produces parts with weight μ = 200 g and σ = 8 g. A box holds n = 36 parts. What is P(total weight > 7300 g)?
Reformulate
Total weight T = ∑Xi = n×x̄. So P(T > 7300) = P(x̄ > 7300/36) = P(x̄ > 202.78).
When the parameter of interest is a population proportion p, the sample proportion
p̂ = X/n (where X is the count of successes) is the natural estimator. By the CLT
applied to a Bernoulli population, p̂ has an approximately normal sampling distribution
for large samples.
Rule of thumb: approximation is good when np ≥ 10 and n(1−p) ≥ 10.
Example 08Sampling distribution of a proportion
In a large city, 35% of residents recycle regularly (p = 0.35). A survey samples n = 200 residents. Find the mean and standard error of p̂, and find P(p̂ > 0.40).
There is about a 6.9% chance that a random sample of 200 shows more than 40% recycling.
§ 07The Law of Large Numbers
The CLT tells us about the shape of the sampling distribution. The Law of Large
Numbers (LLN) tells us about where it centres as n grows. There are two versions,
but both say the same thing intuitively: averages of large samples converge to the true mean.
Weak LLN
For any ε > 0:
P(|x̄ − μ| > ε) → 0 as n → ∞
The sample mean converges in probability to μ. For any fixed tolerance, the probability of being outside it goes to zero.
Strong LLN
P(x̄ → μ as n → ∞) = 1
The sample mean converges almost surely to μ. With probability 1, every sequence of sample means eventually stays arbitrarily close to μ. Stronger than the Weak LLN.
The LLN underpins the consistency property of x̄ and justifies long-run frequency
interpretations of probability. It is also why the insurance industry works: individual
claims are random, but the average claim over millions of policies converges reliably to
the expected value.
Example 09LLN and the CLT working together — sample size determination
A quality engineer wants x̄ to be within 1.5 units of μ with 95% probability. The population has σ = 10. How large must n be?
Set Up
By CLT, x̄ ∼ N(μ, σ²/n). We want P(|x̄ − μ| ≤ 1.5) = 0.95.
n = 171 (always round up to guarantee the required precision)
§ 08Unknown σ and the t-Distribution
In the CLT results above, we assumed σ is known. In practice it almost never is, so we
substitute the sample standard deviation s. This introduces extra uncertainty, and the
standardised sample mean no longer follows the standard normal — it follows a
t-distribution with n−1 degrees of freedom.
Studentised Sample Mean
T = (x̄ − μ) / (s/√n) ∼ tn−1
The t-distribution has heavier tails than the normal — reflecting extra uncertainty from estimating σ.
As n → ∞, tn−1 → N(0,1).
The t-distribution was discovered by William Gosset (writing as “Student”) while
working at Guinness Brewery, where he needed to make inferences from small samples of barley.
It is now the standard distribution used in confidence intervals and hypothesis tests when
σ is estimated.
Use z (Normal)
σ known, any n; or n ≥ 30When the population SD is known, or when n is large enough that s ≈ σ reliably, the z-distribution gives a valid and slightly tighter result.
Use t
σ unknown, any nWhen σ must be estimated by s. Always valid for normal populations. For non-normal populations, the CLT ensures approximate validity when n ≥ 30.
Small n, non-normal
Caution requiredWith n < 30 and an unknown, non-normal population, both z and t approximations can be unreliable. Consider non-parametric methods or bootstrap confidence intervals.
Example 10Probability under the t-distribution
A sample of n = 12 measurements gives x̄ = 47.3 and s = 5.8. The population mean is claimed to be μ = 50. Compute the t-statistic and find the p-value for a two-tailed test.
From t-tables: P(|t11| > 1.612) ≈ 0.136 (two-tailed)
Conclusion
p ≈ 0.136 > 0.05 → insufficient evidence to reject μ = 50 at the 5% level.
§ 09Common Mistakes and Misconceptions
Confusing SE with SD
SE = σ/√n ≠ σThe standard deviation σ (or s) describes spread in the population (or sample). The standard error SE describes how much x̄ varies between samples. They differ by a factor of √n.
Using n instead of n−1
s² = ∑(xᵢ−x̄)² / (n−1)Dividing by n gives the biased estimator. Most software uses n−1 by default (called the “corrected” or “unbiased” sample variance). Always check which formula your calculator uses.
Over-relying on n ≥ 30
Rule of thumb, not a lawThe “n ≥ 30 is enough for CLT” rule is a starting point. Heavily skewed or thick-tailed populations may need n ≥ 100 or more for good normal approximation. Always check residuals.
Larger sample = no bias
n ↑ reduces variance, not biasA larger biased sample is still biased. The LLN only works when E[x̄] = μ, i.e., the estimator is unbiased. Convenience samples remain biased regardless of size.
Using z when t is needed
Unknown σ → always use tReplacing s with z is anti-conservative: it understates the tails and gives too-narrow intervals. Use tn−1 whenever σ is estimated from data.
Interpreting SE as uncertainty about individuals
SE concerns x̄, not individualsThe SE of the mean is NOT the spread of individual data. It only measures how precisely we have estimated the population mean. An individual prediction requires the full SD, not the SE.
§ 10Summary — The Inferential Chain
Sampling and estimation form the bridge between raw data and population knowledge. Every
subsequent technique — confidence intervals, hypothesis tests, regression — rests
on the ideas introduced here.
The Key Results at a Glance
x̄ is unbiased, consistent, and efficient for μ
s² (with n−1) is unbiased for σ²
SE(x̄) = σ/√n → precision improves with √n
CLT: x̄ ≈ N(μ, σ²/n) for large n, any population
Unknown σ: use T = (x̄−μ)/(s/√n) ∼ tn−1
Where to Go Next
The sampling distribution of x̄ derived here is the direct input to confidence intervals
(which give a range of plausible values for μ) and hypothesis tests (which formally
assess a specific claim about μ). Master the CLT and standard error, and the rest of
inferential statistics follows naturally.