§ 01What Is Linear Regression?
Linear regression models the relationship between a response variable y and one or more predictor variables x by fitting a straight line (or hyperplane) through the data. The goal is both to understand the relationship and to make predictions.
Consider predicting a student’s exam score from the number of hours studied, or a house’s sale price from its floor area. In each case, there is a quantitative response y and a quantitative predictor x, and we want to describe how y tends to change as x changes. Linear regression provides a precise, testable, and predictive answer.
This page covers simple linear regression — one predictor, one response. The ideas extend naturally to multiple regression (several predictors), but simple regression contains every key concept and is the essential foundation.
The parameters β0 and β1 are unknown population values. We estimate them from data using sample estimates b0 and b1, giving the fitted line ŷ = b0 + b1x (read “y-hat”).
§ 02The Least Squares Criterion
With n data points (x1, y1), …, (xn, yn), infinitely many lines could be drawn through or near the scatter. We need a unique, optimal choice. The least squares method selects the line that minimises the sum of squared residuals (SSE):
Deriving the Normal Equations
Minimise SSE by taking partial derivatives with respect to b0 and b1 and setting both to zero. This yields the normal equations:
∑xiyi = b0∑xi + b1∑xi2
Solving these two simultaneous equations gives the unique least squares estimates.
The Closed-Form Solution
Solving the normal equations algebraically produces compact formulas for the slope and intercept that can be computed directly from the data.
b0 = ȳ − b1x̄
A computationally convenient equivalent for Sxy and Sxx avoids computing deviations explicitly:
Sxy = ∑xiyi − n x̄ȳ
These are faster to compute in a summary table and give identical results.
Five students studied x hours and scored y marks. Find the least squares regression line.
| i | xᵢ (hours) | yᵢ (marks) | xᵢ² | xᵢyᵢ |
|---|---|---|---|---|
| 1 | 1 | 40 | 1 | 40 |
| 2 | 2 | 50 | 4 | 100 |
| 3 | 3 | 55 | 9 | 165 |
| 4 | 4 | 65 | 16 | 260 |
| 5 | 5 | 70 | 25 | 350 |
| Σ | 15 | 280 | 55 | 915 |
§ 03Interpreting the Coefficients
Getting numbers for b0 and b1 is straightforward. Understanding what they mean in context is equally important.
A regression of house price y (in £1000s) on floor area x (m²) gives ŷ = 120 + 2.4x.
For each additional m² of floor area, the predicted price increases by £2,400 (b1 = 2.4 in units of £1000/m²).
A house with 0 m² would be predicted to cost £120,000 — a physically meaningless extrapolation (no house has zero area). The intercept is a mathematical anchor, not an interpretable quantity here.
§ 04Residuals — Measuring What the Model Misses
The residual for observation i is the difference between the actual and fitted value: ei = yi − ŷi. Residuals carry all the information the model did not capture. Analysing them reveals whether the model’s assumptions hold.
∑ xiei = 0 (residuals are uncorrelated with x)
The fitted line passes through (x̄, ȳ)
These three properties hold for every least squares regression, by construction.
The Assumptions to Check
What We Assume
- Linearity: the true relationship is linear
- Independence: observations are independent
- Normality: errors εi ∼ N(0, σ²)
- Homoscedasticity: constant variance σ²
How to Check
- Plot residuals vs fitted values (look for curves or fans)
- Normal Q-Q plot of residuals (check for straight line)
- Scale-location plot (check variance is constant)
- Residuals vs order plot (check independence over time)
Using the fitted line ŷ = 33.5 + 7.5x from Example 01, compute the residuals.
| xᵢ | yᵢ | ŷᵢ = 33.5 + 7.5xᵢ | eᵢ = yᵢ − ŷᵢ | eᵢ² |
|---|---|---|---|---|
| 1 | 40 | 41.0 | −1.0 | 1.00 |
| 2 | 50 | 48.5 | +1.5 | 2.25 |
| 3 | 55 | 56.0 | −1.0 | 1.00 |
| 4 | 65 | 63.5 | +1.5 | 2.25 |
| 5 | 70 | 71.0 | −1.0 | 1.00 |
| Σ | — | — | 0.0 | SSE = 7.50 |
§ 05R² — The Coefficient of Determination
R² measures what fraction of the total variation in y is explained by the regression on x. It is the single most widely used summary of regression fit.
R² = SSR / SST = 1 − SSE / SST
R² always lies between 0 and 1. An R² of 0.85 means 85% of the variability in y is explained by the linear relationship with x. The remaining 15% is unexplained scatter around the line.
§ 06Standard Error and Testing the Slope
Even with a strong-looking fit, we must ask: could the apparent slope have arisen by chance from a population where β1 = 0? The significance test for the slope answers this directly.
Estimating σ²
The population error variance σ² is estimated by the mean squared error (MSE), using n − 2 degrees of freedom because two parameters (β0, β1) were estimated from the data.
s is called the residual standard error — it measures the typical size of a residual.
Standard Error of b1 and the t-Test
t = b1 / SE(b1) ∼ tn−2
Using the hours-vs-marks example: SSE = 7.50, n = 5, Sxx = 10, b1 = 7.5. Test H0: β1 = 0 at α = 0.05.
Construct a 95% confidence interval for β1 using the values from Example 05.
§ 07Prediction — Point Estimates and Intervals
A regression line can answer two types of prediction questions, which require different intervals even though both start from the same point estimate ŷ* = b0 + b1x*.
Confidence Interval for the Mean
Estimates the mean response at x = x*. Narrower because it targets a population average.
ŷ* ± tα/2 × s√[1/n + (x*−x̄)²/Sxx]
Prediction Interval for a Single Observation
Estimates where a new individual observation at x = x* will fall. Wider because it includes individual variability.
ŷ* ± tα/2 × s√[1 + 1/n + (x*−x̄)²/Sxx]
For x* = 3.5 hours, using ŷ = 33.5 + 7.5x, s = 1.581, n = 5, x̄ = 3, Sxx = 10. Compute the 95% prediction interval.
§ 08The ANOVA Table for Regression
Regression results are commonly presented in an Analysis of Variance (ANOVA) table, which partitions SST into SSR and SSE, computes F-statistics, and gives a significance test for the whole model. For simple linear regression, the F-test is equivalent to the t-test on b1, but the ANOVA framework generalises to multiple regression.
| Source | SS | df | MS | F |
|---|---|---|---|---|
| Regression | SSR | 1 | SSR/1 | MSR/MSE |
| Error (Residual) | SSE | n−2 | SSE/(n−2) | — |
| Total | SST | n−1 | — | — |
Under H0: β1 = 0, the F-statistic follows an F-distribution with 1 and n−2 degrees of freedom. Reject H0 if F > Fcrit.
§ 09Correlation vs Regression — and Common Mistakes
Correlation and regression both measure linear association, but they answer different questions and should not be confused.
Pearson Correlation r
- Symmetric: r(x,y) = r(y,x)
- Dimensionless, always −1 ≤ r ≤ 1
- Measures strength and direction only
- No distinction between x and y
- Does not allow prediction
Regression
- Asymmetric: y on x ≠ x on y
- Slope has units of y per x
- Measures how much y changes per unit x
- x is the predictor, y is the response
- Enables prediction and causal modelling
Using Sxy = 75, Sxx = 10, Syy = SST = 570 from the hours-vs-marks data.
Daily temperature x (°C) and ice cream sales y (units): (20, 80), (22, 90), (25, 110), (28, 130), (30, 140). Find ŷ = b0 + b1x, R², and predict sales at 27°C.
§ 10Summary and Decision Guide
Linear regression is a powerful, interpretable method for modelling the relationship between two quantitative variables. The entire procedure from data collection to inference rests on a small set of key quantities: Sxx, Sxy, Syy, SSE, and MSE.
2. Compute x̄, ȳ, Sxx, Sxy, Syy
3. Estimate b1 = Sxy/Sxx, b0 = ȳ − b1x̄
4. Compute SSE, MSE, s
5. Test H0: β1 = 0 (t-test or F-test)
6. Compute R² = 1 − SSE/SST
7. Check residual plots for assumption violations
8. Predict (within x-range only) with appropriate interval
Continue the Statistics Series
§ 11Quiz — Test Your Understanding
Ten questions covering the full regression workflow. Select the best answer for each.