Perpendicular vectors, orthonormal bases, projection onto subspaces, and the Gram–Schmidt process: the geometry that makes numerical linear algebra work.
5Core Concepts
12Worked Examples
10Quiz Questions
Share this page
§ 01Orthogonal Vectors — Perpendicularity via the Dot Product
Two vectors are orthogonal when they meet at a right angle. In any number of dimensions, that geometric condition reduces to a single algebraic check: their dot product is zero.
Recall that the dot product of u = (u₁, u₂, …, uₙ) and v = (v₁, v₂, …, vₙ) is:
Definition — Orthogonality
u · v = u₁v₁ + u₂v₂ + … + uₙvₙ
u ⊥ v ⟺ u · v = 0
The zero vector is orthogonal to every vector by convention.
The geometric reason: the dot product equals |u||v|cos θ, where θ is the angle between the vectors. Setting this to zero forces cos θ = 0, so θ = 90°.
Properties of Orthogonality
Symmetry: if u ⊥ v then v ⊥ u.
Not transitive:u ⊥ v and v ⊥ w does not imply u ⊥ w.
Pythagoras in ℝⁿ: if u ⊥ v, then |u + v|² = |u|² + |v|².
Linearity: if u ⊥ v and u ⊥ w, then u ⊥ (cv + dw) for any scalars c, d.
Example 1Check whether u = (2, −3, 1) and v = (1, 0, −2) are orthogonal
Dot product
u · v = (2)(1) + (−3)(0) + (1)(−2) = 2 + 0 − 2 = 0
Conclusion
u · v = 0, so u ⊥ v. ✓
Example 2Find all vectors in ℝ² orthogonal to v = (3, −1)
Setup
Let u = (x, y). Require u · v = 0:
3x − y = 0 ⟹ y = 3x
General solution
u = x(1, 3), x ∈ ℝ
All scalar multiples of (1, 3) are orthogonal to (3, −1).
Answer
The orthogonal complement is span{(1, 3)}.
§ 02Orthonormal Sets and Bases
An orthogonal set becomes orthonormal when every vector is also a unit vector. Orthonormal bases are the "dream basis", coordinates are trivial to compute and many matrix operations become beautiful.
Definitions
Orthogonal set: {v₁, …, vₖ} with vᵢ · vⱼ = 0 for i ≠ jOrthonormal set: also |vᵢ| = 1 for all i, i.e. vᵢ · vⱼ = δᵢⱼ
Here δᵢⱼ is the Kronecker delta: 1 if i = j, 0 otherwise.
To convert an orthogonal set to orthonormal, simply divide each vector by its norm: û = v/|v|. This is called normalisation.
Why Orthonormal Bases Are Special
If {u₁, …, uₙ} is an orthonormal basis for ℝⁿ, then for any vector x:
Expansion in an orthonormal basis
x = (x · u₁)u₁ + (x · u₂)u₂ + … + (x · uₙ)uₙ
Each coordinate (x · uᵢ) is computed by a single dot product, no linear system to solve.
Orthogonal Matrix Q
A square matrix Q whose columns form an orthonormal set satisfies:
QᵀQ = I (columns orthonormal)
QQᵀ = I (rows also orthonormal)
Q⁻¹ = Qᵀ (inversion is free!)
det(Q) = ±1
Examples of Orthogonal Matrices
Rotation matrix in ℝ²:
Q = [[cos θ, −sin θ], [sin θ, cos θ]]
Reflection matrices
Permutation matrices
Householder reflectors
Example 3Verify {(1/√2, 1/√2, 0), (−1/√2, 1/√2, 0), (0, 0, 1)} is orthonormal
All pairwise dot products are 0 and all norms are 1 — this is an orthonormal set. ✓
§ 03Orthogonal Complements and the Four Fundamental Subspaces
The orthogonal complement of a subspace W consists of every vector perpendicular to all of W. It is itself a subspace: and together, W and W⊥ tile the entire space.
Orthogonal Complement
W⊥ = {x ∈ ℝⁿ : x · w = 0 for all w ∈ W}
Key fact: dim(W) + dim(W⊥) = n, and W ∩ W⊥ = {0}.
Crucial recovery: (W⊥)⊥ = W.
For an m×n matrix A, the four fundamental subspaces pair up orthogonally:
Column Space & Left Null SpaceCol(A) ⊥ Null(Aᵀ)
Both live in ℝᵐ. Their dimensions sum to m.
Row Space & Null SpaceRow(A) ⊥ Null(A)
Both live in ℝⁿ. Their dimensions sum to n.
Practical consequence: when does Ax = b have a solution?
The system Ax = b is consistent if and only if b ⊥ Null(Aᵀ), i.e. b lies in Col(A). When Ax = b has no exact solution, the best approximate solution — the least-squares solution — comes from projecting b onto Col(A).
Example 4Find W⊥ where W = span{(1, 2, −1)}
Set up
We need all (x, y, z) with (x, y, z) · (1, 2, −1) = 0:
x + 2y − z = 0 ⟹ x = −2y + z
Parametrise
Free variables: y = s, z = t:
(x, y, z) = s(−2, 1, 0) + t(1, 0, 1)
Answer
W⊥ = span{(−2, 1, 0), (1, 0, 1)}, a 2-dimensional subspace of ℝ³.
§ 04Projection onto a Subspace
The projection of b onto W is the unique point in W closest to b. It decomposes ℝⁿ into a component inside W and a component perpendicular to W: the foundation of least-squares.
Projection onto a Single Vector
Projection of b onto vector a
proja b = (b · a / a · a) a = (a · b / |a|²) a
The scalar coefficient b·a / a·a is the "shadow length" of b along â = a/|a|.
Projection Matrix onto span{a}
Projection matrix — one vector
P = aaᵀ / (aᵀa) (outer product divided by inner product)
P² = P (projecting twice does nothing extra): P is idempotent.
Pᵀ = P, projection matrices are symmetric.
Projection onto a Higher-Dimensional Subspace
When W = Col(A) for an m×n matrix A with linearly independent columns, the projection of b onto W is:
Projection onto Col(A)
p̂ = A(AᵀA)⁻¹Aᵀ b
P = A(AᵀA)⁻¹Aᵀ (projection matrix onto Col(A))
If the columns of A are already orthonormal, this simplifies to P = AAᵀ — no matrix inversion needed.
The error vector e = b − p̂ is perpendicular to W: Aᵀe = 0. This is the normal equation, and it is the key to least-squares.
Example 5Project b = (1, 2, 3) onto the line through a = (1, 1, 1)
Compute
b · a = 1 + 2 + 3 = 6, a · a = 3proja b = (6/3)(1,1,1) = 2(1,1,1) = (2, 2, 2)
Error vector
e = b − p̂ = (1,2,3) − (2,2,2) = (−1, 0, 1)
Check: e · a = −1 + 0 + 1 = 0 ✓ (error is perpendicular to a)
Answer
p̂ = (2, 2, 2), e = (−1, 0, 1)
Example 6Find the projection matrix onto W = span{(1,0,1), (0,1,1)} and project b = (1, 2, 3)
Form A
Columns are the basis vectors: A = [[1,0],[0,1],[1,1]].
b itself lies in W! So the projection is b exactly and the error is 0.
Answer
p̂ = (1, 2, 3), e = 0 — b ∈ W.
Example 7Least-squares: fit a line y = c + dt to the data (t,y): (0,1),(1,2),(2,0)
Set up Ax = b
A = [[1,0],[1,1],[1,2]], b = [1, 2, 0]ᵀ, x = [c, d]ᵀ
Normal equations
AᵀA = [[3,3],[3,5]], Aᵀb = [3, 2]
Solve: 3c + 3d = 3, 3c + 5d = 2 ⟹ 2d = −1, d = −1/2, c = 3/2.
Best-fit line
y = 3/2 − (1/2)t
§ 05The Gram–Schmidt Process
Given any basis {x₁, x₂, …, xₖ} for a subspace, Gram–Schmidt constructs an orthonormal basis {q₁, q₂, …, qₖ} that spans the same space, one vector at a time.
The idea is iterative: at each step, take the next basis vector and subtract off its projections onto all the orthonormal vectors already constructed. What remains is perpendicular to everything built so far. Then normalise.
Why the subtracted terms are projections
Each term (xₖ · qᵢ)qᵢ is exactly the projection of xₖ onto qᵢ. Subtracting it removes the component of xₖ in the direction of qᵢ, leaving a residual perpendicular to qᵢ. Doing this for all previous q-vectors simultaneously ensures vₖ is perpendicular to the entire span of {q₁, …, qₖ₋₁}.
Example 8Apply Gram–Schmidt to x₁ = (1, 1, 0), x₂ = (1, 0, 1), x₃ = (0, 1, 1)
Gram–Schmidt is not just a theoretical tool, it directly produces the QR decomposition of a matrix, one of the most important factorisations in numerical linear algebra.
If A is an m×n matrix with linearly independent columns, Gram–Schmidt on the columns of A gives:
QR Decomposition
A = QR
Q is m×n with orthonormal columns (Qᵀ Q = I). R is n×n upper triangular with positive diagonal entries. R encodes the "how much" of each new vector was subtracted in Gram–Schmidt.
The entries of R are exactly the coefficients produced during Gram–Schmidt:
Entries of R
Rᵢⱼ = qᵢ · xⱼ for i ≤ j (upper triangular), Rᵢᵢ = |vᵢ| > 0
The normal equation Aᵀb = AᵀAx̂ simplifies beautifully with QR: Rx̂ = Qᵀb, a triangular system solved by back substitution.
Example 9Find the QR decomposition of A = [[1,1],[1,0],[0,1]]
P² = P confirms P is a projection matrix (idempotent). ✓
§ 08Common Mistakes
Mistake 1 — Confusing orthogonal and orthonormal
An orthogonal set has all pairwise dot products equal to zero; an orthonormal set additionally requires every vector to have unit length. Gram–Schmidt produces an orthonormal basis only after the normalisation step (dividing by |vₖ|). Forgetting to normalise gives you an orthogonal basis, which is useful but not orthonormal.
Mistake 2 — Thinking "orthogonal matrix" means "orthogonal columns"
The standard definition of an orthogonal matrix Q requires the columns to be orthonormal (unit length AND pairwise perpendicular), not merely orthogonal. This gives Qᵀ Q = I. A matrix with just orthogonal (but not unit) columns does not satisfy this identity.
Mistake 3 — Wrong projection formula for higher dimensions
For a single vector a, the projection is (a · b / a · a) a. For a subspace Col(A), the formula is A(AᵀA)⁻¹Aᵀb. Confusing the two — especially using the single-vector scalar formula when projecting onto a 2D subspace — gives incorrect results.
Mistake 4 — Subtracting only the first projection in Gram–Schmidt
At step k, you must subtract the projections onto all k−1 previous orthonormal vectors, not just q₁. Missing even one projection will leave vₖ with a component that is not perpendicular to all previous vectors, and the resulting set will not be orthogonal.
Mistake 5 — Assuming orthogonality implies a complete basis
A set of orthogonal vectors is linearly independent, but it may not span the full space. An orthogonal basis for a subspace W has exactly dim(W) vectors. Adding fewer vectors than dim(W) gives an orthogonal set for a proper subspace, not the full W.
Pro tip — The normal equation and projections
The least-squares solution x̂ satisfies Aᵀ(b − Ax̂) = 0. This is just saying the error b − Ax̂ is perpendicular to the column space of A. Every least-squares problem is a projection problem — understanding one is understanding the other.
Ten questions covering orthogonal vectors, orthonormal sets, complements, projections, Gram–Schmidt, and QR decomposition. Think carefully before selecting your answer.
Orthogonality Quiz
Question 1 of 10
Which condition makes two vectors u and v orthogonal?
Question 2 of 10
An orthonormal set {u₁, …, uₖ} satisfies uᵢ · uⱼ = ?
Question 3 of 10
If Q is an orthogonal (orthonormal-columned) matrix, what is Q⁻¹?
Question 4 of 10
What is the projection of b = (4, 0) onto the vector a = (1, 1)?
Question 5 of 10
If W is a subspace of ℝⁿ with dim(W) = k, what is dim(W⊥)?
Question 6 of 10
Which of the following is a key property of a projection matrix P?
Question 7 of 10
In the Gram–Schmidt process applied to {x₁, x₂}, the vector v₂ is computed as:
Question 8 of 10
The projection matrix onto Col(A) (with A having independent columns) is:
Question 9 of 10
In QR decomposition A = QR, which of the following correctly describes R?
Question 10 of 10
The null space Null(A) and the row space Row(A) of a matrix A are related by: