Geometric Interpretation of Least Squares (OLS): Projection onto Column Space

Abstract

This work presents a geometric interpretation of ordinary least squares (OLS) as a projection of a data vector onto the column space of a design matrix.

The vector y is decomposed into two orthogonal components: y∥ ∈ col(X) and y⊥ ⟂ col(X).

The fitted values are given by ŷ = Xβ, while the residual y − ŷ lies in the orthogonal complement of the column space.

A concrete visualization in ℝ³ (three observations) is used to illustrate the geometry of least squares, where the column space forms a plane and the solution corresponds to the orthogonal projection of y onto that plane.

This representation emphasizes the structural relationship between data, model space, and residual, rather than focusing on computational procedures.

Notes

The material is part of the GraphMath project, which focuses on geometric and structural understanding of linear algebra.

Context and reuse

These visual materials are used in the Wikipedia article on ordinary least squares (projection interpretation), where they illustrate least squares as an orthogonal projection onto the column space.

A corresponding diagram is available on Wikimedia Commons for reuse in educational contexts.