Geometric Interpretation

The algebraic derivation of OLS—setting derivatives to zero, solving systems of equations—gives us formulas. But it leaves unanswered a deeper question: What does least squares actually do, geometrically?

The answer is beautiful and profound: OLS finds the orthogonal projection of the observed data onto the space of possible predictions. This geometric interpretation transforms linear regression from a collection of formulas into a unified visual concept.

Understanding this geometry isn't just intellectually satisfying—it provides practical intuition for diagnosing regression problems, understanding R-squared, and extending to more complex models.

To understand the geometry of regression, we need to think of our data as vectors in high-dimensional space. This may feel abstract at first, but it's remarkably powerful.

Data as Vectors

Consider our $n$ observations. Each variable—$y$, $x$, even the constant intercept term—can be viewed as a vector in $\mathbb{R}^n$:

$$\mathbf{y} = \begin{pmatrix} y_1 \ y_2 \ \vdots \ y_n \end{pmatrix}, \quad \mathbf{x} = \begin{pmatrix} x_1 \ x_2 \ \vdots \ x_n \end{pmatrix}, \quad \mathbf{1} = \begin{pmatrix} 1 \ 1 \ \vdots \ 1 \end{pmatrix}$$

Each component of the vector corresponds to one observation. The vector $\mathbf{y}$ represents all $n$ observed responses stacked together.

Inner Products and Orthogonality

In $\mathbb{R}^n$, the inner product of two vectors is:

$$\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} = \sum_{i=1}^n u_i v_i$$

Two vectors are orthogonal (perpendicular) if their inner product is zero:

$$\mathbf{u} \perp \mathbf{v} \iff \mathbf{u}^T \mathbf{v} = 0$$

The length (norm) of a vector is:

$$|\mathbf{u}| = \sqrt{\mathbf{u}^T\mathbf{u}} = \sqrt{\sum_{i=1}^n u_i^2}$$

These concepts from linear algebra are the building blocks of the geometric view.

The design matrix $\mathbf{X}$ for simple linear regression has two columns:

$$\mathbf{X} = \begin{pmatrix} 1 & x_1 \ 1 & x_2 \ \vdots & \vdots \ 1 & x_n \end{pmatrix} = [\mathbf{1} | \mathbf{x}]$$

What Can the Model Predict?

Any prediction from our model takes the form:

$$\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta} = \beta_0 \mathbf{1} + \beta_1 \mathbf{x}$$

As we vary $\beta_0$ and $\beta_1$ over all possible real values, what predictions can we generate?

The Column Space

The set of all possible predictions is the column space (or range) of $\mathbf{X}$:

$$\mathcal{C}(\mathbf{X}) = {\mathbf{X}\boldsymbol{\beta} : \boldsymbol{\beta} \in \mathbb{R}^2} = {\beta_0 \mathbf{1} + \beta_1 \mathbf{x} : \beta_0, \beta_1 \in \mathbb{R}}$$

This is a 2-dimensional subspace of $\mathbb{R}^n$ (assuming $\mathbf{1}$ and $\mathbf{x}$ are linearly independent). It's the plane spanned by the vectors $\mathbf{1}$ and $\mathbf{x}$.

Key Insight: The model can only produce predictions that lie in this plane. No matter what values of $\beta_0$ and $\beta_1$ we choose, $\hat{\mathbf{y}}$ is always in $\mathcal{C}(\mathbf{X})$. If the observed $\mathbf{y}$ doesn't lie in this plane, we can never predict it exactly.

Visualization Intuition

Imagine $n = 3$ for visualization purposes (we can draw 3D). In this simplified case:

• $\mathbf{y}$ is a point in $\mathbb{R}^3$
• $\mathbf{1}$ and $\mathbf{x}$ are two vectors in $\mathbb{R}^3$
• Their span is a 2D plane through the origin
• Any prediction $\hat{\mathbf{y}} = \beta_0 \mathbf{1} + \beta_1 \mathbf{x}$ lies on this plane

The question becomes: which point on the plane is closest to $\mathbf{y}$?

Here's the central geometric insight:

$$\text{Minimizing } |\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|^2 \text{ is equivalent to finding the closest point in } \mathcal{C}(\mathbf{X}) \text{ to } \mathbf{y}$$

Why? Because:

$$|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \text{RSS}$$

Minimizing RSS is minimizing the squared Euclidean distance in $\mathbb{R}^n$.

The Closest Point is the Orthogonal Projection

A fundamental theorem in linear algebra states: the closest point in a subspace to an external point is the orthogonal projection onto that subspace.

This means:

$$\hat{\mathbf{y}} = \text{proj}_{\mathcal{C}(\mathbf{X})} \mathbf{y}$$

The fitted values $\hat{\mathbf{y}}$ are the orthogonal projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$!

The Residual is Perpendicular

The residual vector is:

$$\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}}$$

Since $\hat{\mathbf{y}}$ is the orthogonal projection, $\mathbf{e}$ must be perpendicular to the column space:

$$\mathbf{e} \perp \mathcal{C}(\mathbf{X})$$

Specifically, $\mathbf{e}$ is orthogonal to every column of $\mathbf{X}$:

$$\mathbf{1}^T \mathbf{e} = 0 \implies \sum_i e_i = 0$$ $$\mathbf{x}^T \mathbf{e} = 0 \implies \sum_i x_i e_i = 0$$

These are exactly the normal equations we derived algebraically! The geometric and algebraic perspectives connect perfectly.

The projection operation can be expressed as matrix multiplication. From the OLS solution:

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

The fitted values are:

$$\hat{\mathbf{y}} = \mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

Define the hat matrix (or projection matrix):

$$\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$$

Then simply:

$$\hat{\mathbf{y}} = \mathbf{H}\mathbf{y}$$

The matrix $\mathbf{H}$ "puts a hat on $\mathbf{y}$"—hence the name.

The Residual-Maker Matrix

The residuals are:

$$\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}} = \mathbf{y} - \mathbf{H}\mathbf{y} = (\mathbf{I} - \mathbf{H})\mathbf{y}$$

Define $\mathbf{M} = \mathbf{I} - \mathbf{H}$ (the residual-maker matrix):

$$\mathbf{e} = \mathbf{M}\mathbf{y}$$

$\mathbf{M}$ is also symmetric and idempotent. It projects onto the orthogonal complement of $\mathcal{C}(\mathbf{X})$—the space of all vectors perpendicular to predictions.

Since $\hat{\mathbf{y}}$ and $\mathbf{e}$ are orthogonal, the Pythagorean theorem applies:

$$|\mathbf{y}|^2 = |\hat{\mathbf{y}}|^2 + |\mathbf{e}|^2$$

But wait—this isn't quite right because we need to center around means. Let's be more careful.

Centered Vectors

Define centered vectors:

$$\tilde{\mathbf{y}} = \mathbf{y} - \bar{y}\mathbf{1}$$ $$\tilde{\hat{\mathbf{y}}} = \hat{\mathbf{y}} - \bar{y}\mathbf{1}$$

(Note: $\bar{\hat{y}} = \bar{y}$ from the normal equations, so we can use $\bar{y}$ for both.)

Variance Decomposition

Now we have the orthogonal decomposition:

$$\tilde{\mathbf{y}} = \tilde{\hat{\mathbf{y}}} + \mathbf{e}$$

Applying Pythagorean theorem:

$$|\tilde{\mathbf{y}}|^2 = |\tilde{\hat{\mathbf{y}}}|^2 + |\mathbf{e}|^2$$

Translating back to sums of squares:

$$\underbrace{\sum_i (y_i - \bar{y})^2}_{\text{SST}} = \underbrace{\sum_i (\hat{y}i - \bar{y})^2}{\text{SSR}} + \underbrace{\sum_i (y_i - \hat{y}i)^2}{\text{SSE}}$$

Or symbolically:

$$\text{SST} = \text{SSR} + \text{SSE}$$

Total Sum of Squares = Regression Sum of Squares + Error Sum of Squares

R-Squared: The Coefficient of Determination

The proportion of total variability explained by the model:

$$R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}}$$

Geometric Interpretation: $R^2$ is the squared cosine of the angle between $\tilde{\mathbf{y}}$ and $\tilde{\hat{\mathbf{y}}}$:

$$R^2 = \cos^2(\theta)$$

where $\theta$ is the angle between the centered observation vector and the centered prediction vector.

• $R^2 = 1$: Perfect fit. The angle is 0°; $\mathbf{y}$ lies in the column space.
• $R^2 = 0$: No fit. The angle is 90°; predictions are all equal to $\bar{y}$.
• $0 < R^2 < 1$: Partial fit. The smaller the angle, the better.

The hat matrix reveals important information about individual observations. Each fitted value is a weighted combination of all observed $y$ values:

$$\hat{y}i = \sum{j=1}^n h_{ij} y_j$$

where $h_{ij}$ is the $(i,j)$ element of $\mathbf{H}$.

Leverage Values

The diagonal elements $h_{ii}$ are called leverage values or hat values:

$$h_{ii} = [\mathbf{H}]_{ii}$$

For simple linear regression, leverage has a closed form:

$$h_{ii} = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_j (x_j - \bar{x})^2} = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{S_{xx}}$$

Residual Variance and Leverage

The variance of residuals is not constant—it depends on leverage:

$$\text{Var}(e_i) = \sigma^2(1 - h_{ii})$$

High-leverage points have smaller residual variance. This makes sense: if $h_{ii}$ is close to 1, the fitted line is forced to pass near $(x_i, y_i)$, leaving little residual.

This is why we use studentized residuals for diagnostics:

$$t_i = \frac{e_i}{s\sqrt{1 - h_{ii}}}$$

where $s$ is the estimated standard error. Studentization adjusts for the unequal variance.

Let's visualize the geometric concepts with code. We'll compute the hat matrix, verify its properties, and examine the projection.