Multiple Linear Regression
Projection Perspective
Read aloud is unavailable in this browser.
Deepening the Geometric Understanding
We've established that OLS is orthogonal projection onto the column space of $\mathbf{X}$. This final page deepens that perspective, revealing additional insights that emerge when we fully embrace the projection viewpoint.
We'll explore how coefficients can be understood through sequential projections (Gram-Schmidt), why regularization modifies the projection geometry, and how the projection view illuminates the interpretation of coefficients in the presence of correlated predictors.
These insights are not merely theoretical—they directly inform how we interpret regression output and understand model behavior in practice.
By the end of this page, you will understand Gram-Schmidt in the regression context, see how sequential regression reveals partial effects, grasp the geometric interpretation of regularization, and synthesize the complete picture of multiple regression.
Gram-Schmidt Orthogonalization
The Gram-Schmidt process transforms a set of linearly independent vectors into an orthonormal set that spans the same subspace. Applied to the columns of $\mathbf{X}$, it reveals the structure of multiple regression.
The Gram-Schmidt algorithm:
Given vectors $\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_p$ (columns of $\mathbf{X}$), produce orthonormal vectors $\mathbf{q}_0, \mathbf{q}_1, \ldots, \mathbf{q}_p$:
$$\begin{aligned} \mathbf{u}_0 &= \mathbf{x}_0, \quad \mathbf{q}_0 = \mathbf{u}_0 / |\mathbf{u}_0| \mathbf{u}_j &= \mathbf{x}j - \sum{k=0}^{j-1} (\mathbf{q}_k^\top \mathbf{x}_j) \mathbf{q}_k, \quad \mathbf{q}_j = \mathbf{u}_j / |\mathbf{u}_j| \end{aligned}$$
Each $\mathbf{u}_j$ is what remains of $\mathbf{x}_j$ after removing its projections onto all previous orthogonalized vectors.
Gram-Schmidt is algorithmic QR decomposition. Writing X = QR, where Q has orthonormal columns and R is upper triangular, the normal equations become R⊤Rβ = R⊤Q⊤y, which simplifies to Rβ = Q⊤y—a triangular system solved by back-substitution.
Interpretation in regression:
When we orthogonalize the design matrix, the $j$-th orthogonalized column $\mathbf{u}_j$ represents what's "new" in predictor $j$ that isn't explained by predictors $0, 1, \ldots, j-1$.
The regression coefficient $\beta_j$ in OLS measures the effect of this "new" part—the partial effect of $x_j$ after controlling for other predictors.
Sequential Regression and Partial Effects
Gram-Schmidt provides a powerful interpretation: the coefficient of $x_j$ in multiple regression is the simple regression coefficient of $y$ on the residualized $x_j$.
The Frisch-Waugh-Lovell theorem:
Consider partitioning $\mathbf{X} = [\mathbf{X}_1 | \mathbf{X}_2]$ and writing $\mathbf{y} = \mathbf{X}_1\boldsymbol{\beta}_1 + \mathbf{X}_2\boldsymbol{\beta}_2 + \boldsymbol{\epsilon}$.
The coefficient $\boldsymbol{\beta}_2$ can be obtained by:
- Regress $\mathbf{y}$ on $\mathbf{X}_1$, save residuals $\mathbf{M}_1\mathbf{y}$
- Regress $\mathbf{X}_2$ on $\mathbf{X}_1$, save residuals $\mathbf{M}_1\mathbf{X}_2$
- Regress residuals from step 1 on residuals from step 2
where $\mathbf{M}_1 = \mathbf{I} - \mathbf{X}_1(\mathbf{X}_1^\top\mathbf{X}_1)^{-1}\mathbf{X}_1^\top$ is the residual maker.
Interpretation:
$$\boldsymbol{\beta}_2 = (\tilde{\mathbf{X}}_2^\top\tilde{\mathbf{X}}_2)^{-1}\tilde{\mathbf{X}}_2^\top\tilde{\mathbf{y}}$$
where $\tilde{\mathbf{X}}_2 = \mathbf{M}_1\mathbf{X}_2$ and $\tilde{\mathbf{y}} = \mathbf{M}_1\mathbf{y}$.
This says: $\boldsymbol{\beta}_2$ measures the relationship between the parts of $\mathbf{y}$ and $\mathbf{X}_2$ that are not explained by $\mathbf{X}_1$.
This is the formal statement of "controlling for" or "adjusting for" other variables. When you include control variables in a regression, you're measuring the effect of your variables of interest after removing the linear influence of the controls.
In multiple regression with correlated predictors, each coefficient represents the unique contribution of that predictor—what it explains beyond what other predictors already explain. This is why coefficients change when you add or remove predictors.
Why Coefficients Change When Predictors Are Added
A common source of confusion: why does the coefficient of $x_1$ change when you add $x_2$ to the model?
The geometric answer:
When you have only $x_1$, you project $\mathbf{y}$ onto the line spanned by $[\mathbf{1}, \mathbf{x}_1]$. The coefficient $\beta_1$ scales how much of $\mathbf{x}_1$ you need.
When you add $x_2$, you project onto the plane spanned by $[\mathbf{1}, \mathbf{x}_1, \mathbf{x}_2]$. The coefficient of $\mathbf{x}_1$ now scales only the part of $\mathbf{x}_1$ that is orthogonal to $\mathbf{x}_2$.
When do coefficients stay the same?
The coefficient $\beta_1$ remains unchanged when adding $x_2$ if and only if:
- $x_2$ is uncorrelated with $x_1$ (orthogonal in variable space), OR
- $x_2$ is uncorrelated with $y$ after controlling for $x_1$
| Scenario | What Happens to β₁ | Why |
|---|---|---|
| x₂ ⊥ x₁ and x₂ ⊥ y | Unchanged | New variable is noise orthogonal to everything |
| x₂ ⊥ x₁ but x₂ correlated with y | Unchanged | x₂ explains different variance than x₁ |
| x₂ correlated with x₁ but not y | May change (toward zero) | x₂ 'steals' variance x₁ was explaining |
| x₂ correlated with both x₁ and y | Changes (often substantially) | Part of x₁'s effect was really x₂'s effect |
| x₂ is confounder (causes x₁ and y) | Changes (toward causal effect) | Removing confounding reveals true relationship |
If a relevant variable is omitted from the regression, coefficients of included variables absorb its effect. The bias equals the product of (1) the effect of the omitted variable on y, and (2) the regression of the omitted variable on included variables. This is the core of omitted variable bias.
The Geometry of Regularization
Regularization modifies the projection geometry in illuminating ways. Let's see how Ridge regression changes the picture.
Ridge regression:
$$\hat{\boldsymbol{\beta}}_{\text{ridge}} = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}$$
The ridge estimator is no longer a pure projection onto $\mathcal{C}(\mathbf{X})$. Instead, it's a shrinkage operator.
SVD perspective:
Using the SVD $\mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^\top$, we can write:
$$\hat{\boldsymbol{\beta}}{\text{OLS}} = \sum{j=1}^{r} \frac{1}{\sigma_j} (\mathbf{u}_j^\top \mathbf{y}) \mathbf{v}_j$$
$$\hat{\boldsymbol{\beta}}{\text{ridge}} = \sum{j=1}^{r} \frac{\sigma_j}{\sigma_j^2 + \lambda} (\mathbf{u}_j^\top \mathbf{y}) \mathbf{v}_j$$
Interpretation:
- OLS divides by $\sigma_j$: small singular values get amplified
- Ridge multiplies by $\sigma_j/(\sigma_j^2 + \lambda)$: small singular values get shrunk
Geometric effect:
Directions in parameter space corresponding to small singular values (high-variance directions) are shrunk more aggressively. Ridge "pulls" the OLS solution toward the origin along directions the data doesn't constrain well.
The shrinkage factor:
$$d_j = \frac{\sigma_j^2}{\sigma_j^2 + \lambda}$$
- When $\sigma_j \gg \sqrt{\lambda}$: $d_j \approx 1$ (little shrinkage)
- When $\sigma_j \ll \sqrt{\lambda}$: $d_j \approx 0$ (heavy shrinkage)
Ridge regression is OLS in a modified geometry where the column space is 'stretched' by λ in all directions. The solution is no longer the orthogonal projection in the original space, but it is the orthogonal projection in this modified space.
Effective Degrees of Freedom
The projection view provides a natural definition of model complexity through effective degrees of freedom.
For OLS: $$\text{df}_{\text{OLS}} = \text{trace}(\mathbf{H}) = p + 1$$
The degrees of freedom equals the rank of the hat matrix, which equals the number of parameters.
For Ridge: $$\text{df}{\text{ridge}}(\lambda) = \text{trace}(\mathbf{H}\lambda) = \sum_{j=1}^{p} \frac{\sigma_j^2}{\sigma_j^2 + \lambda}$$
where $\mathbf{H}_\lambda = \mathbf{X}(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top$.
- •At λ = 0: df = p + 1 (full OLS complexity)
- •As λ → ∞: df → 0 (model shrinks to zero)
- •Monotonic: df(λ) decreases as λ increases
- •Continuous: Unlike subset selection, regularization provides continuous complexity control
- •Interpretation: df measures how many 'equivalent parameters' the model effectively uses
Effective degrees of freedom allows fair comparison between models of different types. A Ridge regression with df = 5 uses complexity equivalent to OLS with 5 parameters, even though Ridge nominally has p+1 parameters. This is essential for information criteria (AIC, BIC) with regularized models.
Constrained Least Squares
The projection view extends naturally to constrained least squares problems, where we minimize $|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|^2$ subject to linear constraints $\mathbf{A}\boldsymbol{\beta} = \mathbf{c}$.
Geometric interpretation:
The constraint $\mathbf{A}\boldsymbol{\beta} = \mathbf{c}$ defines an affine subspace of the parameter space. We seek the point in this subspace whose image under $\mathbf{X}$ is closest to $\mathbf{y}$.
Solution via projection:
$$\hat{\boldsymbol{\beta}}c = \hat{\boldsymbol{\beta}}{\text{OLS}} - (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{A}^\top[\mathbf{A}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{A}^\top]^{-1}(\mathbf{A}\hat{\boldsymbol{\beta}}_{\text{OLS}} - \mathbf{c})$$
This adjusts the unconstrained solution to satisfy the constraints, moving minimally in the metric defined by $\mathbf{X}^\top\mathbf{X}$.
Applications:
-
Testing linear hypotheses ($\mathbf{c} = \mathbf{0}$): Impose $\mathbf{A}\boldsymbol{\beta} = \mathbf{0}$ and test whether the increase in SSE is significant
-
Monotonicity constraints: Require coefficients to be non-negative or ordered
-
Sum-to-zero constraints: In ANOVA-style models, constraint coefficients to sum to zero for identifiability
-
Known relationships: Incorporate prior knowledge that certain coefficient ratios are fixed
Each independent linear constraint removes one dimension from the parameter space. With k constraints, the effective dimensionality drops from p+1 to p+1-k. This is another form of regularization—incorporating prior knowledge to reduce model complexity.
Weighted Least Squares
When errors have unequal variances (heteroscedasticity), OLS remains unbiased but is no longer BLUE. Weighted Least Squares (WLS) restores efficiency.
The problem: $$\text{Var}(\boldsymbol{\epsilon}) = \sigma^2 \mathbf{W}^{-1}$$
where $\mathbf{W}$ is a known diagonal matrix of weights (larger weights = smaller variance = more reliable observations).
WLS objective: $$\min_{\boldsymbol{\beta}} (\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^\top\mathbf{W}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})$$
Solution: $$\hat{\boldsymbol{\beta}}_{\text{WLS}} = (\mathbf{X}^\top\mathbf{W}\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{W}\mathbf{y}$$
Geometric interpretation:
WLS is orthogonal projection in a modified inner product space. Instead of the standard inner product $\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\top\mathbf{v}$, we use $\langle \mathbf{u}, \mathbf{v} \rangle_\mathbf{W} = \mathbf{u}^\top\mathbf{W}\mathbf{v}$.
The projection onto $\mathcal{C}(\mathbf{X})$ with this weighted inner product gives WLS. Geometrically, we're stretching the space so that low-variance observations contribute more.
Equivalent transformation:
Let $\mathbf{W} = \mathbf{D}^2$ where $\mathbf{D}$ is diagonal with $d_{ii} = \sqrt{w_{ii}}$. Then: $$\hat{\boldsymbol{\beta}}_{\text{WLS}} = ((\mathbf{D}\mathbf{X})^\top(\mathbf{D}\mathbf{X}))^{-1}(\mathbf{D}\mathbf{X})^\top(\mathbf{D}\mathbf{y})$$
WLS on $(\mathbf{X}, \mathbf{y})$ equals OLS on the transformed data $(\mathbf{D}\mathbf{X}, \mathbf{D}\mathbf{y})$.
When errors are both heteroscedastic AND correlated (e.g., time series), the full covariance matrix Σ replaces W⁻¹. The solution is β̂_GLS = (X⊤Σ⁻¹X)⁻¹X⊤Σ⁻¹y. This is orthogonal projection in the inner product defined by Σ⁻¹.
KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.