Ridge Regression (L2 Regularization)

We've derived the closed-form solution for Ridge regression and proved its mathematical properties. But to truly master Ridge regression, we must develop deep intuition for what it does to our coefficients and why this leads to better predictions.

The answer lies in understanding shrinkage—the systematic pulling of coefficient estimates toward zero. Shrinkage is the mechanism by which Ridge regression trades bias for reduced variance, achieving better overall prediction accuracy.

To visualize shrinkage, consider the coefficient space in two dimensions (two features $\beta_1$ and $\beta_2$).

The OLS contours:

The OLS objective function defines elliptical level sets (contours of constant RSS) centered at the OLS solution $\hat{\boldsymbol{\beta}}_{\text{OLS}}$. The shape of these ellipses depends on the eigenstructure of $\mathbf{X}^T\mathbf{X}$:

• Circular contours: When features are orthogonal and equally scaled, $\mathbf{X}^T\mathbf{X} \propto \mathbf{I}$, producing circular level sets.
• Elliptical contours: When features are correlated or have different variances, the contours become elliptical, stretched along certain directions.

The L2 constraint:

The L2 constraint $|\boldsymbol{\beta}|_2^2 \leq t$ defines a circle (in 2D) or hypersphere (in higher dimensions) centered at the origin. The constraint radius $\sqrt{t}$ decreases as the regularization parameter $\lambda$ increases.

Finding the Ridge solution:

The Ridge solution lies at the point where the RSS ellipse is tangent to the L2 ball. Key observations:

1. The tangent point is never at an axis (except in degenerate cases): Unlike L1 regularization (Lasso), the smooth L2 ball never "catches" the solution exactly on an axis. Thus, Ridge shrinks all coefficients but never sets any exactly to zero.

2. The solution moves continuously: As $\lambda$ increases, the tangent point traces a smooth path from $\hat{\boldsymbol{\beta}}_{\text{OLS}}$ toward the origin.

3. Shrinkage is proportional to distance from origin: Coefficients farther from zero are pulled more strongly toward zero.

Recall from the eigenvalue analysis that Ridge regression shrinks each principal direction by a specific factor. Let's develop this insight more fully.

Eigendecomposition perspective:

The design matrix $\mathbf{X}^T\mathbf{X}$ has eigendecomposition $\mathbf{V}\mathbf{D}\mathbf{V}^T$ where:

• Columns of $\mathbf{V}$ are the principal component directions
• Diagonal entries of $\mathbf{D}$ are eigenvalues $d_j$

The OLS coefficients projected onto principal direction $j$ are scaled by:

$$\text{Shrinkage factor } s_j = \frac{d_j}{d_j + \lambda}$$

Interpreting shrinkage factors:

The shrinkage factor $s_j$ depends on the ratio of eigenvalue to regularization:

$$s_j = \frac{d_j}{d_j + \lambda} = \frac{1}{1 + \lambda/d_j}$$

Large eigenvalue ($d_j >> \lambda$):

• $s_j \approx 1$: Almost no shrinkage
• This direction has high variance in the data
• Coefficients in this direction are well-determined by the data
• We trust the OLS estimate and don't regularize much

Small eigenvalue ($d_j << \lambda$):

• $s_j \approx d_j/\lambda \approx 0$: Severe shrinkage
• This direction has low variance in the data
• Coefficients in this direction are poorly determined
• OLS would be unstable here; we shrink heavily toward zero

As we vary $\lambda$ from 0 to $\infty$, the Ridge solution traces a continuous path from the OLS solution to the origin. This shrinkage path (or regularization path) reveals how coefficients evolve with regularization strength.

Mathematical description:

$$\hat{\boldsymbol{\beta}}(\lambda) = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$

This is a continuous, monotonically shrinking function of $\lambda$:

• $\hat{\boldsymbol{\beta}}(0) = \hat{\boldsymbol{\beta}}_{\text{OLS}}$ (if OLS exists)
• $|\hat{\boldsymbol{\beta}}(\lambda)|_2$ is strictly decreasing in $\lambda$
• $\lim_{\lambda \to \infty} \hat{\boldsymbol{\beta}}(\lambda) = \mathbf{0}$

Properties of the Ridge shrinkage path:

1. Smooth and continuous: No jumps or discontinuities as $\lambda$ varies. This is a direct consequence of the smooth L2 penalty.

2. Monotonic shrinkage: The L2 norm of coefficients strictly decreases with increasing $\lambda$: $|\hat{\boldsymbol{\beta}}(\lambda_1)|_2 > |\hat{\boldsymbol{\beta}}(\lambda_2)|_2$ for $\lambda_1 < \lambda_2$.

3. Coefficients approach zero at different rates: Coefficients in low-eigenvalue directions shrink faster than those in high-eigenvalue directions.

4. Never exactly zero: Unlike Lasso, Ridge coefficients approach zero asymptotically but (theoretically) never reach it exactly at finite $\lambda$.

5. Sign preservation: Ridge coefficients maintain their signs throughout the path—if $\hat{\beta}_j > 0$ at $\lambda = 0$, it remains positive for all $\lambda > 0$.

A natural question arises: why not just scale the OLS solution by some constant factor? Why is Ridge shrinkage better than simple multiplication?

Uniform scaling:

$$\hat{\boldsymbol{\beta}}{\text{scaled}} = c \cdot \hat{\boldsymbol{\beta}}{\text{OLS}}, \quad c \in (0, 1)$$

This applies the same shrinkage factor to all coefficients.

Ridge shrinkage:

$$\hat{\boldsymbol{\beta}}{\text{Ridge}} = \mathbf{V} \cdot \text{diag}(s_1, \ldots, s_p) \cdot \mathbf{V}^T \cdot \hat{\boldsymbol{\beta}}{\text{OLS}}$$

This applies different shrinkage factors $s_j = d_j/(d_j + \lambda)$ to different principal directions.

Why adaptive shrinkage is superior:

The key insight is that not all directions are equally informative:

• High-variance directions (large $d_j$): The data provides strong signal; OLS estimates are reliable. Shrinking these heavily would lose valuable information.

• Low-variance directions (small $d_j$): The data provides weak signal; OLS estimates are dominated by noise. These should be shrunk aggressively.

Ridge shrinkage adaptively adjusts based on this information content. Uniform scaling would over-shrink confident estimates while under-shrinking noisy ones—the worst of both worlds.

Ridge regression handles correlated (multicollinear) features particularly well. Understanding this behavior is crucial for real-world applications where features are rarely independent.

The multicollinearity problem in OLS:

When features are highly correlated:

• $\mathbf{X}^T\mathbf{X}$ has one or more very small eigenvalues
• Small changes in data cause large swings in coefficients
• Coefficient estimates have huge variance
• Signs may flip randomly; magnitudes explode

Ridge solution for correlated features:

With two perfectly correlated features ($X_2 = X_1$), the OLS solution is non-unique—any combination $\beta_1 + \beta_2 = c$ works. Ridge resolves this by shrinking both coefficients toward zero, preferring the solution that minimizes $\beta_1^2 + \beta_2^2$.

The "grouping effect":

When features are highly correlated, Ridge regression tends to:

1. Shrink both coefficients rather than assigning all the effect to one
2. Produce similar coefficient magnitudes for correlated predictors
3. Average the effect across the correlated group

This is often sensible: if $X_1$ and $X_2$ are measuring the same underlying phenomenon, it's more stable to spread the effect across both.

Mathematical insight:

For perfectly correlated features, the direction of correlation (e.g., $X_1 = X_2$) has a very large eigenvalue, while the perpendicular direction (e.g., $X_1 = -X_2$) has eigenvalue zero. Ridge:

• Preserves coefficients in the high-eigenvalue direction (constructive combination)
• Shrinks to zero in the low-eigenvalue direction (difference between coefficients)

We can understand Ridge shrinkage through the lens of projection geometry—a perspective that unifies several concepts in linear models.

OLS as orthogonal projection:

The OLS fitted values are the orthogonal projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$:

$$\hat{\mathbf{y}}{\text{OLS}} = \mathbf{H}{\text{OLS}} \mathbf{y} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$$

This projection has norm $|\hat{\mathbf{y}}_{\text{OLS}}|_2 \leq |\mathbf{y}|_2$ (projection reduces norm).

Ridge as shrunken projection:

$$\hat{\mathbf{y}}{\text{Ridge}} = \mathbf{H}{\text{Ridge}} \mathbf{y} = \mathbf{X}(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T \mathbf{y}$$

The Ridge smoother matrix $\mathbf{H}_{\text{Ridge}}$ shrinks fitted values toward zero, beyond what OLS projection does:

$$|\hat{\mathbf{y}}_{\text{Ridge}}|2 \leq |\hat{\mathbf{y}}{\text{OLS}}|_2 \leq |\mathbf{y}|_2$$

SVD interpretation:

Using the SVD $\mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T$:

$$\hat{\mathbf{y}}{\text{Ridge}} = \sum{j=1}^{r} \frac{\sigma_j^2}{\sigma_j^2 + \lambda} (\mathbf{u}_j^T\mathbf{y}) \mathbf{u}_j$$

Compare to OLS:

$$\hat{\mathbf{y}}{\text{OLS}} = \sum{j=1}^{r} (\mathbf{u}_j^T\mathbf{y}) \mathbf{u}_j$$

Ridge multiplies each component by $\sigma_j^2/(\sigma_j^2 + \lambda) < 1$, shrinking the projection.

The geometric picture:

1. Project $\mathbf{y}$ onto each left singular vector $\mathbf{u}_j$ to get component $(\mathbf{u}_j^T\mathbf{y})$
2. OLS keeps these components unchanged
3. Ridge shrinks each component by factor $\sigma_j^2/(\sigma_j^2 + \lambda)$
4. Components corresponding to small singular values $\sigma_j$ (low-information directions) are shrunk most

At first glance, shrinkage seems counterproductive: we're deliberately biasing our estimates toward zero. How can introducing bias improve predictions?

The answer lies in the bias-variance tradeoff. Let's make this precise.

Mean Squared Error decomposition:

For any estimator $\hat{\boldsymbol{\beta}}$, its MSE can be decomposed:

$$\text{MSE}(\hat{\boldsymbol{\beta}}) = \text{Bias}(\hat{\boldsymbol{\beta}})^2 + \text{Var}(\hat{\boldsymbol{\beta}})$$

where:

• Bias = $\mathbb{E}[\hat{\boldsymbol{\beta}}] - \boldsymbol{\beta}_{\text{true}}$: systematic error
• Variance = $\mathbb{E}[(\hat{\boldsymbol{\beta}} - \mathbb{E}[\hat{\boldsymbol{\beta}}])^2]$: sensitivity to training data

OLS properties:

• Unbiased: $\text{Bias}(\hat{\boldsymbol{\beta}}_{\text{OLS}}) = 0$
• Potentially high variance: $\text{Var}(\hat{\boldsymbol{\beta}}_{\text{OLS}}) \propto (\mathbf{X}^T\mathbf{X})^{-1}$

When $\mathbf{X}^T\mathbf{X}$ is ill-conditioned, variance can be enormous, dominating MSE.

Ridge properties:

• Biased: $\text{Bias}(\hat{\boldsymbol{\beta}}{\text{Ridge}}) = -\lambda(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\boldsymbol{\beta}{\text{true}} \neq 0$
• Lower variance: $\text{Var}(\hat{\boldsymbol{\beta}}{\text{Ridge}}) < \text{Var}(\hat{\boldsymbol{\beta}}{\text{OLS}})$

The variance reduction can outweigh the bias increase, reducing total MSE.

The variance reduction mechanism:

Recall that Ridge shrinks by factors $s_j = d_j/(d_j + \lambda)$. The variance of the $j$-th principal component of the OLS estimator is proportional to $1/d_j$. After Ridge shrinkage, the variance scales as:

$$\text{Var}{\text{Ridge},j} = s_j^2 \cdot \text{Var}{\text{OLS},j} = \left(\frac{d_j}{d_j + \lambda}\right)^2 \cdot \frac{\sigma^2}{d_j}$$

For small $d_j$, the shrinkage factor $s_j^2$ dramatically reduces what would otherwise be massive variance. This is why Ridge is especially effective when data is ill-conditioned.

We've developed deep intuition for how Ridge regression shrinks coefficients and why this shrinkage improves predictions:

What's next:

With the shrinkage intuition in place, we're ready to formalize the bias-variance tradeoff for Ridge regression. The next page provides mathematical analysis of how Ridge balances these competing quantities and when the tradeoff is most favorable.