In the previous page, we established that prior distributions encode our beliefs about model parameters and that the negative log of a prior becomes a regularization penalty. We claimed that Gaussian priors lead to L2 (Ridge) regularization.

Now we'll prove this claim rigorously. By the end of this page, you'll understand that Ridge regression isn't just a convenient technique for handling multicollinearity or overfitting—it's the exact solution to a Bayesian inference problem where we assume coefficients are drawn from a Gaussian distribution centered at zero.

This perspective transforms Ridge regression from an algorithm to be applied into a statement of belief to be understood. When you use Ridge, you're asserting: 'I believe my coefficients are normally distributed with mean zero and some variance that balances signal against overfitting.'

Let's establish our statistical model precisely. We have:

• Design matrix: $\mathbf{X} \in \mathbb{R}^{n \times p}$ (n observations, p features)
• Response vector: $\mathbf{y} \in \mathbb{R}^n$
• Coefficient vector: $\boldsymbol{\beta} \in \mathbb{R}^p$ (to be estimated)

The Likelihood Model:

We assume the standard linear regression model with Gaussian noise:

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I}_n)$$

Equivalently, the response is conditionally Gaussian given the predictors:

$$\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}, \sigma^2 \sim \mathcal{N}(\mathbf{X}\boldsymbol{\beta}, \sigma^2\mathbf{I}_n)$$

The likelihood function for the parameters given the data is:

$$p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}, \sigma^2) = \frac{1}{(2\pi\sigma^2)^{n/2}} \exp\left(-\frac{1}{2\sigma^2}|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2\right)$$

The Gaussian Prior on Coefficients:

We now place a Gaussian prior on the coefficient vector. The key assumption is that coefficients are independent, each with mean zero and variance $\tau^2$:

$$\boldsymbol{\beta} \sim \mathcal{N}(\mathbf{0}, \tau^2\mathbf{I}_p)$$

Expanded: $$p(\boldsymbol{\beta}) = \frac{1}{(2\pi\tau^2)^{p/2}} \exp\left(-\frac{1}{2\tau^2}|\boldsymbol{\beta}|_2^2\right)$$

This prior encodes several beliefs:

• Centered at zero: We expect coefficients to be small on average
• Symmetric: Positive and negative coefficients with equal probability
• Independent: No prior correlation between different coefficients
• Finite variance: We're confident coefficients won't be arbitrarily large

By Bayes' theorem, the posterior distribution is proportional to the product of likelihood and prior:

$$p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) \propto p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta})$$

Substituting our expressions:

$$p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) \propto \exp\left(-\frac{1}{2\sigma^2}|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2\right) \cdot \exp\left(-\frac{1}{2\tau^2}|\boldsymbol{\beta}|_2^2\right)$$

Combining the Exponents:

Since $\exp(a) \cdot \exp(b) = \exp(a + b)$:

$$p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) \propto \exp\left(-\frac{1}{2}\left[\frac{1}{\sigma^2}|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 + \frac{1}{\tau^2}|\boldsymbol{\beta}|_2^2\right]\right)$$

Let's define $\lambda = \sigma^2 / \tau^2$. Then multiplying by $\sigma^2$:

$$p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) \propto \exp\left(-\frac{1}{2\sigma^2}\left[|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 + \lambda|\boldsymbol{\beta}|_2^2\right]\right)$$

Identifying the Posterior Distribution:

The posterior is proportional to $\exp(-\text{quadratic form in }\boldsymbol{\beta})$. This is the kernel of a multivariate Gaussian distribution. By completing the square (detailed below), we can identify the posterior mean and covariance.

Completing the Square:

We need to rewrite the exponent as $(\boldsymbol{\beta} - \boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\boldsymbol{\beta} - \boldsymbol{\mu})$ plus constants.

Expanding the quadratic forms: $$|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 = \mathbf{y}^T\mathbf{y} - 2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{y} + \boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}$$

$$\lambda|\boldsymbol{\beta}|_2^2 = \lambda\boldsymbol{\beta}^T\boldsymbol{\beta}$$

Combining: $$\mathbf{y}^T\mathbf{y} - 2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{y} + \boldsymbol{\beta}^T(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})\boldsymbol{\beta}$$

Extracting the Posterior Parameters:

From the completed square form, the posterior is:

$$\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}_n, \boldsymbol{\Sigma}_n)$$

where the posterior mean is: $$\boldsymbol{\mu}_n = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$

and the posterior covariance is: $$\boldsymbol{\Sigma}_n = \sigma^2(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$$

Let's present the complete derivation with all steps explicit, suitable for rigorous study.

The derivation reveals the precise meaning of $\lambda$:

$$\lambda = \frac{\sigma^2}{\tau^2} = \frac{\text{Noise variance}}{\text{Prior variance}}$$

This ratio has profound interpretive value:

Limiting Cases:

As $\tau^2 \to \infty$ (flat prior): $$\lambda \to 0, \quad \boldsymbol{\hat\beta}{\text{Ridge}} \to (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} = \boldsymbol{\hat\beta}{\text{OLS}}$$

With an infinitely diffuse prior, we recover ordinary least squares—the prior contributes nothing.

As $\tau^2 \to 0$ (spike prior at zero): $$\lambda \to \infty, \quad \boldsymbol{\hat\beta}_{\text{Ridge}} \to \mathbf{0}$$

With a prior concentrated entirely at zero, the posterior is forced to zero regardless of data—the prior dominates completely.

The Bayesian framework provides more than a point estimate—we get a full distribution over coefficients:

$$\boldsymbol{\Sigma}_n = \sigma^2(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$$

What This Tells Us:

1. Uncertainty about each coefficient: The diagonal elements $[\boldsymbol{\Sigma}n]{jj}$ give the marginal variance of $\beta_j$

2. Correlations between coefficients: Off-diagonal elements $[\boldsymbol{\Sigma}n]{ij}$ indicate how uncertainty in $\beta_i$ relates to uncertainty in $\beta_j$

3. Credible intervals: For coefficient $\beta_j$, a 95% credible interval is $\mu_{n,j} \pm 1.96\sqrt{[\boldsymbol{\Sigma}n]{jj}}$

Effect of Regularization on Uncertainty:

As $\lambda$ increases:

• The posterior covariance shrinks
• Credible intervals narrow
• We become more confident about our estimates

This might seem paradoxical—stronger regularization gives more certainty? The resolution is that we're becoming more certain the coefficients are near zero, not more certain about their true values. The prior is dominating the inference.

In Matrix Terms:

Using the SVD $\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^T$ where $d_1 \geq d_2 \geq \ldots \geq d_p$ are singular values:

$$\boldsymbol{\Sigma}_n = \sigma^2 \mathbf{V} \text{diag}\left(\frac{1}{d_1^2 + \lambda}, \ldots, \frac{1}{d_p^2 + \lambda}\right) \mathbf{V}^T$$

Directions with small singular values (poorly determined by data) get the most uncertainty reduction from the prior.

A fundamental property of Ridge regression is that it shrinks all coefficients toward zero but never sets them exactly to zero. The Bayesian perspective explains why.

The Gaussian Prior Argument:

The Gaussian distribution is continuous with full support on $\mathbb{R}$: $$p(\beta_j = 0) = 0$$

The prior assigns zero probability to any single point, including zero. Since the likelihood (also Gaussian) is continuous with full support, the posterior remains continuous with full support: $$p(\beta_j = 0 \mid \mathbf{y}) = 0$$

The Optimization Perspective:

From an optimization viewpoint, the Ridge objective: $$f(\boldsymbol{\beta}) = |\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 + \lambda|\boldsymbol{\beta}|_2^2$$

has gradient: $$\nabla f = -2\mathbf{X}^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) + 2\lambda\boldsymbol{\beta}$$

Setting to zero: $$\mathbf{X}^T\mathbf{X}\boldsymbol{\beta} + \lambda\boldsymbol{\beta} = \mathbf{X}^T\mathbf{y}$$

For $\beta_j = 0$, we'd need the $j$-th equation: $$\sum_{i \neq j} [\mathbf{X}^T\mathbf{X}]_{ji}\beta_i = [\mathbf{X}^T\mathbf{y}]_j$$

This is a measure-zero event in the space of possible data configurations—it almost never happens naturally.

Implications for Feature Selection:

Since Ridge never produces exact zeros:

• It cannot perform automatic feature selection
• All features remain in the model with shrunken coefficients
• Interpretability can suffer in high dimensions
• When sparsity is desired, Lasso or Elastic Net are preferable

However, this is not always a disadvantage. If you believe all features contribute (even if weakly), Ridge's full-coefficient approach may be more appropriate.

The SVD representation of Ridge provides deep insight into its shrinkage behavior. Let $\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^T$ with singular values $d_1, \ldots, d_p$.

The OLS estimator in the SVD basis: $$\boldsymbol{\hat\beta}_{OLS} = \mathbf{V}\mathbf{D}^{-1}\mathbf{U}^T\mathbf{y}$$

The Ridge estimator: $$\boldsymbol{\hat\beta}_{Ridge} = \mathbf{V}(\mathbf{D}^2 + \lambda\mathbf{I})^{-1}\mathbf{D}\mathbf{U}^T\mathbf{y}$$

We can write this as: $$\boldsymbol{\hat\beta}{Ridge} = \mathbf{V}\mathbf{S}{\lambda}\mathbf{D}^{-1}\mathbf{U}^T\mathbf{y}$$

where $\mathbf{S}_{\lambda}$ is a diagonal matrix with entries: $$s_j = \frac{d_j^2}{d_j^2 + \lambda}$$

The Bayesian Interpretation:

The shrinkage factor $s_j = d_j^2/(d_j^2 + \lambda)$ can be rewritten: $$s_j = \frac{1}{1 + \lambda/d_j^2}$$

This is the precision-weighted average between data and prior. Directions well-estimated by data (large $d_j^2$, high data precision) get weights close to 1. Poorly-estimated directions (small $d_j^2$, low data precision) get weights close to 0.

This is Bayesian learning: we trust data more where it's informative and fall back on the prior where data is uninformative.

One of Ridge regression's most celebrated properties is its ability to handle multicollinearity—highly correlated features. The Bayesian perspective explains why.

The OLS Problem with Multicollinearity:

When features are nearly collinear, $\mathbf{X}^T\mathbf{X}$ is nearly singular:

• Some eigenvalues (and singular values) are near zero
• $(\mathbf{X}^T\mathbf{X})^{-1}$ has huge entries
• OLS estimates have enormous variance
• Small changes in data cause wild swings in estimates

How Ridge Solves This:

Ridge adds $\lambda\mathbf{I}$ to $\mathbf{X}^T\mathbf{X}$ before inverting: $$(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$$

Even if $\mathbf{X}^T\mathbf{X}$ has eigenvalue 0, $\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I}$ has eigenvalue $\lambda$.

The minimum eigenvalue is now at least $\lambda$, bounding the maximum eigenvalue of the inverse at $1/\lambda$.

The Bayesian Interpretation:

Multicollinearity means the data doesn't distinguish between certain linear combinations of coefficients. In Bayesian terms, the likelihood is flat in those directions—many coefficient configurations explain the data equally well.

The Gaussian prior isn't flat. It prefers smaller coefficients. So even where the data provides no information, the prior guides us toward the origin. The "arbitrary" choice among equivalent OLS solutions is resolved by the prior's preference for parsimony.

Quantitative Effect:

For an eigenvalue $\mu$ of $\mathbf{X}^T\mathbf{X}$:

OLS variance contribution: proportional to $1/\mu$ (explodes as $\mu \to 0$)

Ridge variance contribution: proportional to $1/(\mu + \lambda)$ (bounded by $1/\lambda$)

The regularization parameter $\lambda$ sets a floor on how much any direction can be amplified. This is precisely the regularizing effect of the prior: it contributes information that stabilizes inference in data-poor directions.

The Bayesian derivation yields the same computational formula as the optimization approach. Let's examine the practical aspects.

The Closed-Form Solution:

$$\boldsymbol{\hat\beta}_{Ridge} = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$

Computational Approaches:

The SVD Advantage:

Computing the SVD $\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^T$ upfront allows:

• Rapid computation for multiple $\lambda$ values
• Easy extraction of shrinkage factors
• Computation of posterior covariance
• Degrees of freedom calculation: $\text{df}(\lambda) = \sum_j d_j^2/(d_j^2 + \lambda)$

For Full Bayesian Inference:

If we need the full posterior (not just the mean), we need the covariance $\boldsymbol{\Sigma}_n = \sigma^2(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$.

Sampling from this Gaussian is straightforward—much simpler than MCMC approaches needed for non-Gaussian priors.

We've established the deep connection between Ridge regression and Bayesian inference with Gaussian priors.

What's Next:

In Page 3, we'll develop the parallel analysis for Lasso regression, showing how Laplace priors lead to L1 regularization. The mathematics will be more subtle—Laplace priors don't yield closed-form posteriors—but the conceptual connection remains equally profound. We'll see why Laplace priors produce sparsity where Gaussian priors cannot.