Bayesian Interpretation of Regularization

In the preceding pages, we established that regularization emerges from Bayesian priors—Gaussian priors yield Ridge regression, Laplace priors yield Lasso. But we glossed over a crucial detail: we weren't performing full Bayesian inference. We were finding the mode of the posterior, not exploring its entire distribution.

This approach is called Maximum A Posteriori (MAP) estimation. It represents a pragmatic middle ground between pure maximum likelihood (no prior) and full Bayesian inference (entire posterior). MAP estimation incorporates prior information while producing a single point estimate rather than a distribution.

Understanding MAP estimation is essential because it's what we're actually doing when we run Ridge or Lasso regression. This page develops the framework rigorously, explores its properties, and clarifies when MAP is appropriate versus when full Bayesian inference is needed.

Definition:

The Maximum A Posteriori (MAP) estimate is the value of the parameters that maximizes the posterior probability:

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\max{\boldsymbol{\theta}} , p(\boldsymbol{\theta} \mid \mathcal{D})$$

where $\mathcal{D}$ represents the observed data.

Expanding via Bayes' Theorem:

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\max{\boldsymbol{\theta}} , \frac{p(\mathcal{D} \mid \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta})}{p(\mathcal{D})}$$

Since $p(\mathcal{D})$ doesn't depend on $\boldsymbol{\theta}$:

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\max{\boldsymbol{\theta}} , p(\mathcal{D} \mid \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta})$$

$$= \arg\max_{\boldsymbol{\theta}} , \left[\text{Likelihood} \times \text{Prior}\right]$$

Converting to Optimization:

Taking the logarithm (monotonic transformation preserves the argmax):

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\max{\boldsymbol{\theta}} , \left[\log p(\mathcal{D} \mid \boldsymbol{\theta}) + \log p(\boldsymbol{\theta})\right]$$

Equivalently, minimizing the negative log:

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\min{\boldsymbol{\theta}} , \left[-\log p(\mathcal{D} \mid \boldsymbol{\theta}) - \log p(\boldsymbol{\theta})\right]$$

$$= \arg\min_{\boldsymbol{\theta}} , \left[\text{Negative Log-Likelihood} + \text{Negative Log-Prior}\right]$$

Let's make this concrete for linear regression with various priors.

The General Setup:

Likelihood (Gaussian noise): $$\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta} \sim \mathcal{N}(\mathbf{X}\boldsymbol{\beta}, \sigma^2\mathbf{I})$$

Negative log-likelihood (ignoring constants): $$-\log p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) \propto \frac{1}{2\sigma^2}|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2$$

This is proportional to the squared error loss.

The Complete MAP Objective:

For Gaussian likelihood with prior $p(\boldsymbol{\beta})$:

$$\boldsymbol{\hat\beta}{\text{MAP}} = \arg\min{\boldsymbol{\beta}} \left[\frac{1}{2\sigma^2}|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 - \log p(\boldsymbol{\beta})\right]$$

Absorbing constants into $\lambda$:

$$\boldsymbol{\hat\beta}{\text{MAP}} = \arg\min{\boldsymbol{\beta}} \left[|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}|_2^2 + \lambda \cdot R(\boldsymbol{\beta})\right]$$

where $R(\boldsymbol{\beta}) = -\log p(\boldsymbol{\beta})$ (up to constants) is the regularization function.

Maximum Likelihood Estimation (MLE) maximizes only the likelihood, ignoring any prior:

$$\boldsymbol{\hat\theta}{\text{MLE}} = \arg\max{\boldsymbol{\theta}} , p(\mathcal{D} \mid \boldsymbol{\theta})$$

The Relationship:

MAP with a flat (uniform) prior reduces to MLE:

• If $p(\boldsymbol{\theta}) = c$ (constant), then $\log p(\boldsymbol{\theta}) = \log c$ doesn't affect the optimization
• $\arg\max [\log p(\mathcal{D} \mid \boldsymbol{\theta}) + \log c] = \arg\max \log p(\mathcal{D} \mid \boldsymbol{\theta})$

MLE is MAP with a flat prior.

The Asymptotic Convergence:

As sample size $n \to \infty$:

• The likelihood becomes increasingly peaked around the true parameter
• The prior becomes relatively less influential
• MAP and MLE converge to the same value

Formally, if the prior has positive density at the true parameter and the MLE is consistent: $$\boldsymbol{\hat\theta}{\text{MAP}} - \boldsymbol{\hat\theta}{\text{MLE}} \xrightarrow{p} \mathbf{0}$$

The prior's influence is an $O(1/n)$ correction that vanishes asymptotically.

MAP finds the posterior mode; full Bayesian inference characterizes the entire posterior distribution. This distinction has profound implications.

What Each Approach Provides:

The Mode vs. Mean Issue:

For symmetric posteriors (like Gaussian), the mode and mean coincide. But for asymmetric or multimodal posteriors:

• MAP gives the mode (peak of the posterior)
• Posterior mean integrates over entire distribution

For skewed posteriors, these can differ substantially. The mode might be at zero (sparsity) while the mean is non-zero.

Example: Laplace prior with Gaussian likelihood

• Posterior mode (Lasso): can be exactly zero
• Posterior mean: never exactly zero
• Posterior median: somewhere between

Which is "right"? It depends on your goal. For prediction, posterior mean often performs better. For variable selection, the mode (MAP) is more interpretable.

MAP estimation reduces to optimization, leveraging the rich theory and efficient algorithms developed for that purpose.

Convexity:

If both the negative log-likelihood and negative log-prior are convex, the MAP objective is convex, guaranteeing a unique global optimum:

• Gaussian likelihood → convex (quadratic)
• Gaussian prior → convex (quadratic)
• Laplace prior → convex (linear in $|\beta|$)
• Ridge/Lasso objectives are convex

Non-convex priors (some heavy-tailed priors, spike-and-slab) create optimization challenges.

Handling Non-Differentiability:

The L1 penalty $|\boldsymbol{\beta}|_1$ is not differentiable at zero. Several approaches handle this:

1. Subgradient methods: Use any subgradient in place of gradient
2. Coordinate descent: Each coordinate update has closed-form solution
3. Proximal operators: The proximal operator of L1 is soft thresholding: $$\text{prox}_{\lambda|\cdot|}(z) = \text{sign}(z) \cdot \max(|z| - \lambda, 0)$$
4. Smooth approximations: Replace $|\beta|$ with $\sqrt{\beta^2 + \epsilon}$ for small $\epsilon$

Coordinate descent and proximal methods are most common in practice.

From an optimization perspective, MAP can be viewed as penalized maximum likelihood:

$$\boldsymbol{\hat\theta}{\text{MAP}} = \arg\max{\boldsymbol{\theta}} \left[\ell(\boldsymbol{\theta}; \mathcal{D}) - \lambda \cdot P(\boldsymbol{\theta})\right]$$

where $\ell$ is the log-likelihood and $P$ is a penalty function.

The Penalty-Prior Duality:

Non-Convex Penalties:

Some penalties don't correspond to proper priors but have useful properties:

• SCAD (Smoothly Clipped Absolute Deviation): Reduces bias for large coefficients while maintaining sparsity. Non-convex but has oracle properties.

• MCP (Minimax Concave Penalty): Similar to SCAD, interpolates between L1 and no penalty.

• Log penalty: $\sum_j \log(|\theta_j| + \epsilon)$ — approximates L0 (counting non-zeros)

These don't arise from proper priors but can still be viewed as penalized optimization with interpretation as 'adaptive' regularization.

Although MAP gives only a point estimate, we can approximate uncertainty using the curvature of the posterior at the mode.

The Laplace Approximation:

Approximate the posterior by a Gaussian centered at the MAP estimate:

$$p(\boldsymbol{\theta} \mid \mathcal{D}) \approx \mathcal{N}(\boldsymbol{\hat\theta}{\text{MAP}}, \mathbf{\Sigma}{\text{LA}})$$

where the covariance is the inverse Hessian of the negative log-posterior:

$$\mathbf{\Sigma}{\text{LA}} = \left[-\nabla^2 \log p(\boldsymbol{\theta} \mid \mathcal{D}) \Big|{\boldsymbol{\theta}=\boldsymbol{\hat\theta}_{\text{MAP}}}\right]^{-1}$$

$$= \left[\mathbf{H}{\text{NLL}} + \mathbf{H}{\text{prior}}\right]^{-1}$$

For Linear Regression with Gaussian Prior:

The Laplace approximation is exact! The posterior is truly Gaussian:

$$\mathbf{\Sigma}_{\text{LA}} = \sigma^2(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$$

This is exactly the Ridge posterior covariance we derived earlier.

For Lasso (Laplace Prior):

The Laplace approximation is NOT exact. At coefficients that are exactly zero, the Hessian is undefined (non-smooth objective). Practical approaches:

1. Compute Hessian only for non-zero coefficients
2. Use bootstrap to estimate variance
3. Perform full MCMC for proper uncertainty
4. Use approximate methods like variational inference

Bootstrap Uncertainty:

An alternative frequentist approach:

1. Resample the data (rows) with replacement
2. Compute MAP estimate on each bootstrap sample
3. The distribution of estimates approximates sampling uncertainty

This is computationally expensive but doesn't require posterior smoothness.

The regularization parameter $\lambda$ is directly related to prior hyperparameters. How should we choose it?

Approaches to Hyperparameter Selection:

Empirical Bayes (Marginal Likelihood):

For Gaussian models, the marginal likelihood is tractable:

$$p(\mathbf{y} \mid \mathbf{X}, \lambda, \sigma^2) = \int p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}, \sigma^2) p(\boldsymbol{\beta} \mid \lambda) , d\boldsymbol{\beta}$$

For Gaussian prior and likelihood: $$\mathbf{y} \mid \mathbf{X}, \lambda, \sigma^2 \sim \mathcal{N}\left(\mathbf{0}, \sigma^2\mathbf{I} + \frac{\sigma^2}{\lambda}\mathbf{X}\mathbf{X}^T\right)$$

Maximizing this over $\lambda$ (and optionally $\sigma^2$) gives a data-driven regularization strength.

Neither approach is universally superior. The choice depends on your goals, resources, and requirements.

Use MAP When:

We've developed a comprehensive understanding of MAP estimation and its role in regularized machine learning.

What's Next:

In the final page of this module, we'll explore full posterior inference—moving beyond the mode to characterize the complete posterior distribution. We'll see how Bayesian methods provide not just estimates but probability distributions over parameters, enabling genuine uncertainty quantification and coherent decision-making under uncertainty.