MAP estimation gives us the most probable parameter values given our data and prior beliefs. But probability is a richer language than single point estimates. What if the posterior is flat and we're highly uncertain? What if it's bimodal with two plausible explanations? What if we need to propagate parameter uncertainty into predictions?

Full Bayesian posterior inference addresses these questions. Instead of finding just the mode, we characterize the entire posterior distribution $p(\boldsymbol{\theta} \mid \mathcal{D})$. This distribution encapsulates everything we know about parameters after observing data—not just the best guess, but our complete uncertainty landscape.

This page develops the theory and practice of posterior inference, connecting it back to regularization and showing when full Bayesian methods matter most.

Definition and Meaning:

The posterior distribution $p(\boldsymbol{\theta} \mid \mathcal{D})$ is the probability distribution over parameters after observing data $\mathcal{D}$. It combines:

• Prior information $p(\boldsymbol{\theta})$: What we believed before data
• Data evidence via likelihood $p(\mathcal{D} \mid \boldsymbol{\theta})$: What the data tells us

By Bayes' theorem: $$p(\boldsymbol{\theta} \mid \mathcal{D}) = \frac{p(\mathcal{D} \mid \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta})}{p(\mathcal{D})}$$

Interpreting the Posterior:

The posterior is not just a computational object—it has direct probabilistic meaning:

• $p(\theta_j > 0 \mid \mathcal{D})$: Probability that coefficient $j$ is positive
• $p(\theta_j \in [a, b] \mid \mathcal{D})$: Probability coefficient lies in an interval
• $\mathbb{E}[\theta_j \mid \mathcal{D}]$: Expected value of coefficient (posterior mean)
• $\text{Var}(\theta_j \mid \mathcal{D})$: Uncertainty about coefficient
• $\text{Cov}(\theta_i, \theta_j \mid \mathcal{D})$: How uncertainty in coefficients is correlated

These are genuine probability statements about parameters—something frequentist methods cannot provide directly.

For certain combinations of likelihood and prior, the posterior has a known closed-form. These conjugate pairs are computationally precious.

Bayesian Linear Regression (Gaussian Prior):

The most important example for regularization:

Likelihood: $\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}, \sigma^2 \sim \mathcal{N}(\mathbf{X}\boldsymbol{\beta}, \sigma^2\mathbf{I})$

Prior: $\boldsymbol{\beta} \sim \mathcal{N}(\mathbf{0}, \tau^2\mathbf{I})$

Posterior (derived in Page 2): $$\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}_n, \boldsymbol{\Sigma}_n)$$

where: $$\boldsymbol{\mu}_n = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$ $$\boldsymbol{\Sigma}_n = \sigma^2(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}$$

with $\lambda = \sigma^2/\tau^2$.

Other Conjugate Pairs:

Limitations of Exact Inference:

• Requires special likelihood-prior combinations
• Not available for Laplace prior (Lasso)
• Not available for complex/non-linear models
• Extension to non-conjugate settings requires approximation

When exact posterior computation is intractable, we can approximate it by drawing samples. MCMC constructs a Markov chain whose stationary distribution is the posterior.

The Key Idea:

If we can draw samples $\boldsymbol{\theta}^{(1)}, \boldsymbol{\theta}^{(2)}, \ldots, \boldsymbol{\theta}^{(S)}$ from $p(\boldsymbol{\theta} \mid \mathcal{D})$, we can approximate any posterior quantity:

$$\mathbb{E}[g(\boldsymbol{\theta}) \mid \mathcal{D}] \approx \frac{1}{S}\sum_{s=1}^S g(\boldsymbol{\theta}^{(s)})$$

MCMC generates correlated samples that, after enough iterations, behave like independent samples from the posterior.

MCMC for Bayesian Lasso (Laplace Prior):

As noted in Page 3, the Laplace prior can be written as a scale-mixture of Gaussians:

$$\beta_j \mid \tau_j^2 \sim \mathcal{N}(0, \tau_j^2)$$ $$\tau_j^2 \sim \text{Exponential}(\lambda^2/2)$$

This enables Gibbs sampling:

1. Sample $\boldsymbol{\beta} \mid {\tau_j^2}, \mathbf{y}$: Gaussian conditional (like Ridge for that $\tau$ setting)
2. Sample $\tau_j^2 \mid \beta_j$: Inverse-Gaussian conditional

Each step has a known distribution, making sampling efficient despite the non-conjugacy of the original model.

Variational Inference (VI) approximates the posterior with a simpler distribution from a tractable family, making inference an optimization problem rather than a sampling problem.

The Core Idea:

Find the distribution $q(\boldsymbol{\theta})$ in family $\mathcal{Q}$ that is closest to the true posterior:

$$q^*(\boldsymbol{\theta}) = \arg\min_{q \in \mathcal{Q}} \text{KL}(q(\boldsymbol{\theta}) | p(\boldsymbol{\theta} \mid \mathcal{D}))$$

The KL divergence measures how different $q$ is from the posterior.

The Evidence Lower Bound (ELBO):

Direct KL minimization is intractable (requires $p(\mathcal{D})$). Instead, maximize the ELBO:

$$\mathcal{L}(q) = \mathbb{E}_q[\log p(\mathcal{D}, \boldsymbol{\theta})] - \mathbb{E}_q[\log q(\boldsymbol{\theta})]$$

$$= \mathbb{E}_q[\log p(\mathcal{D} \mid \boldsymbol{\theta})] - \text{KL}(q(\boldsymbol{\theta}) | p(\boldsymbol{\theta}))$$

maximizing ELBO is equivalent to minimizing $\text{KL}(q | p(\boldsymbol{\theta} \mid \mathcal{D}))$.

Mean-Field Approximation:

The most common assumption: parameters are independent in the variational family: $$q(\boldsymbol{\theta}) = \prod_j q_j(\theta_j)$$

This factorization makes optimization tractable but ignores posterior correlations.

A key advantage of full Bayesian inference: we can propagate parameter uncertainty into predictions.

Definition:

For a new input $\mathbf{x}_*$, the posterior predictive distribution is:

$$p(y_* \mid \mathbf{x}*, \mathcal{D}) = \int p(y* \mid \mathbf{x}_*, \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta} \mid \mathcal{D}) , d\boldsymbol{\theta}$$

This integrates predictions over all plausible parameter values, weighted by their posterior probability.

Contrast with Point Estimate Prediction:

• MAP/MLE: $\hat{y}* = f(\mathbf{x}*; \boldsymbol{\hat\theta})$ — single prediction using point estimate
• Bayesian: Full distribution $p(y_* \mid \mathbf{x}_*, \mathcal{D})$ — accounts for parameter uncertainty

For Bayesian Linear Regression:

With posterior $\boldsymbol{\beta} \mid \mathcal{D} \sim \mathcal{N}(\boldsymbol{\mu}n, \boldsymbol{\Sigma}n)$ and $y* = \mathbf{x}*^T\boldsymbol{\beta} + \epsilon$:

$$y_* \mid \mathbf{x}*, \mathcal{D} \sim \mathcal{N}(\mathbf{x}^T\boldsymbol{\mu}n, \mathbf{x}^T\boldsymbol{\Sigma}n\mathbf{x}* + \sigma^2)$$

The predictive variance has two components:

1. Epistemic uncertainty: $\mathbf{x}_^T\boldsymbol{\Sigma}n\mathbf{x}$ — uncertainty about parameters
2. Aleatoric uncertainty: $\sigma^2$ — irreducible noise in observations

With more data, epistemic uncertainty shrinks; aleatoric uncertainty remains constant.

Computing Posterior Predictive with MCMC:

If we have posterior samples ${\boldsymbol{\theta}^{(s)}}_{s=1}^S$:

1. For each sample, compute $p(y_* \mid \mathbf{x}_*, \boldsymbol{\theta}^{(s)})$
2. Average: $p(y_* \mid \mathbf{x}*, \mathcal{D}) \approx \frac{1}{S}\sum{s=1}^S p(y_* \mid \mathbf{x}_*, \boldsymbol{\theta}^{(s)})$

Or draw predictive samples:

1. For each $\boldsymbol{\theta}^{(s)}$, sample $y_^{(s)} \sim p(y_ \mid \mathbf{x}_*, \boldsymbol{\theta}^{(s)})$
2. The collection ${y_*^{(s)}}$ forms samples from the posterior predictive

Full posterior inference enables principled model comparison through the marginal likelihood (evidence).

The Marginal Likelihood:

$$p(\mathcal{D} \mid \mathcal{M}) = \int p(\mathcal{D} \mid \boldsymbol{\theta}, \mathcal{M}) \cdot p(\boldsymbol{\theta} \mid \mathcal{M}) , d\boldsymbol{\theta}$$

This is the probability of the data under model $\mathcal{M}$, averaged over all parameter values weighted by the prior.

Bayes Factors:

To compare models $\mathcal{M}_1$ and $\mathcal{M}_2$:

$$\text{BF}_{12} = \frac{p(\mathcal{D} \mid \mathcal{M}_1)}{p(\mathcal{D} \mid \mathcal{M}_2)}$$

Bayes factor > 1 favors $\mathcal{M}_1$; < 1 favors $\mathcal{M}_2$.

Automatic Occam's Razor:

The marginal likelihood automatically penalizes complexity. Complex models spread prior probability over a larger parameter space, reducing the average likelihood:

• Simple model: Prior concentrated; if data matches, high marginal likelihood
• Complex model: Prior diffuse; even if best fit is good, average over prior is lower

This is why Bayesian model comparison favors simpler models that fit well—complexity is penalized without explicit regularization terms.

The posterior distribution is not the end goal—it's input to decisions. Bayesian decision theory provides a coherent framework for acting under uncertainty.

The Framework:

1. Actions $a \in \mathcal{A}$: Possible decisions we can make
2. States $\theta \in \Theta$: Unknown parameter values
3. Loss function $L(\theta, a)$: Cost of action $a$ when true state is $\theta$

Optimal Bayesian Decision:

Choose action minimizing expected posterior loss:

$$a^* = \arg\min_{a \in \mathcal{A}} \mathbb{E}_{\theta \mid \mathcal{D}}[L(\theta, a)]$$

$$= \arg\min_{a \in \mathcal{A}} \int L(\theta, a) \cdot p(\theta \mid \mathcal{D}) , d\theta$$

Example: Variable Selection Decision:

Suppose we must decide whether feature $j$ is relevant (include in model) or not:

• Action $a=1$: Include feature $j$
• Action $a=0$: Exclude feature $j$
• Loss: False inclusion costs $c_1$; false exclusion costs $c_0$

Optimal decision:

• Include if $P(\beta_j \neq 0 \mid \mathcal{D}) > \frac{c_1}{c_0 + c_1}$

With equal costs: include if posterior probability of relevance > 0.5.

This is principled variable selection based on the posterior, not on arbitrary significance thresholds.

Credible Intervals:

A $(1-\alpha)$ credible interval contains the parameter with posterior probability $(1-\alpha)$:

$$P(\theta \in [\theta_L, \theta_U] \mid \mathcal{D}) = 1 - \alpha$$

Types of Credible Intervals:

1. Equal-tailed: $P(\theta < \theta_L \mid \mathcal{D}) = P(\theta > \theta_U \mid \mathcal{D}) = \alpha/2$

2. Highest Posterior Density (HPD): Smallest interval containing $(1-\alpha)$ probability. Every point inside has higher density than any point outside.

For symmetric posteriors, these coincide. For skewed posteriors, HPD is more informative.

Credible vs. Confidence Intervals:

The Bayesian interpretation is often what practitioners actually want.

ROPE (Region of Practical Equivalence):

Often, we care not whether $\theta = 0$ exactly, but whether $\theta$ is negligibly small. Define a ROPE, e.g., $[-0.1, 0.1]$:

• If entire credible interval is in ROPE: Practically null effect
• If entire credible interval is outside ROPE: Practically significant effect
• If credible interval overlaps ROPE boundary: Uncertain

This avoids the problem of rejecting $H_0: \theta = 0$ when the true $\theta = 0.0001$—technically non-zero but practically negligible.

Modern machine learning often involves thousands or millions of parameters. Full posterior inference at this scale is challenging but possible with the right techniques.

Challenges in High Dimensions:

Scalable Approaches:

1. Stochastic Gradient MCMC: Use mini-batches to approximate gradients. Stochastic Gradient Langevin Dynamics (SGLD), Stochastic Gradient HMC.

2. Structured Variational Inference: Use factored or low-rank covariance in variational family. Trade accuracy for tractability.

3. Sparse Posterior Approximations: Approximate posterior with few support points (sparse VI, ensemble methods).

4. Neural Network Posteriors: For deep learning, use dropout as approximate Bayesian inference, or variational BNNs.

5. Laplace Approximation at Scale: Compute Hessian via automatic differentiation; use low-rank or diagonal approximations.

We've completed our exploration of the Bayesian interpretation of regularization. Let's consolidate the key insights from this page and the entire module.

Module Synthesis: The Bayesian Interpretation of Regularization

Across this module, we've established:

1. Priors encode beliefs about parameters before data (Page 1)
2. Gaussian priors yield Ridge (L2) regularization (Page 2)
3. Laplace priors yield Lasso (L1) regularization (Page 3)
4. MAP estimation is regularized optimization (Page 4)
5. Full posteriors enable uncertainty quantification (Page 5)

This perspective transforms regularization from an algorithmic convenience into a principled framework for incorporating prior knowledge. When you choose $\lambda$, you're choosing a prior. When you choose L1 vs L2, you're choosing Laplace vs Gaussian. Every regularization decision is a statement about what you believe.

What's Next in the Curriculum:

With the Bayesian foundation established, you're prepared for advanced topics that build on these ideas:

• Gaussian Processes: Full Bayesian treatment of nonparametric regression
• Bayesian Optimization: Using posteriors for efficient hyperparameter search
• Probabilistic Deep Learning: Uncertainty in neural networks
• Hierarchical Models: Multi-level priors for complex data structures

The Bayesian perspective you've developed here will serve as a unifying thread through all of advanced machine learning.