Occam's razor—the principle that simpler explanations should be preferred over complex ones—has guided scientific thinking for centuries. But in machine learning, we often struggle to formalize what 'simple' means and why simpler should be better.

Bayesian inference provides a principled answer. As we've seen, marginal likelihood automatically penalizes model complexity. But this isn't an ad hoc penalty or a regularization term we manually add—it emerges directly from the axioms of probability theory.

This page explores the deep connection between Bayesian inference and Occam's razor. We'll understand exactly why complex models are penalized, what 'complexity' means in this context, and when the Bayesian Occam's razor succeeds or fails.

Models as Predictive Devices

Every model defines a prior predictive distribution—the set of data patterns it considers possible, weighted by probability. This is the marginal distribution over data:

$$p(\mathcal{D} | \mathcal{M}) = \int p(\mathcal{D} | \boldsymbol{\theta}) p(\boldsymbol{\theta}) d\boldsymbol{\theta}$$

A crucial constraint applies: this distribution must integrate to 1 over all possible datasets. The model has a fixed 'probability budget' to allocate across the space of possible data.

Simple Models: High Commitment, High Reward

A simple model has limited flexibility. It can only generate certain data patterns:

• Its prior predictive distribution is concentrated
• It assigns high probability to specific patterns
• If those patterns actually occur, it scores well (high marginal likelihood)
• If they don't occur, it scores poorly

The intuition: Simple models 'commit' to specific predictions. When right, they're rewarded for their specificity.

Complex Models: Low Commitment, Diluted Reward

A complex model has many parameters and can generate diverse data patterns:

• Its prior predictive distribution is spread across many possibilities
• It assigns modest probability to many patterns, high probability to few
• Regardless of what data appears, it has 'hedged its bets'
• It never scores as well as a simple model that committed to the right answer

The analogy: Imagine betting on a horse race.

• Betting everything on one horse (simple model): Big win if right, big loss if wrong
• Spreading bets across all horses (complex model): Never lose much, but never win much either

If you know which horse will win, you should commit. The 'reward' for correct commitment is what drives Occam's razor.

Visualizing the Prediction Space

Consider the space of all possible datasets as a high-dimensional region:

Simple Model (e.g., straight line):

• Can only generate data that looks linear
• Prior predictive gives high density to linear-looking datasets
• Low density to curved, oscillating, or noisy patterns
• If true data is linear: high evidence
• If true data is curved: low evidence

Complex Model (e.g., high-degree polynomial):

• Can generate linear, curved, oscillating, and complex patterns
• Prior predictive spreads density across all these possibilities
• No single pattern gets very high density
• Regardless of true data: moderate evidence at best

When does complexity pay off?

Only when the complex model can achieve so much higher likelihood that it overcomes the dilution penalty. This requires the true data to exhibit complexity that the simple model genuinely cannot capture.

Let's formalize the intuition using the geometry of parameter spaces.

The Prior Volume Problem

For a model with $d$ parameters, the prior occupies a volume in $d$-dimensional space. If parameters are roughly independent with characteristic scale $\Delta\theta_{\text{prior}}$, the prior 'volume' scales as:

$$V_{\text{prior}} \sim (\Delta\theta_{\text{prior}})^d$$

With more parameters:

• The prior volume grows exponentially with $d$
• The prior probability density at any point decreases exponentially
• Points in parameter space become increasingly 'unlikely' under the prior

Data Constrains Parameters to a Subregion

After seeing data, the posterior concentrates on parameters consistent with the observations. The posterior occupies a much smaller volume:

$$V_{\text{posterior}} \sim (\Delta\theta_{\text{posterior}})^d$$

For well-identified parameters, $\Delta\theta_{\text{posterior}} \ll \Delta\theta_{\text{prior}}$:

• Data reduces uncertainty about each parameter
• The posterior is much more concentrated than the prior

The Occam Factor: Posterior-to-Prior Volume Ratio

The Occam factor is approximately the ratio of posterior to prior volume:

$$\text{Occam Factor} \approx \frac{V_{\text{posterior}}}{V_{\text{prior}}} = \left(\frac{\Delta\theta_{\text{posterior}}}{\Delta\theta_{\text{prior}}}\right)^d$$

Key observations:

1. Always less than 1: Since $\Delta\theta_{\text{posterior}} < \Delta\theta_{\text{prior}}$, the Occam factor is $< 1$.

2. Exponential in dimension: Adding parameters multiplies the penalty by the per-parameter factor.

3. Data informativeness matters: If data strongly constrains parameters (small $\Delta\theta_{\text{posterior}}$), the penalty is larger per parameter.

4. Unused parameters hurt most: Parameters not constrained by data (large $\Delta\theta_{\text{posterior}}$) still incur the prior volume, providing no benefit.

A naive view of model complexity equates it with parameter count. The Bayesian perspective reveals a more sophisticated picture.

Parameter Count Is Misleading

Example 1: Unused parameters A model with 100 parameters where 90 are unconstrained by data is effectively a 10-parameter model. The Occam factor penalizes the 90 unused parameters only mildly (their posterior equals prior, so the ratio is ~1).

Example 2: Equivalent parameterizations Any model can be reparameterized with more or fewer named parameters. The number of parameters is coordinate-dependent, but model evidence is coordinate-independent.

Example 3: Regularization effects A neural network with strong weight decay behaves like a simpler model despite having many nominal parameters. The effective complexity is reduced by regularization.

Effective Degrees of Freedom

The effective number of parameters (or effective degrees of freedom) is a better complexity measure:

$$d_{\text{eff}} = \sum_j \left(1 - \frac{\sigma^2_{\text{posterior},j}}{\sigma^2_{\text{prior},j}}\right)$$

For each parameter:

• If posterior variance $\approx$ prior variance: Data didn't constrain it → contributes $\approx 0$ to $d_{\text{eff}}$
• If posterior variance $\ll$ prior variance: Data strongly constrained it → contributes $\approx 1$ to $d_{\text{eff}}$

The Laplace approximation provides an explicit formula showing how model evidence decomposes into fit and complexity.

The Laplace Formula

Approximating the posterior as Gaussian around its mode $\hat{\boldsymbol{\theta}}$:

$$\log p(\mathcal{D} | \mathcal{M}) \approx \log p(\mathcal{D} | \hat{\boldsymbol{\theta}}) + \log p(\hat{\boldsymbol{\theta}}) + \frac{d}{2}\log(2\pi) - \frac{1}{2}\log|\mathbf{H}|$$

where $\mathbf{H}$ is the Hessian of the negative log-posterior at its mode.

Rearranging:

$$\log p(\mathcal{D} | \mathcal{M}) \approx \underbrace{\log p(\mathcal{D} | \hat{\boldsymbol{\theta}})}{\text{Goodness of fit}} + \underbrace{\log p(\hat{\boldsymbol{\theta}})}{\text{Prior plausibility}} + \underbrace{\frac{d}{2}\log(2\pi) - \frac{1}{2}\log|\mathbf{H}|}_{\text{Occam factor}}$$

Interpreting Each Term

Term 1: Log-likelihood at mode

• Measures how well the best parameters fit the data
• Improves with more parameters (more flexibility)
• This is what maximum likelihood maximizes

Term 2: Log prior at mode

• Measures how plausible the best parameters are a priori
• Penalizes extreme parameter values
• Often negligible for diffuse priors

Term 3: Occam factor

• The $\frac{d}{2}\log(2\pi)$ grows with dimension (favors complexity)
• The $-\frac{1}{2}\log|\mathbf{H}|$ captures posterior concentration
  • Large $|\mathbf{H}|$ = tight posterior = strong Occam penalty
  • Small $|\mathbf{H}|$ = loose posterior = weak Occam penalty

The balance: Complex models gain in Term 1 (better fit) but lose in the Occam factor (more parameters to constrain). Evidence is maximized when these balance optimally.

Let's see the Bayesian Occam's razor in action across different model classes.

Example 1: Polynomial Regression Order Selection

As shown in previous pages, evidence naturally selects the correct polynomial degree. For data from a quadratic:

• Linear model: Poor fit → low evidence
• Quadratic model: Good fit, appropriate complexity → highest evidence
• Higher-degree models: Slightly better fit, but complexity penalty dominates → lower evidence

Example 2: Feature Selection in Regression

With many potential predictors, evidence automatically identifies relevant features:

• Models including irrelevant features are penalized (extra parameters, no fit improvement)
• Models omitting relevant features are penalized (worse fit)
• The 'right' feature set maximizes evidence

Example 3: Gaussian Mixture Model Components

When clustering data with Gaussian mixture models, evidence helps select the number of clusters:

• Too few clusters: Poor fit to multimodal structure
• Too many clusters: Extra means, variances, and mixing weights penalized
• Optimal number: Best trade-off between capturing structure and avoiding redundancy

Example 4: Neural Network Pruning

Bayesian neural networks can use evidence to prune unnecessary weights:

• Weights with posterior ≈ prior (not constrained by data) have high Occam penalty
• Setting these weights to zero and removing them improves evidence
• Automatic Relevance Determination (ARD) implements this via hyperparameter learning

The Bayesian Occam's razor has failure modes. Recognizing these is crucial for appropriate application.

Failure 1: Prior Misalignment

Occam's razor works by rewarding models that 'predicted' the data well from their prior. If priors are poorly calibrated:

• A correctly specified model with a bad prior is penalized
• An incorrectly specified model with a lucky prior might be rewarded

Example: True parameter is $\theta = 50$.

• Model A (correct structure): Prior $\theta \sim \mathcal{N}(0, 1)$
• Model B (wrong structure): Prior happens to concentrate near 50

Model B might have higher evidence despite being structurally wrong, simply because its prior was accidentally better calibrated.

Solution: Use principled priors based on domain knowledge and conduct sensitivity analysis.

The 'Simplicity' Caveat

Occam's razor prefers models that are 'simple' in the sense of making specific predictions—not necessarily simple in terms of interpretability or parameter count.

Example: A model with one parameter but a very diffuse prior is 'complex' in the Bayesian sense because it predicts many data patterns. A model with many parameters but strong priors might be 'simpler' if it predicts specific patterns.

Bayesian 'simplicity' is about predictive commitment, not about parsimony of representation.

How does the Bayesian Occam's razor compare to other methods of controlling complexity?

AIC (Akaike Information Criterion)

$$\text{AIC} = -2\log p(\mathcal{D} | \hat{\boldsymbol{\theta}}) + 2d$$

• Penalizes by raw parameter count (2 per parameter)
• Derived from asymptotic arguments about prediction error
• No Bayesian interpretation (doesn't involve priors)
• Tends to select more complex models than BIC

BIC (Bayesian Information Criterion)

$$\text{BIC} = -2\log p(\mathcal{D} | \hat{\boldsymbol{\theta}}) + d\log n$$

• Penalty grows with both parameters and sample size
• Approximates log marginal likelihood (as we derived)
• More conservative than AIC (prefers simpler models)
• Ignores prior information beyond flat priors

Cross-Validation

• Estimates prediction error on held-out data
• No explicit complexity penalty (complexity is penalized implicitly via overfitting)
• Data-inefficient (uses only fraction of data for training)
• Variance can be high, especially with small samples

Let's consolidate what we've learned about the Bayesian Occam's razor:

Looking ahead: We've now thoroughly understood how to compare models using marginal likelihood, Bayes factors, and Occam's razor. But model comparison isn't the end of the story. Often, no single model is clearly best, and we want to combine the strengths of multiple models. In the next page, we'll explore Bayesian Model Averaging—a principled approach to making predictions by weighting models according to their posterior probabilities.

Model averaging addresses the uncomfortable reality that model uncertainty is real. Rather than committing to a single 'best' model, we can coherently combine all models, weighted by their evidence.