We've established that the marginal likelihood $p(\mathcal{D}|\mathcal{M})$—also called model evidence—forms the basis of Bayesian model comparison. But what exactly does this quantity measure? Why should the integral of likelihood over the prior distribution be the 'right' way to evaluate a model?

This page explores the deep conceptual foundations of model evidence. We'll see that it naturally encapsulates the bias-variance trade-off, embodies predictive accuracy, and implements an automatic Occam's razor that prefers simpler models when complexity isn't justified by the data.

Understanding model evidence at this level transforms it from a mathematical convenience into a principled philosophy of model quality.

The Prior Predictive Distribution

Before observing any data, a Bayesian model makes probabilistic predictions. If we sample parameters from the prior and then sample data from the likelihood, we get the prior predictive distribution:

$$p(\mathcal{D} | \mathcal{M}) = \int p(\mathcal{D} | \boldsymbol{\theta}, \mathcal{M}) \cdot p(\boldsymbol{\theta} | \mathcal{M}) , d\boldsymbol{\theta}$$

This is precisely the marginal likelihood! The model evidence is the probability that the model assigns to the observed data before looking at the data to learn parameters.

Interpretation: Model evidence asks: 'If I hadn't seen the data yet and sampled parameters randomly from my prior, how likely would I be to generate exactly the data I observed?'

A model that assigns high probability to the observed data before learning is a model that 'predicted' the data well. A model that assigns low probability made poor predictions—even if it can fit the data well after parameter tuning.

Sequential Updating Perspective

Model evidence has another interpretation via sequential prediction. Consider observing data points $\mathcal{D} = {y_1, y_2, \ldots, y_n}$ one at a time. By the chain rule:

$$p(\mathcal{D} | \mathcal{M}) = p(y_1 | \mathcal{M}) \cdot p(y_2 | y_1, \mathcal{M}) \cdot p(y_3 | y_1, y_2, \mathcal{M}) \cdots p(y_n | y_1, \ldots, y_{n-1}, \mathcal{M})$$

Each term is a one-step-ahead predictive probability:

$$p(y_t | y_1, \ldots, y_{t-1}, \mathcal{M}) = \int p(y_t | \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta} | y_1, \ldots, y_{t-1}, \mathcal{M}) , d\boldsymbol{\theta}$$

This is the posterior predictive given all previous data.

Interpretation: Model evidence is the product of sequential predictions, where each prediction is made using only the data seen so far. A model with high evidence consistently made accurate predictions throughout the data stream.

The prequential (predictive-sequential) view: This shows that model evidence is essentially a measure of out-of-sample prediction, but evaluated on the actual sample using the Bayesian updating mechanism. It's like leave-one-out cross-validation, but with a specific update scheme.

We can formally decompose model evidence to see how it balances data fit against model complexity.

The Evidence Lower Bound (ELBO) Decomposition

For any distribution $q(\boldsymbol{\theta})$ over parameters, we have:

$$\log p(\mathcal{D} | \mathcal{M}) = \underbrace{\mathbb{E}q[\log p(\mathcal{D} | \boldsymbol{\theta})]}{\text{Expected Log-Likelihood}} - \underbrace{\text{KL}(q | p_{\text{prior}})}{\text{Complexity Penalty}} + \underbrace{\text{KL}(q | p{\text{posterior}})}_{\geq 0}$$

When $q = p_{\text{posterior}}$, the last term vanishes, giving:

$$\log p(\mathcal{D} | \mathcal{M}) = \mathbb{E}{\text{posterior}}[\log p(\mathcal{D} | \boldsymbol{\theta})] - \text{KL}(p{\text{posterior}} | p_{\text{prior}})$$

Intuition: The Fit-Complexity Trade-off

This decomposition makes the trade-off explicit:

Simple model with moderate fit:

• Expected log-likelihood: Moderate (can't capture all patterns)
• KL divergence: Small (posterior close to prior—didn't need to learn much)
• Evidence: Can be high if the balance is right

Complex model that overfits:

• Expected log-likelihood: High (fits all patterns, including noise)
• KL divergence: Large (posterior very different from prior—learned a lot)
• Evidence: Often low because the complexity penalty dominates

Appropriately complex model:

• Expected log-likelihood: High (captures real patterns)
• KL divergence: Moderate (learned what was necessary, no more)
• Evidence: Highest among competitors

The key insight: Model evidence doesn't just reward good fit—it rewards efficient fit. A model that achieves good fit while staying close to its prior beliefs is preferred.

A beautiful perspective on model evidence comes from information theory: maximizing evidence is equivalent to finding the model that achieves the best data compression.

The Minimum Description Length Perspective

Minimum Description Length (MDL) is a principle from information theory: the best model is the one that provides the shortest description of the data. Description length is measured in bits (or nats, using natural logarithm).

The total description length for transmitting data using a model has two parts:

$$L_{\text{total}} = L_{\text{model}} + L_{\text{data|model}}$$

• $L_{\text{model}}$: Bits to describe the model (including learned parameters)
• $L_{\text{data|model}}$: Bits to describe data given the model

Simple models are cheap to describe but may fit data poorly (high $L_{\text{data|model}}$). Complex models describe data well but are expensive to transmit (high $L_{\text{model}}$).

The connection to evidence: Under certain coding schemes, the optimal description length is:

$$L_{\text{total}} = -\log p(\mathcal{D} | \mathcal{M})$$

Maximizing marginal likelihood = Minimizing description length

Prequential Codes and Evidence

The connection becomes explicit with prequential coding:

1. Start with the prior $p(\boldsymbol{\theta})$
2. Observe $y_1$, encode using predictive distribution $p(y_1 | \mathcal{M})$
3. Update to posterior $p(\boldsymbol{\theta} | y_1)$
4. Observe $y_2$, encode using $p(y_2 | y_1, \mathcal{M})$
5. Continue for all data...

The total code length is:

$$L = -\sum_{t=1}^{n} \log p(y_t | y_1, \ldots, y_{t-1}, \mathcal{M}) = -\log p(\mathcal{D} | \mathcal{M})$$

This is exactly the negative log marginal likelihood! The Bayesian updates naturally implement an optimal sequential compression scheme.

Implications:

• Models with high evidence are good at predicting the next data point throughout observation
• Evidence captures both learning efficiency (how fast the model adapts) and prediction quality (how accurately it predicts)
• Overfitting is penalized because an overfit model predicts noise, which is costly to encode

An important application of model evidence is evidence maximization (also called Type-II Maximum Likelihood or empirical Bayes): learning hyperparameters by maximizing marginal likelihood.

The Setup

Consider a model with:

• Parameters $\boldsymbol{\theta}$: What we want to infer (e.g., regression weights)
• Hyperparameters $\boldsymbol{\alpha}$: Control the prior on $\boldsymbol{\theta}$ (e.g., prior variance)

The marginal likelihood is a function of hyperparameters:

$$p(\mathcal{D} | \boldsymbol{\alpha}) = \int p(\mathcal{D} | \boldsymbol{\theta}) p(\boldsymbol{\theta} | \boldsymbol{\alpha}) d\boldsymbol{\theta}$$

Evidence maximization finds:

$$\boldsymbol{\alpha}^* = \arg\max_{\boldsymbol{\alpha}} p(\mathcal{D} | \boldsymbol{\alpha})$$

This learns 'how much regularization' directly from the data, without cross-validation.

Example: Automatic Relevance Determination

In Automatic Relevance Determination (ARD), each feature has its own prior precision:

$$\beta_j \sim \mathcal{N}(0, \alpha_j^{-1})$$

Maximizing evidence over ${\alpha_j}$ automatically:

• Drives $\alpha_j \to \infty$ for irrelevant features (shrinking $\beta_j \to 0$)
• Keeps $\alpha_j$ modest for relevant features (allowing $\beta_j$ to fit data)

This achieves automatic feature selection purely through evidence maximization, without cross-validation or L1 regularization.

Example: Learning Kernel Parameters

In Gaussian process regression, the kernel function has hyperparameters (length scales, signal variance). Evidence maximization finds kernel hyperparameters that best explain the data:

$${\ell^, \sigma_f^, \sigma_n^*} = \arg\max \log p(\mathbf{y} | \mathbf{X}, \ell, \sigma_f, \sigma_n)$$

This is the standard approach for GP hyperparameter learning.

Model evidence is a powerful principle, but it has failure modes. Understanding when it fails is as important as knowing when it succeeds.

Failure Mode 1: Prior Misspecification

Model evidence rewards models that 'predicted' the data well from their prior. If the prior is badly misspecified, evidence unfairly penalizes the model:

• Example: True parameter is $\theta = 100$, but prior is $\theta \sim \mathcal{N}(0, 1)$
• Result: Prior assigns near-zero probability to the truth, so evidence is near-zero
• Problem: A perfectly good model is rejected due to bad prior calibration

Solution: Use principled priors based on domain knowledge or previous data. Conduct prior predictive checks to ensure priors allow reasonable data patterns.

Failure Mode 2: M-Open World

The M-open scenario occurs when the true data-generating process is not among the candidate models. In this case:

• Evidence picks the 'least wrong' model
• This may not be the best for prediction or understanding
• All models are misspecified, so evidence comparisons lose meaning

Example: True relationship is $y = \sin(x) + \epsilon$. We compare:

• Linear model
• Quadratic model

Evidence will favor quadratic (it's closer to the truth), but neither is 'right,' and the evidence values don't indicate how far either is from reality.

Solution: Supplement evidence with posterior predictive checks. Look for systematic patterns in residuals that indicate model misspecification.

Let's work through a complete example showing how model evidence guides model selection in a realistic scenario.

The Problem: Polynomial Regression Order Selection

We observe noisy data from an unknown function and must select the polynomial degree. This is a classic bias-variance trade-off:

• Too low degree: underfitting (high bias)
• Too high degree: overfitting (high variance)

Model evidence should automatically find the sweet spot.

What the Example Shows

1. Evidence peaks at the true complexity: For data from a cubic, evidence is highest around degree 3.

2. Simpler models have lower evidence: They can't capture the true pattern, hurting the likelihood term.

3. More complex models have lower evidence: Extra parameters spread prior mass thin without improving fit.

4. Posterior probabilities are decisive: With enough data, evidence strongly concentrates on the correct model.

5. BIC approximates evidence: For this smooth case, BIC rankings match evidence rankings closely.

This example demonstrates that model evidence operationalizes the intuition that we want 'as simple as possible, but no simpler.'

Let's consolidate what we've learned about model evidence:

Looking ahead: We've seen that model evidence embodies an automatic preference for simpler models—Occam's razor. In the next page, we'll explore this connection in depth, understanding why Bayesian inference inherently favors parsimony and what 'simplicity' really means in a probabilistic context.

The Bayesian Occam's razor isn't a heuristic or add-on—it emerges directly from the mathematics of probability. Understanding this provides deep insight into why Bayesian model comparison works.