Bayesian Model Comparison

In the previous page, we developed marginal likelihood—the probability of observing our data under a model, integrated over all possible parameter values. While marginal likelihood quantifies the absolute evidence for a single model, scientific inference rarely asks 'How probable is this model?' in isolation. Instead, we ask: 'Which model better explains the data?' or 'How much more evidence supports Model A over Model B?'

The Bayes factor answers precisely this question. It is the ratio of marginal likelihoods between two competing models, providing a principled, calibrated measure of relative evidential support. Unlike p-values (which measure how surprising data would be under a null hypothesis), Bayes factors directly quantify how much one model is favored over another given the observed data.

The Bayes Factor Defined

Given two models $\mathcal{M}_1$ and $\mathcal{M}_2$, the Bayes factor in favor of $\mathcal{M}_1$ over $\mathcal{M}_2$ is defined as the ratio of their marginal likelihoods:

$$\text{BF}_{12} = \frac{p(\mathcal{D} | \mathcal{M}_1)}{p(\mathcal{D} | \mathcal{M}_2)}$$

Expanding the marginal likelihoods:

$$\text{BF}_{12} = \frac{\int p(\mathcal{D} | \boldsymbol{\theta}_1, \mathcal{M}_1) p(\boldsymbol{\theta}_1 | \mathcal{M}_1) d\boldsymbol{\theta}_1}{\int p(\mathcal{D} | \boldsymbol{\theta}_2, \mathcal{M}_2) p(\boldsymbol{\theta}_2 | \mathcal{M}_2) d\boldsymbol{\theta}_2}$$

Interpretation: $\text{BF}_{12} = 10$ means the data is 10 times more probable under Model 1 than Model 2. The data provides 10-to-1 odds in favor of Model 1.

Connection to Posterior Model Probabilities

Bayes factors are intimately connected to posterior probabilities over models via Bayes' theorem at the model level:

$$\frac{p(\mathcal{M}_1 | \mathcal{D})}{p(\mathcal{M}_2 | \mathcal{D})} = \frac{p(\mathcal{D} | \mathcal{M}_1)}{p(\mathcal{D} | \mathcal{M}_2)} \cdot \frac{p(\mathcal{M}_1)}{p(\mathcal{M}_2)}$$

$$\underbrace{\text{Posterior Odds}}{\text{after seeing data}} = \underbrace{\text{Bayes Factor}}{\text{evidence from data}} \times \underbrace{\text{Prior Odds}}_{\text{before seeing data}}$$

The Bayes factor is the multiplicative update from prior odds to posterior odds. It represents purely the evidence contributed by the data, independent of prior model preferences.

If we start with equal prior probabilities ($p(\mathcal{M}_1) = p(\mathcal{M}_2) = 0.5$):

$$\text{Posterior Odds} = \text{BF}_{12}$$

This allows conversion to posterior probabilities:

$$p(\mathcal{M}1 | \mathcal{D}) = \frac{\text{BF}{12}}{1 + \text{BF}_{12}}$$

A Bayes factor of 5 means the data is 5 times more likely under one model. But is that 'strong' evidence? Researchers have proposed calibration scales to interpret Bayes factors consistently.

The Jeffreys Scale (1961)

Harold Jeffreys, one of the founders of Bayesian statistics, proposed the first widely-used interpretation scale. He worked with $\log_{10}(\text{BF})$ for convenience:

The Kass-Raftery Scale (1995)

Robert Kass and Adrian Raftery proposed a modified scale using $2 \ln(\text{BF})$, which has the same scale as likelihood ratio test statistics and the deviance:

Symmetric Interpretation

Bayes factors are symmetric under inversion:

$$\text{BF}{21} = \frac{1}{\text{BF}{12}}$$

This means the same scale applies in both directions:

• $\text{BF}_{12} = 100$: Strong evidence for Model 1
• $\text{BF}{12} = 0.01$: Strong evidence for Model 2 (since $\text{BF}{21} = 100$)
• $\text{BF}_{12} = 1$: No evidence either way (data equally likely under both models)

This symmetry is philosophically important. Unlike p-values, which only measure evidence against a null hypothesis, Bayes factors measure evidence in both directions. A small Bayes factor IS evidence—evidence for the alternative model.

Understanding how Bayes factors differ from frequentist hypothesis testing illuminates their strengths and appropriate use cases.

What P-Values Actually Measure

A p-value answers: 'If the null hypothesis were true, how often would I see data at least as extreme as what I observed?'

• It does NOT measure the probability that the null is true
• It does NOT measure the probability that the alternative is true
• It does NOT quantify evidence for the alternative
• It is computed entirely under the null, ignoring the alternative

What Bayes Factors Actually Measure

A Bayes factor answers: 'How many times more probable is my observed data under Model 1 compared to Model 2?'

• It directly compares both models
• It quantifies evidence strength on a continuous scale
• It can favor either model (not just reject one)
• It incorporates model complexity through priors

The Lindley Paradox

A famous example illustrating the difference: Consider testing whether a coin is fair ($\theta = 0.5$) vs. biased ($\theta \neq 0.5$).

With n = 100,000 flips and 50,500 heads:

• Frequentist test: p ≈ 0.0016 (highly significant!)
• Bayes factor: BF ≈ 0.08 (evidence FOR the null!)

How can this be?

The frequentist test asks: 'Is 50,500/100,000 significantly different from 0.5?' With huge sample size, even tiny differences are 'significant.'

The Bayesian test compares two models:

• Model 1 ($\theta = 0.5$): Makes a precise prediction
• Model 2 ($\theta$ unknown): Must spread prior over all possible values

The observed proportion (0.505) is very close to 0.5. Model 1, having made a precise prediction that was nearly correct, gets rewarded. Model 2, having hedged its bets across all possibilities, gets penalized for not committing.

The philosophical lesson: Testing for 'any difference' is different from testing for 'meaningful difference.' Bayes factors respect this distinction; p-values do not.

Since Bayes factors are ratios of marginal likelihoods, all methods for computing marginal likelihoods apply here. But there are also specialized methods for Bayes factor computation.

4.1 Direct Computation from Marginal Likelihoods

If both marginal likelihoods can be computed (analytically or by approximation), the Bayes factor is simply their ratio:

$$\log \text{BF}_{12} = \log p(\mathcal{D} | \mathcal{M}_1) - \log p(\mathcal{D} | \mathcal{M}_2)$$

This straightforward approach works well when both models have tractable marginal likelihoods.

4.2 Savage-Dickey Density Ratio

For nested models where one model is a special case of the other (e.g., testing $\theta = \theta_0$ vs. $\theta \neq \theta_0$), the Savage-Dickey density ratio provides an elegant shortcut:

$$\text{BF}_{01} = \frac{p(\theta = \theta_0 | \mathcal{D}, \mathcal{M}_1)}{p(\theta = \theta_0 | \mathcal{M}_1)}$$

Interpretation: The Bayes factor equals the posterior density at the null value divided by the prior density at the null value.

• If the posterior is higher than the prior at $\theta_0$, data supports the null
• If the posterior is lower than the prior at $\theta_0$, data favors the alternative

Advantage: You only need to fit the more complex model and evaluate densities at a point—no need to integrate.

4.3 Bridge Sampling

Bridge sampling is a powerful method for estimating Bayes factors, especially when models share parameters. The idea is to construct an 'optimal bridge' between two distributions.

Given samples from two posteriors, bridge sampling estimates:

$$\text{BF}_{12} \approx \frac{\sum_i h(\boldsymbol{\theta}^{(1)}_i) \cdot p(\boldsymbol{\theta}^{(1)}_i | \mathcal{M}_2)}{\sum_j h(\boldsymbol{\theta}^{(2)}_j) \cdot p(\boldsymbol{\theta}^{(2)}_j | \mathcal{M}_1)}$$

where $h(\cdot)$ is an optimally chosen 'bridge function.'

Key properties:

• Uses samples from both posteriors efficiently
• Achieves lower variance than importance sampling
• Works even when priors differ substantially
• Implemented in packages like bridgesampling in R

A particularly important application of Bayes factors is testing point null hypotheses: $H_0: \theta = \theta_0$ vs. $H_1: \theta \neq \theta_0$.

The Challenge with Point Nulls

In frequentist testing, the null hypothesis $\theta = 0$ has probability zero in a continuous parameter space, yet we can still compute p-values (probability of data more extreme than observed, given null).

In Bayesian testing, we must assign actual probability to the null hypothesis. This is done using a mixture prior:

$$p(\theta) = \pi_0 \cdot \delta(\theta - \theta_0) + (1 - \pi_0) \cdot p_1(\theta)$$

where:

• $\pi_0$ is the prior probability that $H_0$ is true
• $\delta(\cdot)$ is a point mass at $\theta_0$
• $p_1(\theta)$ is the prior under $H_1$

The JZS Prior for t-Tests

For testing whether a mean differs from zero, Rouder et al. (2009) developed the JZS (Jeffreys-Zellner-Siow) prior, now widely used for Bayesian t-tests:

$$\text{Effect size } \delta \sim \text{Cauchy}(0, r)$$

where $r$ (often 0.707) controls the expected effect size.

Why Cauchy?

• Heavy tails allow large effects if data supports them
• Scale parameter $r$ provides reasonable default expectations
• Closed-form Bayes factor exists for this choice

The resulting Bayes factor compares:

• $H_0$: True effect is exactly zero
• $H_1$: True effect is non-zero (Cauchy-distributed)

Despite their advantages, Bayes factors have important limitations. Understanding these is essential for appropriate application.

Prior Sensitivity

We emphasized this for marginal likelihoods, and it's doubly important for Bayes factors. Since BF = ML₁/ML₂, prior misspecification in either model affects the comparison.

Example: Two researchers test the same hypothesis with the same data:

• Researcher A uses tight priors: BF₁₀ = 25
• Researcher B uses diffuse priors: BF₁₀ = 3

Who's right? Both—given their priors. The solution is transparency about prior choices and sensitivity analysis.

The Bayes Factor 'Paradox'

Empty models (which make no predictions) can have arbitrarily high Bayes factors. Consider a model that says 'anything is possible' with uniform prior over vast ranges—its marginal likelihood is essentially the prior density, which can be made arbitrarily small.

Lesson: Bayes factors compare specific, well-defined models. Comparing a precise model to a vague 'catch-all' alternative isn't informative.

Alternatives to Bayes Factors

When Bayes factors are problematic, consider:

• Posterior predictive checks: Do predictions from your model match observed data patterns?
• Cross-validation: How well does each model predict held-out data?
• Information criteria: AIC, WAIC, LOO-CV provide different trade-offs
• Bayesian model averaging: Don't choose—combine predictions from all models (covered later in this module)

Let's synthesize everything into a practical workflow for using Bayes factors in research.

Reporting Guidelines

When publishing results using Bayes factors:

Essential to report:

• The Bayes factor value (BF₁₀ or BF₀₁ with clear notation)
• Prior distributions for all parameters under both models
• Computation method used
• Interpretation on a standard scale

Recommended to report:

• Posterior model probabilities (stating assumed prior odds)
• Sensitivity analysis results
• Sample code or references for reproducibility

Example reporting: 'The data provided strong evidence for the interaction model over the main-effects model, BF₁₀ = 27.3 (assuming equal prior odds). Under the interaction model, we used a Normal(0, 10²) prior on coefficients. Sensitivity analysis with prior scales [5, 10, 20] yielded BF₁₀ ∈ [18.1, 35.6], consistently supporting the interaction model.'

Let's consolidate what we've learned about Bayes factors:

Looking ahead: Bayes factors quantify relative evidence between two models, but the story doesn't end there. In the next page, we'll explore model evidence—understanding marginal likelihood as a measure of model quality that embodies a principled balance between fit and complexity, leading to the Bayesian interpretation of Occam's razor.