The Beta-Binomial model is the most fundamental and widely-used conjugate family in Bayesian statistics. Whenever we need to estimate a probability, proportion, or rate from binary data—success/failure, yes/no, click/no-click, convert/bounce—the Beta-Binomial model provides the canonical Bayesian solution.

Its ubiquity stems from a perfect storm of desirable properties:

• Mathematical elegance: The posterior update is trivially simple
• Interpretability: Hyperparameters have transparent pseudo-count meanings
• Flexibility: The Beta distribution can represent remarkably diverse prior beliefs
• Tractability: All quantities of interest have closed-form expressions

From A/B testing in tech companies to clinical trial analysis in medicine, from election forecasting in politics to quality control in manufacturing—the Beta-Binomial model underlies a vast array of practical applications. Mastering it provides the template for understanding all other conjugate systems.

The Beta distribution is a continuous probability distribution defined on the interval $[0, 1]$, making it the natural choice for modeling probabilities, proportions, and rates.

Definition: A random variable $\theta$ follows a Beta distribution with parameters $\alpha > 0$ and $\beta > 0$, written $\theta \sim \text{Beta}(\alpha, \beta)$, if its probability density function is:

$$p(\theta | \alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha - 1} (1 - \theta)^{\beta - 1}$$

where $\Gamma(\cdot)$ is the Gamma function, the continuous extension of the factorial: $\Gamma(n) = (n-1)!$ for positive integers.

The normalizing constant $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}$ is called the Beta function.

Key Properties of the Beta Distribution:

Mean (Expected Value): $$\mathbb{E}[\theta] = \frac{\alpha}{\alpha + \beta}$$

This is an intuitive weighted average: $\alpha$ "pulls" toward 1, $\beta$ "pulls" toward 0, and their relative magnitudes determine the balance.

Mode (Most Probable Value): $$\text{Mode}[\theta] = \frac{\alpha - 1}{\alpha + \beta - 2} \quad \text{(for } \alpha, \beta > 1\text{)}$$

The mode exists only when both parameters exceed 1; otherwise the distribution is U-shaped or monotonic.

Variance: $$\text{Var}[\theta] = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$$

The variance decreases as $\alpha + \beta$ (the effective sample size) increases, reflecting greater certainty.

Concentration Parameter: The sum $\kappa = \alpha + \beta$ controls how concentrated the distribution is around its mean. Larger $\kappa$ means lower variance and more peaked distributions.

Visual Intuition for Beta Shapes:

The Beta distribution's shape is entirely determined by $\alpha$ and $\beta$:

• $\alpha = \beta = 1$: Uniform (flat) distribution
• $\alpha = \beta > 1$: Symmetric and unimodal, centered at 0.5
• $\alpha = \beta < 1$: U-shaped, favoring extreme values (0 and 1)
• $\alpha > \beta$: Right-skewed, favoring larger values
• $\alpha < \beta$: Left-skewed, favoring smaller values
• Both large, $\alpha > \beta$: Concentrated, asymmetric, near mode $\frac{\alpha - 1}{\alpha + \beta - 2}$

This flexibility allows Beta priors to encode an enormous variety of prior beliefs about probability values.

The Binomial distribution models the number of successes in a sequence of independent binary trials. It is the natural likelihood when data consists of counts of positive outcomes out of total trials.

Definition: Let $k$ be the number of successes in $n$ independent trials, each with success probability $\theta$. Then $k$ follows a Binomial distribution:

$$P(k | n, \theta) = \binom{n}{k} \theta^k (1 - \theta)^{n-k}$$

where $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ is the binomial coefficient.

As a function of $\theta$ (the likelihood function): $$\mathcal{L}(\theta | k, n) = \binom{n}{k} \theta^k (1 - \theta)^{n-k} \propto \theta^k (1 - \theta)^{n-k}$$

The binomial coefficient $\binom{n}{k}$ is constant with respect to $\theta$, so for inference purposes, we work with the likelihood kernel: $\mathcal{L}(\theta) \propto \theta^k (1 - \theta)^{n-k}$.

Why Binomial Likelihood and Beta Prior are Conjugate:

The key observation is that the Binomial likelihood kernel has the same functional form as the Beta density kernel. Compare:

• Beta kernel: $\theta^{\alpha - 1} (1 - \theta)^{\beta - 1}$
• Binomial kernel: $\theta^k (1 - \theta)^{n-k}$

Both are products of powers of $\theta$ and $(1 - \theta)$. When we multiply them (as Bayes' theorem requires), the result is another product of powers:

$$\text{Posterior kernel} = \theta^{\alpha - 1} (1 - \theta)^{\beta - 1} \cdot \theta^k (1 - \theta)^{n-k} = \theta^{\alpha + k - 1} (1 - \theta)^{\beta + n - k - 1}$$

This is exactly a Beta kernel with updated parameters. The posterior is $\text{Beta}(\alpha + k, \beta + n - k)$.

Let us derive the Beta-Binomial posterior carefully, establishing the reasoning pattern used throughout conjugate analysis.

Setup:

• Prior: $\theta \sim \text{Beta}(\alpha, \beta)$
• Data: $k$ successes in $n$ trials (equivalently, Binomial($n, \theta$) observed as $k$)
• Goal: Find $P(\theta | k, n)$

Step 1: Write Prior Density $$P(\theta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha - 1} (1 - \theta)^{\beta - 1}$$

Step 2: Write Likelihood $$P(k | \theta, n) = \binom{n}{k} \theta^k (1 - \theta)^{n - k}$$

Step 3: Apply Bayes' Theorem $$P(\theta | k, n) = \frac{P(k | \theta, n) \cdot P(\theta)}{P(k | n)}$$

Step 4: Focus on the Kernel Since $P(k | n)$ is independent of $\theta$, we work with the numerator: $$P(\theta | k, n) \propto P(k | \theta, n) \cdot P(\theta)$$

$$\propto \theta^k (1 - \theta)^{n-k} \cdot \theta^{\alpha - 1} (1 - \theta)^{\beta - 1}$$

$$= \theta^{\alpha + k - 1} (1 - \theta)^{\beta + n - k - 1}$$

Step 5: Recognize the Posterior Family This kernel has the form $\theta^{a-1}(1-\theta)^{b-1}$ with $a = \alpha + k$ and $b = \beta + n - k$. This is the Beta$(a, b)$ kernel.

Step 6: Conclude $$\theta | k, n \sim \text{Beta}(\alpha + k, \beta + n - k)$$

The posterior is a Beta distribution with:

• First parameter: prior $\alpha$ plus observed successes $k$
• Second parameter: prior $\beta$ plus observed failures $(n - k)$

The Marginal Likelihood (Evidence):

For completeness, the marginal likelihood $P(k | n)$ that we factored out can be computed:

$$P(k | n) = \int_0^1 P(k | \theta, n) P(\theta) d\theta = \binom{n}{k} \cdot \frac{B(\alpha + k, \beta + n - k)}{B(\alpha, \beta)}$$

This quantity is useful for:

1. Model comparison: Comparing Beta-Binomial against alternative models
2. Bayes factors: Computing the ratio of evidences between models
3. Predictive distributions: As we'll see later

The fact that this integral has a closed form is another manifestation of conjugacy's computational elegance.

While the full posterior distribution is the complete Bayesian answer, we often need to summarize it with point estimates and intervals for decision-making and reporting.

Posterior Mean: $$\mathbb{E}[\theta | k, n] = \frac{\alpha + k}{\alpha + \beta + n}$$

The posterior mean is a weighted average of the prior mean and the maximum likelihood estimate (MLE):

$$\mathbb{E}[\theta | k, n] = \underbrace{\frac{\alpha + \beta}{\alpha + \beta + n}}{\text{prior weight}} \cdot \underbrace{\frac{\alpha}{\alpha + \beta}}{\text{prior mean}} + \underbrace{\frac{n}{\alpha + \beta + n}}{\text{data weight}} \cdot \underbrace{\frac{k}{n}}{\text{MLE}}$$

This formula reveals the Bayesian compromise: the posterior blends prior belief and observed evidence, with weights proportional to their respective "sample sizes" ($\alpha + \beta$ for the prior, $n$ for the data).

Posterior Mode (MAP Estimate): $$\text{Mode}[\theta | k, n] = \frac{\alpha + k - 1}{\alpha + \beta + n - 2} \quad \text{(for } \alpha + k > 1, \beta + n - k > 1 \text{)}$$

The Maximum A Posteriori (MAP) estimate is the most probable value under the posterior. It differs from the posterior mean when the distribution is asymmetric.

Posterior Median: The median satisfies $P(\theta \leq \tilde{\theta} | k, n) = 0.5$. It equals the 0.5 quantile of Beta($\alpha + k, \beta + n - k$). While there's no closed-form expression, it's easily computed numerically.

Credible Intervals: Bayesian credible intervals directly quantify parameter uncertainty. The $100(1-\alpha)$% credible interval $[\theta_L, \theta_U]$ satisfies:

$$P(\theta_L \leq \theta \leq \theta_U | k, n) = 1 - \alpha$$

Two common choices:

1. Equal-tailed interval: $\theta_L$ and $\theta_U$ are the $\alpha/2$ and $1 - \alpha/2$ quantiles
2. Highest Density Interval (HDI): The narrowest interval containing $1-\alpha$ probability mass

The Role of Sample Size:

As $n \to \infty$, the posterior concentrates around the true parameter:

• Posterior mean $\to k/n$ (the MLE)
• Posterior variance $\to 0$
• Credible intervals shrink to a point

This is Bayesian consistency: with sufficient data, the posterior converges to the truth regardless of the prior (assuming the true parameter is in the prior's support). The prior matters most with small samples—precisely when subjective knowledge is most valuable.

Practical Rule of Thumb: The prior's influence is roughly equivalent to $\alpha + \beta - 2$ observations. If you have $n >> \alpha + \beta$, the prior barely matters. If $n << \alpha + \beta$, the prior dominates.

A crucial Bayesian quantity is the posterior predictive distribution—the probability distribution of future observations, accounting for parameter uncertainty. Instead of plugging in a point estimate for $\theta$, we average predictions over all plausible $\theta$ values, weighted by the posterior.

Definition: Given observed data $D = (k, n)$ and a future trial, the posterior predictive probability of success is:

$$P(\tilde{x} = 1 | D) = \int_0^1 P(\tilde{x} = 1 | \theta) P(\theta | D) d\theta = \int_0^1 \theta \cdot \text{Beta}(\theta | \alpha + k, \beta + n - k) d\theta$$

$$= \mathbb{E}[\theta | D] = \frac{\alpha + k}{\alpha + \beta + n}$$

The probability of the next trial being a success equals the posterior mean of $\theta$.

The Beta-Binomial Distribution:

For predictions about multiple future trials, the posterior predictive distribution has a name: the Beta-Binomial distribution.

If $\tilde{k}$ is the number of successes in $m$ future trials, given posterior Beta($\alpha + k, \beta + n - k$):

$$P(\tilde{k} | k, n, m) = \binom{m}{\tilde{k}} \frac{B(\alpha + k + \tilde{k}, \beta + n - k + m - \tilde{k})}{B(\alpha + k, \beta + n - k)}$$

This distribution accounts for both:

1. Binomial variability: randomness in which future trials succeed/fail
2. Parameter uncertainty: uncertainty about the true success rate

Properties of the Beta-Binomial:

Mean: $$\mathbb{E}[\tilde{k}] = m \cdot \frac{\alpha + k}{\alpha + \beta + n}$$

Variance: $$\text{Var}[\tilde{k}] = m \cdot \frac{(\alpha + k)(\beta + n - k)}{(\alpha + \beta + n)^2} \cdot \frac{\alpha + \beta + n + m}{\alpha + \beta + n + 1}$$

The variance is larger than the Binomial variance $mp(1-p)$ because it includes parameter uncertainty. The extra factor $\frac{\alpha + \beta + n + m}{\alpha + \beta + n + 1}$ captures this additional variation.

The Beta-Binomial model underpins countless real-world inference systems. Let's examine several important applications in depth.

Application 1: A/B Testing

A/B testing compares two variants (A and B) to determine which performs better. The Beta-Binomial model provides a fully Bayesian framework.

Setup:

• Variant A: $k_A$ conversions in $n_A$ visitors, prior Beta($\alpha_A, \beta_A$)
• Variant B: $k_B$ conversions in $n_B$ visitors, prior Beta($\alpha_B, \beta_B$)

Posteriors:

• $\theta_A | \text{data} \sim \text{Beta}(\alpha_A + k_A, \beta_A + n_A - k_A)$
• $\theta_B | \text{data} \sim \text{Beta}(\alpha_B + k_B, \beta_B + n_B - k_B)$

Key Quantities:

1. Probability B beats A: $P(\theta_B > \theta_A | \text{data})$

   • No closed form for general Beta distributions
   • Easily computed by sampling or numerical integration

2. Expected lift: $\mathbb{E}[\theta_B - \theta_A | \text{data}] = \mathbb{E}[\theta_B] - \mathbb{E}[\theta_A]$

3. Credible interval for difference: Distribution of $\theta_B - \theta_A$

Application 2: Thompson Sampling for Multi-Armed Bandits

Thompson Sampling is a Bayesian algorithm for the exploration-exploitation tradeoff in multi-armed bandit problems. With Beta-Binomial models, it's remarkably simple:

Algorithm:

1. Maintain Beta posteriors for each arm's success probability
2. At each round:
   • Sample $\tilde{\theta}_i \sim \text{Beta}(\alpha_i, \beta_i)$ for each arm
   • Play the arm with highest sampled value: $\arg\max_i \tilde{\theta}_i$
   • Observe outcome and update posterior

This simple algorithm is provably optimal (in terms of regret bounds) and naturally balances:

• Exploration: Arms with high uncertainty get sampled due to variance
• Exploitation: Arms with high means get sampled frequently

Application 3: Clinical Trial Analysis

In clinical trials, Binary outcomes (cure/no cure, adverse event/no adverse event) are common. Bayesian analysis provides:

• Prior incorporation: Historical trial data informs priors
• Sequential monitoring: Valid inference at any interim analysis
• Predictive probabilities: "What's the probability of trial success given current data?"
• Decision-theoretic stopping rules: Based on expected utility, not p-values

While Beta-Binomial inference is generally well-behaved, practitioners should be aware of edge cases and potential issues.

Edge Case 1: Zero Observations ($k = 0$ or $k = n$)

When all trials are successes or all are failures:

• MLE problem: $\hat{\theta} = 0$ or $\hat{\theta} = 1$ (extreme estimates)
• Bayesian solution: Posterior shrinks toward prior mean

With prior Beta(1, 1) and 0 successes in 10 trials:

• MLE: $\hat{\theta} = 0$
• Posterior: Beta(1, 11), mean = 1/12 ≈ 0.083

The Bayesian estimate is non-zero, reflecting that we shouldn't be certain the rate is exactly zero after just 10 trials.

Edge Case 2: Improper Priors ($\alpha < 1$ or $\beta < 1$)

Priors like Beta(0.5, 0.5) (Jeffreys) or Beta(0, 0) (Haldane) are improper—they don't integrate to 1. However:

• Beta(0.5, 0.5) with any data $k \in {1, \ldots, n-1}$ yields a proper posterior
• Beta(0, 0) requires at least one success AND one failure for a proper posterior

Edge Case 3: Very Strong Priors

If $\alpha + \beta >> n$, the prior dominates. This can be problematic if:

• The prior is based on outdated information
• The current data represents a different population
• The prior was incorrectly specified

Solution: Always check prior predictive distribution. If prior predicts data very different from what's observed, reconsider the prior.

Overdispersion and Beta-Binomial as Likelihood:

Interestingly, the Beta-Binomial distribution (our posterior predictive) can also serve as a likelihood for overdispersed count data:

If counts show more variance than Binomial (due to hidden heterogeneity across trials), the Beta-Binomial likelihood models this: $$k \sim \text{BetaBinomial}(n, \alpha, \beta)$$

Here $\alpha$ and $\beta$ parameterize the heterogeneity distribution of success probabilities across trials, not prior uncertainty. This is a subtle but important distinction in usage.

We have thoroughly explored Beta-Binomial conjugacy—the foundation of Bayesian proportion inference. Let's consolidate the essential knowledge:

What's Next:

Having mastered Beta-Binomial conjugacy, we now extend to continuous data. The next page covers Gaussian-Gaussian conjugacy—the foundation for Bayesian inference on continuous measurements, regression, and countless extensions in modern machine learning.