Common Probability Distributions

The distributions we've studied so far—Bernoulli, Binomial, Gaussian, Poisson—might seem like independent mathematical objects. But beneath the surface lies a remarkable unification: these distributions, along with many others, are all members of a single family called the Exponential Family.

The exponential family isn't just a taxonomic classification; it's a mathematical framework with deep theoretical and practical implications:

• Sufficient statistics exist and are finite-dimensional
• Maximum likelihood estimators have elegant closed-form solutions
• Conjugate priors exist for Bayesian inference
• Generalized Linear Models (GLMs) naturally arise from this framework
• Gradient-based optimization is well-behaved

Understanding exponential families transforms your view of probability from a collection of separate distributions into a coherent mathematical structure. It explains why certain distributions are "nice" to work with and provides a template for constructing new distributions with desirable properties.

A probability distribution belongs to the exponential family if its probability density (or mass) function can be written in the following canonical form:

$$p(x | \boldsymbol{\eta}) = h(x) \exp\left(\boldsymbol{\eta}^T \mathbf{T}(x) - A(\boldsymbol{\eta})\right)$$

or equivalently:

$$p(x | \boldsymbol{\eta}) = \frac{h(x) \exp\left(\boldsymbol{\eta}^T \mathbf{T}(x)\right)}{\exp(A(\boldsymbol{\eta}))}$$

where:

• x is the observation (can be scalar, vector, or more complex)
• η (eta) is the natural parameter (or canonical parameter) vector
• T(x) is the sufficient statistic vector
• h(x) is the base measure or carrier measure
• A(η) is the log-partition function (or log-normalizer, cumulant function)

The Log-Partition Function

The log-partition function A(η) is defined by the normalization requirement:

$$\int h(x) \exp(\boldsymbol{\eta}^T \mathbf{T}(x)) \, dx = \exp(A(\boldsymbol{\eta}))$$

Thus:

$$A(\boldsymbol{\eta}) = \log \int h(x) \exp(\boldsymbol{\eta}^T \mathbf{T}(x)) \, dx$$

Critical insight: The log-partition function is the "key" to the distribution—it encodes all the moments. Specifically:

$$ abla A(\boldsymbol{\eta}) = E[\mathbf{T}(x)]$$ $$ abla^2 A(\boldsymbol{\eta}) = \text{Cov}[\mathbf{T}(x)]$$

The first derivative gives the expected sufficient statistics, and the second derivative (Hessian) gives their covariance matrix.

Convexity

A remarkable property: A(η) is always convex (and often strictly convex). This is because the Hessian ∇²A(η) = Cov[T(x)] is a covariance matrix, which is always positive semi-definite.

This convexity ensures that maximum likelihood estimation for exponential families has globally optimal solutions with no local minima.

Let's express the distributions we've studied in exponential family form. This exercise reveals their underlying structure and connections.

Bernoulli Distribution

Standard form: P(x | p) = p^x (1-p)^{1-x} for x ∈ {0, 1}

Exponential family form:

$$P(x | p) = \exp\left(x \log\frac{p}{1-p} + \log(1-p)\right)$$

• Natural parameter: η = log(p/(1-p)) = logit(p)
• Sufficient statistic: T(x) = x
• Base measure: h(x) = 1
• Log-partition: A(η) = log(1 + e^η) = -log(1-p)

Inverse mapping: p = σ(η) = 1/(1 + e^{-η}) (sigmoid function!)

This explains why logistic regression uses the sigmoid—it's the inverse of the natural parameterization!

Gaussian Distribution (known variance)

Standard form: p(x | μ, σ²) = (1/√(2πσ²)) exp(-(x-μ)²/(2σ²))

With σ² known, the exponential family form is:

$$p(x | \mu) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{\mu x}{\sigma^2} - \frac{\mu^2}{2\sigma^2} - \frac{x^2}{2\sigma^2}\right)$$

• Natural parameter: η = μ/σ²
• Sufficient statistic: T(x) = x
• Base measure: h(x) = (1/√(2πσ²)) exp(-x²/(2σ²))
• Log-partition: A(η) = η²σ²/2 = μ²/(2σ²)

Gaussian (both parameters unknown)

With both μ and σ² unknown, we have a 2-parameter exponential family:

• Natural parameters: η₁ = μ/σ², η₂ = -1/(2σ²)
• Sufficient statistics: T(x) = (x, x²)

Notice: T(x) now includes both x and x²—we need the second moment to estimate variance!

A sufficient statistic T(x) is a function of the data that captures all information about the parameter η. Formally, the conditional distribution of X given T(X) does not depend on η.

The Fisher-Neyman Factorization

A statistic T(X) is sufficient for η if and only if the density can be factored as:

$$p(x | \eta) = g(T(x), \eta) \cdot h(x)$$

where g depends on x only through T(x), and h doesn't depend on η.

For exponential families, this factorization is immediate from the canonical form!

Why Sufficient Statistics Matter

1. Data reduction: You can summarize arbitrarily large datasets by computing T(X), losing no information about η.

Example: For n i.i.d. Gaussian observations, the sufficient statistics are Σxᵢ and Σxᵢ². No matter how large n is, these two numbers contain all information about μ and σ².

2. MLE has closed form: For exponential families, MLE just involves setting the observed sufficient statistics equal to their expected values:

$$E_{\hat{\eta}}[\mathbf{T}(X)] = \frac{1}{n}\sum_{i=1}^n \mathbf{T}(x_i)$$

Sufficient Statistics for n i.i.d. Observations

For i.i.d. data x₁, ..., xₙ from an exponential family, the joint distribution is:

$$p(x_1, \ldots, x_n | \eta) = \prod_{i=1}^n h(x_i) \exp\left(\eta^T \sum_{i=1}^n T(x_i) - nA(\eta)\right)$$

This shows that Σᵢ T(xᵢ) is sufficient for η. The dimensionality of the sufficient statistic is fixed, regardless of sample size!

Examples

This fixed-dimensional sufficiency is unique to exponential families and is one of their key practical advantages.

MLE for exponential families has a beautiful structure arising from the properties of the log-partition function.

The Log-Likelihood

For n i.i.d. observations, the log-likelihood is:

$$\ell(\eta) = \sum_{i=1}^n \log h(x_i) + \eta^T \sum_{i=1}^n T(x_i) - nA(\eta)$$

The gradient with respect to η is:

$$ abla_\eta \ell(\eta) = \sum_{i=1}^n T(x_i) - n abla A(\eta) = \sum_{i=1}^n T(x_i) - n E_\eta[T(X)]$$

Setting the gradient to zero:

$$E_{\hat{\eta}}[T(X)] = \frac{1}{n}\sum_{i=1}^n T(x_i) = \bar{T}$$

The MLE is found by matching expected sufficient statistics to observed sufficient statistics!

Convexity Guarantees

The Hessian of the log-likelihood is:

$$ abla^2_\eta \ell(\eta) = -n abla^2 A(\eta) = -n \cdot \text{Cov}_\eta[T(X)]$$

Since covariance matrices are positive semi-definite, the Hessian is negative semi-definite, confirming that the log-likelihood is concave (convex optimization problem when minimizing negative log-likelihood).

Implications:

• No local maxima to get stuck in
• Gradient descent converges to global optimum
• Newton's method converges quickly

The Moment-Matching Perspective

MLE can be viewed as finding the distribution in the exponential family whose expected sufficient statistics match the sample averages. This is elegant:

1. Compute T̄ = (1/n)Σᵢ T(xᵢ) from data
2. Find η such that E_η[T(X)] = T̄
3. This η is the MLE

For many distributions, this gives closed-form solutions. For others, we can use convex optimization.

For exponential families, there exists a systematic way to construct conjugate priors—priors where the posterior has the same functional form as the prior, just with updated parameters.

The General Form

For a likelihood from an exponential family:

$$p(x | \eta) = h(x) \exp(\eta^T T(x) - A(\eta))$$

The conjugate prior for η has the form:

$$p(\eta | \chi, u) \propto \exp(\eta^T \chi - u A(\eta))$$

where χ and ν are hyperparameters of the prior.

Posterior Update

After observing x₁, ..., xₙ, the posterior is:

$$p(\eta | x_1, \ldots, x_n) \propto p(\eta | \chi, u) \prod_{i=1}^n p(x_i | \eta)$$

$$\propto \exp\left(\eta^T \left(\chi + \sum_{i=1}^n T(x_i)\right) - ( u + n) A(\eta)\right)$$

This is the same form with updated hyperparameters:

$$\chi' = \chi + \sum_{i=1}^n T(x_i), \quad u' = u + n$$

The prior 'pretends' to have seen ν observations with sufficient statistics χ.

Example: Beta-Bernoulli Conjugacy

Prior: p ~ Beta(α, β)

This means p has prior PDF: p(p | α, β) ∝ p^{α-1}(1-p)^{β-1}

After observing n Bernoulli trials with k successes:

Posterior: p | data ~ Beta(α + k, β + n - k)

Interpretation:

• Prior α, β act as "pseudo-counts" of successes and failures
• Each observation updates the appropriate count
• Prior strength is α + β (effective prior sample size)

Why Conjugate Priors Matter

1. Computational convenience: Closed-form posteriors without MCMC

2. Interpretability: Prior parameters have intuitive meanings

3. Sequential updating: Easy to incorporate new data incrementally

4. Posterior predictive: Often has closed form

Generalized Linear Models (GLMs) extend linear regression to response variables from any exponential family distribution. The exponential family framework provides the theoretical foundation.

GLM Structure

A GLM has three components:

1. Random Component: The response Y follows an exponential family distribution: $$Y | X \sim \text{ExpFam}(\eta)$$

2. Systematic Component: A linear predictor: $$\eta_{\text{linear}} = \mathbf{w}^T \mathbf{x} + b$$

3. Link Function: Connects the linear predictor to the natural parameter: $$\eta = g(E[Y]) \quad \text{or equivalently} \quad E[Y] = g^{-1}(\eta_{\text{linear}})$$

The Canonical Link

The canonical link function is the one where the linear predictor equals the natural parameter: $$\eta_{\text{linear}} = \eta_{\text{natural}}$$

This gives the simplest, most numerically stable GLMs.

Why Canonical Links Are Special

1. Sufficient statistics align: The sufficient statistic T(Y) = Y (or simple function of Y) combines directly with features.

2. Convexity guaranteed: The negative log-likelihood is convex in the parameters.

3. Fisher scoring = Newton-Raphson: Optimization algorithms simplify.

4. Variance function determines all: The relationship Var(Y) = V(μ) × φ characterizes the family.

The Exponential Family Unifies GLMs

All GLMs can be written as:

$$\log p(y | \mathbf{x}, \mathbf{w}) = \frac{y \cdot \theta(\mathbf{x}, \mathbf{w}) - b(\theta(\mathbf{x}, \mathbf{w}))}{a(\phi)} + c(y, \phi)$$

where θ = g⁻¹(w^T x) maps features to the natural parameter.

This unified view shows that logistic regression, Poisson regression, and linear regression are all instances of the same framework—just with different exponential family distributions!

The exponential family is a unifying framework that connects most distributions used in machine learning. Let's consolidate the key insights:

What's Next:

We've explored the exponential family as a unifying framework. Next, we advance to the Multivariate Gaussian distribution—extending the univariate Gaussian to multiple dimensions. This distribution is fundamental to multivariate statistics, Gaussian processes, variational autoencoders, and countless other machine learning methods. We'll explore its geometry, conditioning, marginalization, and the central role of the covariance matrix.