Common Probability Distributions

The distributions we've studied so far—Bernoulli, Binomial, Gaussian, Poisson—might seem like independent mathematical objects. But beneath the surface lies a remarkable unification: these distributions, along with many others, are all members of a single family called the Exponential Family.\n\nThe exponential family isn't just a taxonomic classification; it's a mathematical framework with deep theoretical and practical implications:\n\n- Sufficient statistics exist and are finite-dimensional\n- Maximum likelihood estimators have elegant closed-form solutions\n- Conjugate priors exist for Bayesian inference\n- Generalized Linear Models (GLMs) naturally arise from this framework\n- Gradient-based optimization is well-behaved\n\nUnderstanding exponential families transforms your view of probability from a collection of separate distributions into a coherent mathematical structure. It explains why certain distributions are "nice" to work with and provides a template for constructing new distributions with desirable properties.

A probability distribution belongs to the exponential family if its probability density (or mass) function can be written in the following canonical form:\n\n$$p(x | \boldsymbol{\eta}) = h(x) \exp\left(\boldsymbol{\eta}^T \mathbf{T}(x) - A(\boldsymbol{\eta})\right)$$\n\nor equivalently:\n\n$$p(x | \boldsymbol{\eta}) = \frac{h(x) \exp\left(\boldsymbol{\eta}^T \mathbf{T}(x)\right)}{\exp(A(\boldsymbol{\eta}))}$$\n\nwhere:\n\n- x is the observation (can be scalar, vector, or more complex)\n- η (eta) is the natural parameter (or canonical parameter) vector\n- T(x) is the sufficient statistic vector\n- h(x) is the base measure or carrier measure\n- A(η) is the log-partition function (or log-normalizer, cumulant function)

The Log-Partition Function\n\nThe log-partition function A(η) is defined by the normalization requirement:\n\n$$\int h(x) \exp(\boldsymbol{\eta}^T \mathbf{T}(x)) \, dx = \exp(A(\boldsymbol{\eta}))$$\n\nThus:\n\n$$A(\boldsymbol{\eta}) = \log \int h(x) \exp(\boldsymbol{\eta}^T \mathbf{T}(x)) \, dx$$\n\nCritical insight: The log-partition function is the "key" to the distribution—it encodes all the moments. Specifically:\n\n$$\nabla A(\boldsymbol{\eta}) = E[\mathbf{T}(x)]$$\n$$\nabla^2 A(\boldsymbol{\eta}) = \text{Cov}[\mathbf{T}(x)]$$\n\nThe first derivative gives the expected sufficient statistics, and the second derivative (Hessian) gives their covariance matrix.\n\n### Convexity\n\nA remarkable property: A(η) is always convex (and often strictly convex). This is because the Hessian ∇²A(η) = Cov[T(x)] is a covariance matrix, which is always positive semi-definite.\n\nThis convexity ensures that maximum likelihood estimation for exponential families has globally optimal solutions with no local minima.

Let's express the distributions we've studied in exponential family form. This exercise reveals their underlying structure and connections.\n\n### Bernoulli Distribution\n\nStandard form: P(x | p) = p^x (1-p)^{1-x} for x ∈ {0, 1}\n\nExponential family form:\n\n$$P(x | p) = \exp\left(x \log\frac{p}{1-p} + \log(1-p)\right)$$\n\n- Natural parameter: η = log(p/(1-p)) = logit(p)\n- Sufficient statistic: T(x) = x\n- Base measure: h(x) = 1\n- Log-partition: A(η) = log(1 + e^η) = -log(1-p)\n\nInverse mapping: p = σ(η) = 1/(1 + e^{-η}) (sigmoid function!)\n\nThis explains why logistic regression uses the sigmoid—it's the inverse of the natural parameterization!

Gaussian Distribution (known variance)\n\nStandard form: p(x | μ, σ²) = (1/√(2πσ²)) exp(-(x-μ)²/(2σ²))\n\nWith σ² known, the exponential family form is:\n\n$$p(x | \mu) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{\mu x}{\sigma^2} - \frac{\mu^2}{2\sigma^2} - \frac{x^2}{2\sigma^2}\right)$$\n\n- Natural parameter: η = μ/σ²\n- Sufficient statistic: T(x) = x\n- Base measure: h(x) = (1/√(2πσ²)) exp(-x²/(2σ²))\n- Log-partition: A(η) = η²σ²/2 = μ²/(2σ²)\n\n### Gaussian (both parameters unknown)\n\nWith both μ and σ² unknown, we have a 2-parameter exponential family:\n\n- Natural parameters: η₁ = μ/σ², η₂ = -1/(2σ²)\n- Sufficient statistics: T(x) = (x, x²)\n\nNotice: T(x) now includes both x and x²—we need the second moment to estimate variance!

A sufficient statistic T(x) is a function of the data that captures all information about the parameter η. Formally, the conditional distribution of X given T(X) does not depend on η.\n\n### The Fisher-Neyman Factorization\n\nA statistic T(X) is sufficient for η if and only if the density can be factored as:\n\n$$p(x | \eta) = g(T(x), \eta) \cdot h(x)$$\n\nwhere g depends on x only through T(x), and h doesn't depend on η.\n\nFor exponential families, this factorization is immediate from the canonical form!\n\n### Why Sufficient Statistics Matter\n\n1. Data reduction: You can summarize arbitrarily large datasets by computing T(X), losing no information about η.\n\nExample: For n i.i.d. Gaussian observations, the sufficient statistics are Σxᵢ and Σxᵢ². No matter how large n is, these two numbers contain all information about μ and σ².\n\n2. MLE has closed form: For exponential families, MLE just involves setting the observed sufficient statistics equal to their expected values:\n\n$$E_{\hat{\eta}}[\mathbf{T}(X)] = \frac{1}{n}\sum_{i=1}^n \mathbf{T}(x_i)$$

Sufficient Statistics for n i.i.d. Observations\n\nFor i.i.d. data x₁, ..., xₙ from an exponential family, the joint distribution is:\n\n$$p(x_1, \ldots, x_n | \eta) = \prod_{i=1}^n h(x_i) \exp\left(\eta^T \sum_{i=1}^n T(x_i) - nA(\eta)\right)$$\n\nThis shows that Σᵢ T(xᵢ) is sufficient for η. The dimensionality of the sufficient statistic is fixed, regardless of sample size!\n\n### Examples\n\n| Distribution | Sufficient Statistic for n samples |\n|--------------|-----------------------------------|\n| Bernoulli(p) | Σxᵢ (count of 1s) |\n| Poisson(λ) | Σxᵢ (total count) |\n| Gaussian(μ, σ²) | (Σxᵢ, Σxᵢ²) |\n| Exponential(λ) | Σxᵢ |\n| Gamma(α known, β) | Σxᵢ |\n\nThis fixed-dimensional sufficiency is unique to exponential families and is one of their key practical advantages.

MLE for exponential families has a beautiful structure arising from the properties of the log-partition function.\n\n### The Log-Likelihood\n\nFor n i.i.d. observations, the log-likelihood is:\n\n$$\ell(\eta) = \sum_{i=1}^n \log h(x_i) + \eta^T \sum_{i=1}^n T(x_i) - nA(\eta)$$\n\nThe gradient with respect to η is:\n\n$$\nabla_\eta \ell(\eta) = \sum_{i=1}^n T(x_i) - n \nabla A(\eta) = \sum_{i=1}^n T(x_i) - n E_\eta[T(X)]$$\n\nSetting the gradient to zero:\n\n$$E_{\hat{\eta}}[T(X)] = \frac{1}{n}\sum_{i=1}^n T(x_i) = \bar{T}$$\n\nThe MLE is found by matching expected sufficient statistics to observed sufficient statistics!

Convexity Guarantees\n\nThe Hessian of the log-likelihood is:\n\n$$\nabla^2_\eta \ell(\eta) = -n \nabla^2 A(\eta) = -n \cdot \text{Cov}_\eta[T(X)]$$\n\nSince covariance matrices are positive semi-definite, the Hessian is negative semi-definite, confirming that the log-likelihood is concave (convex optimization problem when minimizing negative log-likelihood).\n\nImplications:\n- No local maxima to get stuck in\n- Gradient descent converges to global optimum\n- Newton's method converges quickly\n\n### The Moment-Matching Perspective\n\nMLE can be viewed as finding the distribution in the exponential family whose expected sufficient statistics match the sample averages. This is elegant:\n\n1. Compute T̄ = (1/n)Σᵢ T(xᵢ) from data\n2. Find η such that E_η[T(X)] = T̄\n3. This η is the MLE\n\nFor many distributions, this gives closed-form solutions. For others, we can use convex optimization.

For exponential families, there exists a systematic way to construct conjugate priors—priors where the posterior has the same functional form as the prior, just with updated parameters.\n\n### The General Form\n\nFor a likelihood from an exponential family:\n\n$$p(x | \eta) = h(x) \exp(\eta^T T(x) - A(\eta))$$\n\nThe conjugate prior for η has the form:\n\n$$p(\eta | \chi, \nu) \propto \exp(\eta^T \chi - \nu A(\eta))$$\n\nwhere χ and ν are hyperparameters of the prior.\n\n### Posterior Update\n\nAfter observing x₁, ..., xₙ, the posterior is:\n\n$$p(\eta | x_1, \ldots, x_n) \propto p(\eta | \chi, \nu) \prod_{i=1}^n p(x_i | \eta)$$\n\n$$\propto \exp\left(\eta^T \left(\chi + \sum_{i=1}^n T(x_i)\right) - (\nu + n) A(\eta)\right)$$\n\nThis is the same form with updated hyperparameters:\n\n$$\chi' = \chi + \sum_{i=1}^n T(x_i), \quad \nu' = \nu + n$$\n\nThe prior 'pretends' to have seen ν observations with sufficient statistics χ.

Example: Beta-Bernoulli Conjugacy\n\nPrior: p ~ Beta(α, β)\n\nThis means p has prior PDF: p(p | α, β) ∝ p^{α-1}(1-p)^{β-1}\n\nAfter observing n Bernoulli trials with k successes:\n\nPosterior: p | data ~ Beta(α + k, β + n - k)\n\nInterpretation:\n- Prior α, β act as "pseudo-counts" of successes and failures\n- Each observation updates the appropriate count\n- Prior strength is α + β (effective prior sample size)\n\n### Why Conjugate Priors Matter\n\n1. Computational convenience: Closed-form posteriors without MCMC\n\n2. Interpretability: Prior parameters have intuitive meanings\n\n3. Sequential updating: Easy to incorporate new data incrementally\n\n4. Posterior predictive: Often has closed form

Generalized Linear Models (GLMs) extend linear regression to response variables from any exponential family distribution. The exponential family framework provides the theoretical foundation.\n\n### GLM Structure\n\nA GLM has three components:\n\n1. Random Component: The response Y follows an exponential family distribution:\n$$Y | X \sim \text{ExpFam}(\eta)$$\n\n2. Systematic Component: A linear predictor:\n$$\eta_{\text{linear}} = \mathbf{w}^T \mathbf{x} + b$$\n\n3. Link Function: Connects the linear predictor to the natural parameter:\n$$\eta = g(E[Y]) \quad \text{or equivalently} \quad E[Y] = g^{-1}(\eta_{\text{linear}})$$\n\n### The Canonical Link\n\nThe canonical link function is the one where the linear predictor equals the natural parameter:\n$$\eta_{\text{linear}} = \eta_{\text{natural}}$$\n\nThis gives the simplest, most numerically stable GLMs.

Why Canonical Links Are Special\n\n1. Sufficient statistics align: The sufficient statistic T(Y) = Y (or simple function of Y) combines directly with features.\n\n2. Convexity guaranteed: The negative log-likelihood is convex in the parameters.\n\n3. Fisher scoring = Newton-Raphson: Optimization algorithms simplify.\n\n4. Variance function determines all: The relationship Var(Y) = V(μ) × φ characterizes the family.\n\n### The Exponential Family Unifies GLMs\n\nAll GLMs can be written as:\n\n$$\log p(y | \mathbf{x}, \mathbf{w}) = \frac{y \cdot \theta(\mathbf{x}, \mathbf{w}) - b(\theta(\mathbf{x}, \mathbf{w}))}{a(\phi)} + c(y, \phi)$$\n\nwhere θ = g⁻¹(w^T x) maps features to the natural parameter.\n\nThis unified view shows that logistic regression, Poisson regression, and linear regression are all instances of the same framework—just with different exponential family distributions!

The exponential family is a unifying framework that connects most distributions used in machine learning. Let's consolidate the key insights:

What's Next:\n\nWe've explored the exponential family as a unifying framework. Next, we advance to the Multivariate Gaussian distribution—extending the univariate Gaussian to multiple dimensions. This distribution is fundamental to multivariate statistics, Gaussian processes, variational autoencoders, and countless other machine learning methods. We'll explore its geometry, conditioning, marginalization, and the central role of the covariance matrix.