Common Probability Distributions

Every classification problem in machine learning—spam detection, disease diagnosis, fraud identification, sentiment analysis—fundamentally reduces to predicting binary outcomes. Will the email be spam or not? Is the tumor malignant or benign? Is this transaction fraudulent or legitimate?\n\nTo model these binary outcomes probabilistically, we need a mathematical framework that captures the inherent randomness of such yes/no decisions. This is precisely what the Bernoulli and Binomial distributions provide. They are the simplest yet most fundamental probability distributions, forming the conceptual bedrock upon which more sophisticated models—logistic regression, naive Bayes classifiers, and neural network classification heads—are built.\n\nUnderstanding these distributions is not merely an academic exercise; it is essential for grasping how machine learning models quantify uncertainty, how we evaluate classifier performance, and how we make principled decisions under probabilistic predictions.

The Bernoulli distribution is the simplest discrete probability distribution, modeling a single trial with exactly two possible outcomes: success (typically encoded as 1) and failure (encoded as 0). Named after Swiss mathematician Jacob Bernoulli (1654–1705), who made foundational contributions to probability theory, this distribution captures the essence of random binary events.\n\n### Formal Definition\n\nA random variable X follows a Bernoulli distribution with parameter p, written X ~ Bernoulli(p), if:\n\nProbability Mass Function (PMF):\n\n$$P(X = x) = p^x (1-p)^{1-x}, \quad x \in \{0, 1\}$$\n\nEquivalently, we can write this as:\n\n$$P(X = 1) = p \quad \text{and} \quad P(X = 0) = 1 - p = q$$\n\nwhere:\n- p ∈ [0, 1] is the probability of success\n- q = 1 - p is the probability of failure\n- X takes only two values: 0 or 1\n\nThe parameter p completely characterizes the distribution. Once you know p, you know everything about the randomness of a single binary trial.

Moments of the Bernoulli Distribution\n\nThe moments (expected value, variance, etc.) of the Bernoulli distribution are remarkably simple yet carry profound implications for machine learning.\n\nExpected Value (Mean):\n\n$$E[X] = \sum_{x \in \{0,1\}} x \cdot P(X=x) = 0 \cdot (1-p) + 1 \cdot p = p$$\n\nThe mean of a Bernoulli random variable equals the probability of success. This is intuitive: if you flip a biased coin with P(heads) = 0.7, the average outcome is 0.7.\n\nVariance:\n\n$$Var(X) = E[X^2] - (E[X])^2$$\n\nSince X ∈ {0, 1}, we have X² = X, so E[X²] = E[X] = p. Therefore:\n\n$$Var(X) = p - p^2 = p(1-p) = pq$$\n\nStandard Deviation:\n\n$$\sigma = \sqrt{p(1-p)}$$\n\nKey Insight on Variance: The variance pq is maximized when p = 0.5 (giving Var = 0.25) and minimized when p = 0 or p = 1 (giving Var = 0). This reflects the intuition that a fair coin (p = 0.5) has the most uncertainty, while a deterministic outcome (p = 0 or 1) has no uncertainty.

Higher Moments and Moment Generating Function\n\nFor deeper analysis, we can compute the moment generating function (MGF):\n\n$$M_X(t) = E[e^{tX}] = (1-p) \cdot e^{0 \cdot t} + p \cdot e^{1 \cdot t} = (1-p) + pe^t$$\n\nFrom the MGF, we can derive all moments by differentiation:\n\n$$E[X^n] = \frac{d^n}{dt^n} M_X(t) \Big|_{t=0}$$\n\nThe characteristic function (Fourier transform of the PMF) is:\n\n$$\phi_X(t) = E[e^{itX}] = (1-p) + pe^{it}$$\n\nThese functions are essential for proving limit theorems and analyzing sums of random variables.

While the Bernoulli distribution models a single binary trial, the Binomial distribution extends this to model the total number of successes in n independent, identical Bernoulli trials. This is the natural progression: from one coin flip to many flips, from one classification to many classifications.\n\n### Formal Definition\n\nA random variable X follows a Binomial distribution with parameters n and p, written X ~ Binomial(n, p), if X represents the count of successes in n independent Bernoulli(p) trials.\n\nProbability Mass Function (PMF):\n\n$$P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k \in \{0, 1, 2, \ldots, n\}$$\n\nwhere the binomial coefficient is:\n\n$$\binom{n}{k} = \frac{n!}{k!(n-k)!}$$\n\nThis coefficient counts the number of ways to choose which k trials are successes among n trials.

Derivation from First Principles\n\nWhy does the binomial PMF take this form? Consider n = 3 trials with P(success) = p.\n\nFor exactly k = 2 successes, we could have the patterns:\n- SSF (success, success, failure): probability p · p · (1-p) = p²(1-p)\n- SFS: probability p · (1-p) · p = p²(1-p)\n- FSS: probability (1-p) · p · p = p²(1-p)\n\nEach pattern has probability p²(1-p), and there are C(3,2) = 3 such patterns.\n\nTotal probability: P(X = 2) = 3 · p²(1-p)\n\nGeneralizing: P(X = k) = (number of ways to arrange k successes) × (probability of any specific arrangement)\n\n$$P(X = k) = \binom{n}{k} \cdot p^k (1-p)^{n-k}$$

Moments of the Binomial Distribution\n\nSince X = X₁ + X₂ + ... + Xₙ where each Xᵢ ~ Bernoulli(p), we can derive moments from linearity of expectation.\n\nExpected Value:\n\n$$E[X] = E\left[\sum_{i=1}^n X_i\right] = \sum_{i=1}^n E[X_i] = \sum_{i=1}^n p = np$$\n\nVariance:\n\nSince the Xᵢ are independent:\n\n$$Var(X) = Var\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n Var(X_i) = \sum_{i=1}^n p(1-p) = np(1-p) = npq$$\n\nStandard Deviation:\n\n$$\sigma = \sqrt{np(1-p)}$$\n\nMode (most likely value):\n- If (n+1)p is an integer, there are two modes: (n+1)p and (n+1)p - 1\n- Otherwise, the mode is ⌊(n+1)p⌋

Moment Generating Function\n\nThe MGF of the binomial is derived from the MGF of the Bernoulli:\n\n$$M_X(t) = E[e^{tX}] = E\left[e^{t(X_1 + \cdots + X_n)}\right] = \prod_{i=1}^n E[e^{tX_i}] = ((1-p) + pe^t)^n$$\n\nThis elegant form—the n-th power of the Bernoulli MGF—reflects the sum structure of the binomial.\n\n### The Normalization Property\n\nA valid probability distribution must sum to 1. For the binomial:\n\n$$\sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} = (p + (1-p))^n = 1^n = 1 \quad \checkmark$$\n\nThis follows directly from the binomial theorem, which is why the distribution bears this name.

Understanding probability distributions through visualization deepens intuition beyond formulas. Let's examine how Bernoulli and Binomial distributions behave under different parameter settings.\n\n### Bernoulli PMF Visualization\n\nThe Bernoulli PMF is strikingly simple: two bars at x = 0 and x = 1, with heights (1-p) and p respectively.\n\n| p value | P(X=0) | P(X=1) | Visual Pattern |\n|---------|--------|--------|----------------|\n| 0.1 | 0.9 | 0.1 | Heavily favors 0 (failure) |\n| 0.5 | 0.5 | 0.5 | Symmetric (fair coin) |\n| 0.8 | 0.2 | 0.8 | Heavily favors 1 (success) |\n\n### Binomial PMF: Effect of n and p\n\nThe binomial PMF becomes richer as n increases, displaying a bell-shaped curve for moderate p values.

Key Visual Insights\n\nSkewness behavior:\n- When p < 0.5: Distribution is right-skewed (long tail toward high values)\n- When p = 0.5: Distribution is symmetric\n- When p > 0.5: Distribution is left-skewed (long tail toward low values)\n\nConcentration around the mean:\nAs n increases, the distribution becomes more concentrated around np (relative to the range). The coefficient of variation CV = σ/μ = √((1-p)/(np)) decreases as 1/√n, meaning the distribution becomes relatively tighter.\n\nApproaching normality:\nFor large n (typically np ≥ 5 and n(1-p) ≥ 5), the binomial approaches a normal distribution with mean np and variance np(1-p). This is the celebrated Central Limit Theorem in action.

In machine learning, we rarely know the true parameter p; instead, we must estimate it from observed data. This section develops the theory of parameter estimation for Bernoulli and Binomial distributions using Maximum Likelihood Estimation (MLE).\n\n### MLE for Bernoulli Parameter\n\nSuppose we observe n independent samples x₁, x₂, ..., xₙ from a Bernoulli(p) distribution. The likelihood function is the probability of observing this specific data as a function of p:\n\n$$L(p | x_1, \ldots, x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} = p^{\sum x_i} (1-p)^{n - \sum x_i}$$\n\nLet k = Σxᵢ be the total number of successes. Then:\n\n$$L(p) = p^k (1-p)^{n-k}$$\n\n### The Log-Likelihood\n\nMaximizing the likelihood is equivalent to maximizing its logarithm (since log is monotonic):\n\n$$\ell(p) = \log L(p) = k \log p + (n-k) \log(1-p)$$\n\nTo find the maximum, we take the derivative and set it to zero:\n\n$$\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0$$\n\nSolving for p:\n\n$$\frac{k}{p} = \frac{n-k}{1-p}$$\n$$k(1-p) = (n-k)p$$\n$$k - kp = np - kp$$\n$$k = np$$\n$$\hat{p}{MLE} = \frac{k}{n} = \frac{\sum{i=1}^n x_i}{n}$$\n\nThe MLE estimate is simply the sample proportion—the number of successes divided by total trials.

Properties of the MLE Estimator\n\nUnbiasedness:\n$$E[\hat{p}] = E\left[\frac{\sum X_i}{n}\right] = \frac{1}{n} \sum E[X_i] = \frac{1}{n} \cdot np = p$$\n\nThe estimator is unbiased—on average, it equals the true parameter.\n\nVariance:\n$$Var(\hat{p}) = Var\left(\frac{\sum X_i}{n}\right) = \frac{1}{n^2} Var\left(\sum X_i\right) = \frac{1}{n^2} \cdot np(1-p) = \frac{p(1-p)}{n}$$\n\nThe variance decreases as 1/n—more data means more precision.\n\nStandard Error:\n$$SE(\hat{p}) = \sqrt{\frac{p(1-p)}{n}}$$\n\nConsistency:\nAs n → ∞, the variance approaches 0, so p̂ → p in probability. The estimator is consistent.\n\nAsymptotic Normality:\nFor large n:\n$$\hat{p} \stackrel{d}{\rightarrow} N\left(p, \frac{p(1-p)}{n}\right)$$\n\nThis enables confidence intervals and hypothesis testing.

The Bernoulli and Binomial distributions are not isolated; they connect to a rich web of other probability distributions. Understanding these relationships deepens comprehension and reveals when different distributions are appropriate.\n\n### From Bernoulli to Binomial\n\nThe fundamental relationship: if X₁, X₂, ..., Xₙ are independent Bernoulli(p) random variables, then:\n\n$$X = \sum_{i=1}^n X_i \sim \text{Binomial}(n, p)$$\n\nConversely, a Binomial(1, p) is simply a Bernoulli(p).\n\n### Binomial and Poisson (Poisson Limit Theorem)\n\nWhen n is large, p is small, and λ = np is moderate, the binomial approximates the Poisson distribution:\n\n$$\text{Binomial}(n, p) \approx \text{Poisson}(\lambda) \quad \text{where } \lambda = np$$\n\nPrecisely, as n → ∞ and p → 0 with np → λ:\n\n$$\binom{n}{k} p^k (1-p)^{n-k} \rightarrow \frac{\lambda^k e^{-\lambda}}{k!}$$\n\nRule of thumb: Use Poisson when n ≥ 20 and p ≤ 0.05.\n\nExample: If 1 in 1000 emails is spam (p = 0.001) and you receive 500 emails (n = 500), the number of spam emails approximates Poisson(λ = 0.5).

Normal Approximation (De Moivre-Laplace Theorem)\n\nFor large n with np ≥ 5 and n(1-p) ≥ 5:\n\n$$\frac{X - np}{\sqrt{np(1-p)}} \stackrel{d}{\rightarrow} N(0, 1)$$\n\nOr equivalently:\n\n$$X \approx N(np, np(1-p))$$\n\nContinuity Correction:\nSince we're approximating a discrete distribution with a continuous one, we improve accuracy with a continuity correction:\n\n$$P(X \leq k) \approx \Phi\left(\frac{k + 0.5 - np}{\sqrt{np(1-p)}}\right)$$\n\n### Beta-Binomial Connection\n\nIn Bayesian inference, if we place a Beta(α, β) prior on the parameter p of a Binomial, the posterior after observing k successes in n trials is:\n\n$$p | (k, n) \sim \text{Beta}(\alpha + k, \beta + n - k)$$\n\nThis conjugacy is fundamental to Bayesian statistics and leads to the Beta-Binomial compound distribution.

Bernoulli and Binomial distributions permeate machine learning, underpinning fundamental algorithms and evaluation methodologies. Let's explore their key applications in depth.\n\n### Binary Classification and Logistic Regression\n\nLogistic regression models the probability of class membership using a Bernoulli distribution:\n\n$$P(Y = 1 | \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$\n\nGiven the predicted probability p = σ(w^T x + b), the target Y follows:\n\n$$Y | \mathbf{x} \sim \text{Bernoulli}(\sigma(\mathbf{w}^T \mathbf{x} + b))$$\n\nThe negative log-likelihood loss (binary cross-entropy) is:\n\n$$\mathcal{L} = -\sum_{i=1}^n \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]$$\n\nThis is precisely the negative log-likelihood of Bernoulli observations—training logistic regression is MLE for Bernoulli parameters!

Naive Bayes Classification\n\nThe Bernoulli Naive Bayes classifier models features as independent Bernoulli random variables given the class:\n\n$$P(x_j = 1 | y = c) = \theta_{jc}$$\n\nFor binary features (e.g., word presence/absence in text):\n\n$$P(\mathbf{x} | y = c) = \prod_{j=1}^d \theta_{jc}^{x_j} (1 - \theta_{jc})^{1-x_j}$$\n\nClassification uses Bayes' rule:\n\n$$P(y = c | \mathbf{x}) \propto P(y = c) \prod_{j=1}^d P(x_j | y = c)$$\n\n### A/B Testing and Hypothesis Testing\n\nA/B testing compares conversion rates (Bernoulli parameters) between control and treatment groups:\n\n- Control: n₁ users, k₁ conversions → p̂₁ = k₁/n₁\n- Treatment: n₂ users, k₂ conversions → p̂₂ = k₂/n₂\n\nTest statistic:\n$$Z = \frac{\hat{p}2 - \hat{p}1}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}$$\n\nwhere p̂ is the pooled proportion.\n\nSample size calculation for A/B tests:\n$$n \approx \frac{2(z{\alpha/2} + z\beta)^2 \cdot p(1-p)}{(\delta)^2}$$\n\nwhere δ is the minimum detectable effect.

Evaluating Classifier Performance\n\nWhen evaluating a classifier on n test samples, the number of correct predictions X follows a binomial distribution:\n\n$$X \sim \text{Binomial}(n, p_{\text{accuracy}})$$\n\nThe observed accuracy is a/n where a is the number of correct predictions. A 95% confidence interval for true accuracy uses the binomial properties:\n\n- Normal approximation: a/n ± 1.96√(p̂(1-p̂)/n)\n- Clopper-Pearson (exact): Based on relationship to Beta distribution\n- Wilson score: Better coverage for small n or extreme accuracies\n\nThis is crucial for comparing models: "Model A achieved 85% and Model B achieved 83%"—is this difference significant? The binomial framework provides the answer.

While the Bernoulli distribution is computationally trivial—just a coin flip—the binomial involves factorials that can overflow for large n. Understanding computational approaches is essential for practical implementation.\n\n### Computing Binomial Coefficients\n\nNaive calculation of C(n,k) = n!/(k!(n-k)!) fails for large n due to integer overflow. Better approaches:\n\n1. Iterative calculation (avoids factorial overflow):\n$$\binom{n}{k} = \prod_{i=0}^{k-1} \frac{n-i}{i+1}$$\n\n2. Log-space computation:\n$$\log \binom{n}{k} = \sum_{i=0}^{k-1} [\log(n-i) - \log(i+1)]$$\n\nor using the log-gamma function:\n$$\log \binom{n}{k} = \log \Gamma(n+1) - \log \Gamma(k+1) - \log \Gamma(n-k+1)$$

Generating Random Samples\n\nEfficient sampling from binomial distributions is important for Monte Carlo methods.\n\nMethod 1: Sum of Bernoulli trials (simple but slow for large n)\n$$X = \sum_{i=1}^n \text{Bernoulli}(p)$$\n\nMethod 2: Inverse transform (for small n)\nCompute the CDF and use uniform random numbers.\n\nMethod 3: BTPE algorithm (for large n)\nThe Binomial Triangle Parallelogram Exponential (BTPE) algorithm combines multiple techniques for efficiency across all parameter ranges. This is what NumPy and most libraries use.\n\n### Numerical Precision\n\nWhen p is very small or very large, direct probability calculations can underflow. Always:\n\n1. Work in log-space for likelihoods\n2. Use scipy.special.logsumexp for summing probabilities\n3. Use specialized functions like scipy.stats.binom which handle edge cases\n4. For ML training, use numerically stable loss implementations (e.g., torch.nn.BCEWithLogitsLoss)

The Bernoulli and Binomial distributions form the foundation for modeling binary outcomes in machine learning. Let's consolidate the essential concepts:

What's Next:\n\nThe Bernoulli and Binomial handle discrete binary outcomes. In the next page, we explore the Gaussian (Normal) distribution—the cornerstone of continuous probability modeling. We'll see how it arises as the limit of binomials (Central Limit Theorem), why it's ubiquitous in nature and ML, and how it underpins regression, neural networks, and much more.