Given a joint distribution over multiple variables, we often need to answer questions about a single variable, ignoring all others. If we know $P(X, Y)$ completely, what is the distribution of $X$ alone?

This operation—marginalization—is one of the most fundamental tools in probability theory. The name comes from the fact that in tabular representations, summing across rows or columns places the result in the "margin" of the table.

The Core Question: Given complete knowledge of how $(X, Y)$ behave together, how do we extract the distribution of $X$ by itself, effectively "forgetting" about $Y$?

Definition (Marginal PMF):

For discrete random variables $X$ and $Y$ with joint PMF $p_{X,Y}(x, y)$, the marginal PMF of $X$ is obtained by summing over all possible values of $Y$:

$$p_X(x) = \sum_{y} p_{X,Y}(x, y) = \sum_{y} P(X = x, Y = y)$$

Similarly, the marginal PMF of $Y$ is:

$$p_Y(y) = \sum_{x} p_{X,Y}(x, y)$$

Intuition: The event ${X = x}$ can occur with any value of $Y$. By the axiom of probability applied to a partition, we sum over all mutually exclusive ways this can happen.

In the table above:

• Each row sum gives $p_X(x)$ — the marginal distribution of $X$
• Each column sum gives $p_Y(y)$ — the marginal distribution of $Y$
• The corner (1.00) confirms the joint sums to 1

Key Observation: Knowing only the marginals does NOT let you recover the joint. Many different joint distributions can produce the same marginals.

Definition (Marginal PDF):

For continuous random variables with joint PDF $f_{X,Y}(x, y)$, the marginal PDF of $X$ is obtained by integrating out $Y$:

$$f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y) , dy$$

Similarly for $Y$:

$$f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x, y) , dx$$

Intuition: Integration replaces summation when moving from discrete to continuous. We "integrate out" the variable we want to ignore.

Example: Uniform on the Unit Square

For $(X, Y)$ uniform on $[0, 1] \times [0, 1]$ with $f_{X,Y}(x, y) = 1$:

$$f_X(x) = \int_0^1 1 , dy = 1 \quad \text{for } x \in [0, 1]$$

So $X \sim \text{Uniform}(0, 1)$. Similarly $Y \sim \text{Uniform}(0, 1)$.

Example: Bivariate Gaussian Marginals

For a bivariate Gaussian $(X, Y)$ with means $\mu_X, \mu_Y$, variances $\sigma_X^2, \sigma_Y^2$, and correlation $\rho$:

$$f_X(x) = \frac{1}{\sqrt{2\pi}\sigma_X} \exp\left(-\frac{(x - \mu_X)^2}{2\sigma_X^2}\right)$$

The marginal is Gaussian with the same mean and variance—the correlation $\rho$ disappears completely. This is a key property: marginals don't capture correlation.

Marginalization is a direct application of the Law of Total Probability.

Law of Total Probability:

If ${B_i}$ is a partition of the sample space (mutually exclusive, exhaustive events), then for any event $A$:

$$P(A) = \sum_i P(A \cap B_i) = \sum_i P(A | B_i) P(B_i)$$

Applied to Random Variables:

The possible values of $Y$ partition the sample space. Therefore:

$$P(X = x) = \sum_y P(X = x, Y = y)$$

This is exactly the marginalization formula. In the continuous case, the sum becomes an integral.

Why This Matters:

This connection reveals that marginalization isn't an arbitrary operation—it's logically necessary from the axioms of probability. It's the unique way to derive individual distributions from joint distributions that remains consistent with probability theory.

For random vectors $\mathbf{X} = (X_1, X_2, \ldots, X_d)$, we can marginalize over any subset of variables.

General Formula (Continuous):

To get the marginal of $(X_1, X_2)$ from the joint of $(X_1, X_2, \ldots, X_d)$:

$$f_{X_1, X_2}(x_1, x_2) = \int \cdots \int f_{X_1, \ldots, X_d}(x_1, \ldots, x_d) , dx_3 \cdots dx_d$$

Multivariate Gaussian Case:

For a Gaussian $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, the marginal of any subset is also Gaussian with the corresponding sub-vector of $\boldsymbol{\mu}$ and sub-matrix of $\boldsymbol{\Sigma}$.

If $\mathbf{X} = \begin{pmatrix} \mathbf{X}A \ \mathbf{X}B \end{pmatrix}$ with $\boldsymbol{\mu} = \begin{pmatrix} \boldsymbol{\mu}A \ \boldsymbol{\mu}B \end{pmatrix}$ and $\boldsymbol{\Sigma} = \begin{pmatrix} \boldsymbol{\Sigma}{AA} & \boldsymbol{\Sigma}{AB} \ \boldsymbol{\Sigma}{BA} & \boldsymbol{\Sigma}{BB} \end{pmatrix}$

Then: $\mathbf{X}_A \sim \mathcal{N}(\boldsymbol{\mu}A, \boldsymbol{\Sigma}{AA})$

No integration required—just extract the relevant blocks!

Marginalization appears throughout ML, often in computational challenges.

1. Computing Marginal Likelihood (Model Evidence)

In Bayesian inference, the marginal likelihood integrates over all parameter values:

$$P(\mathbf{X}) = \int P(\mathbf{X} | \boldsymbol{\theta}) P(\boldsymbol{\theta}) , d\boldsymbol{\theta}$$

This integral is often intractable, motivating approximations like MCMC and variational inference.

2. Latent Variable Models

Models like GMMs, VAEs, and HMMs have latent variables $\mathbf{Z}$. The observed likelihood marginalizes over latents:

$$P(\mathbf{X}) = \sum_{\mathbf{z}} P(\mathbf{X}, \mathbf{Z} = \mathbf{z}) = \sum_{\mathbf{z}} P(\mathbf{X} | \mathbf{Z}) P(\mathbf{Z})$$

3. Graphical Models

Variable elimination in Bayesian networks is systematic marginalization—removing variables one by one to answer queries about remaining variables.

What's next:

While marginalization lets us ignore variables, often we want to use information about one variable to refine our knowledge of another. The next page covers conditional distributions—the mathematical foundation for prediction in machine learning.