Generative Model Fundamentals

The observable world is complex, high-dimensional, and seemingly chaotic. Yet underlying this complexity, we often find simple structures: faces vary along dimensions of age, expression, and pose; music varies in tempo, key, and mood; physical systems evolve according to low-dimensional laws.

Latent variable models formalize this intuition. We posit that observed data $x$ is generated from hidden (latent) variables $z$ that capture the underlying structure. These latent variables are not directly observed—they must be inferred from data—but they can dramatically simplify our understanding and manipulation of complex distributions.

This conceptual framework is foundational for modern generative models. Variational autoencoders, factor analysis, mixture models, and representation learning all rest on latent variable foundations. Understanding this framework deeply is essential for mastering modern generative AI.

The Generative Story:

A latent variable model posits that observed data $x$ is generated by:

1. Sample latent variable from prior: $z \sim p(z)$
2. Generate observation from conditional: $x \sim p(x | z)$

The joint distribution decomposes as: $$p(x, z) = p(z) \cdot p(x | z)$$

The Marginal Distribution:

The distribution over observations (marginal likelihood) is obtained by integrating out the latent: $$p(x) = \int p(x, z) , dz = \int p(z) \cdot p(x | z) , dz$$

This integral is typically intractable for interesting models, which is the source of most computational challenges.

Key Terminology:

• Prior: $p(z)$ — Our beliefs about latent variables before seeing data. Often simple (e.g., standard Gaussian).
• Likelihood (Decoder): $p(x | z)$ — The observation model. Given latent state, how is data generated?
• Posterior: $p(z | x)$ — What we learn about latent variables given observed data. Found via Bayes' rule: $$p(z | x) = \frac{p(x | z) p(z)}{p(x)}$$
• Evidence (Marginal likelihood): $p(x)$ — The denominator in Bayes' rule, typically intractable.

Why Latent Variables Create Expressiveness:

Even if $p(z)$ and $p(x|z)$ are simple (e.g., Gaussian), the marginal $p(x)$ can be arbitrarily complex.

Example: Gaussian Mixture Model

Let $z$ be discrete, indicating cluster membership:

• Prior: $p(z = k) = \pi_k$ (categorical)
• Likelihood: $p(x | z = k) = \mathcal{N}(x | \mu_k, \Sigma_k)$ (Gaussian per cluster)

Marginal: $$p(x) = \sum_{k=1}^K \pi_k \cdot \mathcal{N}(x | \mu_k, \Sigma_k)$$

A sum of Gaussians can approximate any continuous distribution given enough components. The simple structure (discrete $z$, Gaussian conditionals) yields a universal approximator.

Example: Continuous Latent + Nonlinear Decoder

Let $z \in \mathbb{R}^d$ be continuous:

• Prior: $p(z) = \mathcal{N}(0, I)$
• Likelihood: $p(x | z) = \mathcal{N}(f_\theta(z), \sigma^2 I)$ where $f_\theta$ is a neural network

The marginal $p(x)$ is a Gaussian convolved with the pushforward of the prior through $f$. For a complex $f$, this can model intricate distributions. This is the VAE setup.

While latent variable models are conceptually elegant, they present fundamental computational challenges. The core issue: computing the marginal likelihood $p(x)$ and the posterior $p(z|x)$ is typically intractable.

Why is $p(x)$ intractable?

$$p(x) = \int p(z) p(x|z) , dz$$

This integral over all possible latent configurations is:

• Continuous $z$: An integral over high-dimensional space (no closed form for nonlinear decoders)
• Discrete $z$: A sum over exponentially many configurations

Why is $p(z|x)$ intractable?

$$p(z|x) = \frac{p(x|z) p(z)}{p(x)}$$

We need $p(x)$ in the denominator—which is intractable!

Why does this matter?

1. Learning: We want to maximize $\log p(x)$ (likelihood of data). But we can't compute it!
2. Inference: We want $p(z|x)$ to understand what latent factors explain an observation. But we can't compute that either!
3. Generation: We want to sample $x \sim p(x)$. While we can sample $z$ then $x|z$, we can't verify the resulting marginal matches the target.

Solutions to Intractability:

1. Exact Methods (Limited Cases)

• Factor Analysis / PCA: Linear Gaussian models have closed-form solutions
• GMM: Discrete $z$ with finite states allows exact enumeration
• Exponential Families with Conjugacy: Posterior is same family as prior

2. Sampling Methods (MCMC)

• Sample from $p(z|x)$ using MCMC (Gibbs, MH)
• Approximate expectations via Monte Carlo
• Can be exact in principle, slow in practice

3. Variational Inference

• Approximate $p(z|x)$ with a tractable distribution $q_\phi(z|x)$
• Optimize $q$ to be close to true posterior
• Fast and scalable, but introduces approximation error
• This is how VAEs work

4. Importance Weighting

• Use importance sampling to estimate $p(x)$
• High variance in high dimensions
• IWAE (Importance-Weighted Autoencoder) uses this

5. Neural Likelihood Surrogates

• Train another network to approximate $p(x)$ or its gradients
• Score matching, noise-contrastive estimation

The latent space is the abstract space where latent variables $z$ live. Understanding its geometry is crucial for manipulating and interpreting generative models.

The Manifold Hypothesis:

High-dimensional data (images, text, audio) is believed to lie on or near low-dimensional manifolds. For example:

• All face images might lie on a ~50-dimensional manifold within million-dimensional pixel space
• The manifold encodes meaningful variations: pose, lighting, expression, identity

Latent variable models formalize this: $z$ parameterizes the manifold, and the decoder $p(x|z)$ maps from manifold coordinates to observations.

Dimensionality Reduction:

If the intrinsic dimensionality of data is $d$, we hope to learn latent spaces of dimension $d$ or slightly higher. This:

• Compresses information (dimensionality reduction)
• Reveals interpretable structure
• Enables efficient sampling and manipulation

Latent Traversals:

A key test of whether a model has learned meaningful structure: traverse the latent space and observe how generations change.

In a well-trained model:

• Moving along one latent dimension might change hair color
• Another dimension might change pose
• Linear interpolation in latent space produces semantically smooth interpolation in data space

If traversals produce random, discontinuous changes, the latent space lacks structure.

Latent Arithmetic:

A remarkable property of well-trained latent spaces: semantic concepts become directions.

Famous example from word embeddings: $$\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$$

Similar operations work in image latent spaces: $$\vec{face_smiling} - \vec{face_neutral} + \vec{another_neutral} = \vec{another_smiling}$$

This enables powerful editing: compute the 'smile direction' by averaging differences between smiling and neutral faces, then add this direction to any face to make it smile.

Holes in Latent Space:

Not all regions of latent space correspond to valid data. If the prior is Gaussian and the model isn't well-regularized, the decoder may only map properly from regions where training samples exist. This creates 'holes'—regions that produce garbage outputs.

VAEs address this by explicitly matching the posterior to the prior, ensuring the entire prior region is 'populated' with meaningful encodings.

Before deep learning, several classical latent variable models were developed. Understanding these provides essential intuition and reveals the lineage of modern approaches.

Principal Component Analysis (PCA)

PCA can be viewed as a probabilistic latent variable model:

$$z \sim \mathcal{N}(0, I_k)$$ $$x | z \sim \mathcal{N}(Wz + \mu, \sigma^2 I_d)$$

• $k$-dimensional latent space, $d$-dimensional observations
• Linear decoder: $x \approx Wz + \mu$
• $W$ are the principal components (directions of maximum variance)

The posterior is Gaussian: $$p(z | x) = \mathcal{N}(M^{-1} W^T (x - \mu), \sigma^2 M^{-1})$$ where $M = W^T W + \sigma^2 I$.

As $\sigma^2 \to 0$, this recovers deterministic PCA.

Factor Analysis

Generalization of probabilistic PCA with non-isotropic noise:

$$x | z \sim \mathcal{N}(Wz + \mu, \Psi)$$

where $\Psi$ is a diagonal covariance (each observed dimension has its own noise level).

Used in psychology and social sciences to model latent 'factors' (intelligence, personality traits) from observed measurements.

Gaussian Mixture Models (GMM)

Discrete latent variable (cluster indicator):

$$z \sim \text{Categorical}(\pi_1, \ldots, \pi_K)$$ $$x | z = k \sim \mathcal{N}(\mu_k, \Sigma_k)$$

The posterior is: $$p(z = k | x) = \frac{\pi_k \mathcal{N}(x | \mu_k, \Sigma_k)}{\sum_j \pi_j \mathcal{N}(x | \mu_j, \Sigma_j)}$$

This is the 'responsibility' of cluster $k$ for observation $x$.

EM algorithm alternates:

• E-step: Compute responsibilities (posterior)
• M-step: Update parameters given responsibilities

Hidden Markov Models (HMM)

Latent variable model for sequences:

$$z_1 \sim \pi, \quad z_t | z_{t-1} \sim A_{z_{t-1}}$$ $$x_t | z_t \sim p(x | z_t)$$

The latent variables $z_t$ form a Markov chain; observations $x_t$ depend only on the current state.

Inference (forward-backward algorithm) and learning (Baum-Welch EM) are tractable due to the chain structure.

HMMs dominated speech recognition for decades before deep learning.

The deep learning revolution enabled deep latent variable models—models with neural network decoders (and often encoders) that can capture arbitrarily complex relationships between latents and observations.

The Setup:

$$z \sim p(z) = \mathcal{N}(0, I)$$ $$x | z \sim p_\theta(x | z) = \mathcal{N}(f_\theta(z), \sigma^2 I)$$

where $f_\theta$ is a neural network (the decoder).

The Challenge:

With nonlinear $f_\theta$, the marginal and posterior become intractable:

$$p(x) = \int p(z) p_\theta(x|z) dz \quad \text{(no closed form)}$$ $$p(z|x) = \frac{p_\theta(x|z) p(z)}{p(x)} \quad \text{(requires intractable } p(x) \text{)}$$

The Variational Solution (VAE):

Introduce an encoder $q_\phi(z|x)$—a neural network that approximates the posterior:

• Encoder: $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma^2_\phi(x))$
• Decoder: $p_\theta(x|z) = \mathcal{N}(f_\theta(z), \sigma^2 I)$

Optimize the Evidence Lower Bound (ELBO):

$$\log p(x) \geq \mathbb{E}{q\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) | p(z))$$

The first term encourages accurate reconstruction; the second regularizes the posterior to match the prior.

Why Deep Latent Models Are Powerful:

1. Expressiveness: Neural network decoders can model arbitrarily complex conditionals $p(x|z)$.

2. Scalability: Amortized inference enables processing millions of data points.

3. Representation Learning: The latent space often captures meaningful structure useful for downstream tasks.

4. Generative Capabilities: Sample $z \sim p(z)$, decode to $x$—create new data.

The VAE Trade-off:

VAEs optimize a lower bound, not the true likelihood. The gap:

$$\log p(x) - \text{ELBO} = D_{KL}(q_\phi(z|x) | p(z|x)) \geq 0$$

This gap is zero only when $q = p(z|x)$—when the approximate posterior is exact. In practice, the approximation gap can be significant, especially when $q$ is restricted (e.g., Gaussian with diagonal covariance).

Extensions:

• Hierarchical VAEs: Multiple layers of latent variables for richer structure
• VQ-VAE: Discrete (quantized) latent space
• β-VAE: Weighted KL term to encourage disentanglement
• Diffusion models: Can be viewed as hierarchical latent models with many layers

Computing or approximating the posterior $p(z|x)$ is called inference. Different approaches trade off accuracy, computation, and generality.

Exact Inference:

Possible only for specific model families:

• Conjugate priors: Posterior is same family as prior (Gaussian-Gaussian, Beta-Binomial)
• Discrete latents with few states: Enumerate all possibilities
• Linear Gaussian models: Closed-form Kalman filter, PCA

For deep generative models, exact inference is essentially never possible.

Variational Inference (VI):

Approximate $p(z|x)$ with a parameterized family $q_\phi(z|x)$, optimizing:

$$\phi^* = \arg\min_\phi D_{KL}(q_\phi(z|x) | p(z|x))$$

Equivalently, maximize the ELBO:

$$\mathcal{L}(\phi) = \mathbb{E}{q\phi(z|x)}[\log p(x|z)] - D_{KL}(q_\phi(z|x) | p(z))$$

VI characteristics:

• Fast (gradient-based optimization)
• Biased (approximation gap)
• Mode-seeking (tends to underfit posterior variance)
• Enables end-to-end learning via reparameterization

MCMC Inference:

Construct a Markov chain with stationary distribution $p(z|x)$:

• Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo
• Asymptotically exact (converges to true posterior)
• Often slow (many iterations for convergence)
• Harder to integrate with gradient-based learning

Hybrid Approaches:

Modern methods often combine VI and MCMC:

• MCMC-VI: Use MCMC to refine VI solutions
• Hamiltonian VI: Use HMC dynamics within variational family
• Normalizing Flow Posteriors: Use flows to parameterize flexible $q$

Importance-Weighted Inference:

Importance-weighted autoencoders (IWAE) use multiple samples:

$$\log p(x) \geq \mathbb{E}{z_1, \ldots, z_K \sim q}\left[\log \frac{1}{K} \sum{k=1}^K \frac{p(x, z_k)}{q(z_k|x)}\right]$$

Tighter bound than ELBO; approaches $\log p(x)$ as $K \to \infty$. But gradient variance increases with $K$.

Beyond generation, latent variable models are powerful tools for representation learning—learning useful features for downstream tasks.

Why Latent Representations?

The latent encoding $z$ of an observation $x$ can serve as a feature vector:

• Dimensionality reduction: $z$ is typically much lower-dimensional than $x$
• Semantic structure: Nearby points in latent space often have similar semantics
• Transfer learning: Representations learned on one task can help others

Self-Supervised Learning Connection:

Latent variable models are fundamentally self-supervised:

• No labels required—learn from data structure alone
• The 'supervision signal' is reconstruction (predict $x$ from $z$)

This connects to contrastive learning, masked modeling, and other self-supervised paradigms.

Evaluating Representations:

How do we know if learned representations are good?

1. Linear probing: Train a linear classifier on frozen representations. High accuracy = representations capture relevant structure.

2. Transfer learning: Use representations for downstream tasks. Good representations transfer broadly.

3. Latent space visualization: Do similar points cluster together? Are trajectories meaningful?

4. Disentanglement metrics: Do dimensions correspond to independent factors?

Representation Learning Applications:

• Feature extraction: Encode images/text into fixed-size vectors for search, clustering
• Preprocessing for supervised learning: Pre-train on unlabeled data, finetune on labels
• Anomaly detection: Points with unusual latent encodings may be anomalous
• Data compression: Encode to latent, transmit latent, decode at receiver
• Controllable generation: Manipulate latent dimensions to control generated outputs

The Foundation Model Paradigm:

Modern 'foundation models' (GPT, CLIP, DALL-E) can be viewed through this lens:

• Train a large generative model on massive unlabeled data
• The model learns rich representations as a byproduct
• Finetune or prompt for specific downstream tasks

The latent variable framework helps us understand why this works: learning to generate forces the model to capture structure useful for many tasks.

Latent variables provide a powerful framework for understanding and building generative models. By positing hidden structure that explains observable data, we can create expressive models, learn meaningful representations, and enable controlled generation.

What's next:

We've established the foundation of generative models: the generative-discriminative distinction, density estimation, sampling, and latent variables. The final page of this module addresses a critical practical challenge: how do we evaluate generative models? Unlike classification, where accuracy is clear, evaluating generative quality is notoriously difficult. We'll explore the landscape of evaluation metrics and their limitations.