Proportion of Variance Explained

You've computed principal components for a dataset with 1,000 features. The first component captures the most variance, the second captures the second most, and so on. But a crucial question remains: how many components should you actually keep?

Keeping all 1,000 defeats the purpose of dimensionality reduction. Keeping just one might discard too much information. The answer lies in understanding the proportion of variance explained by each component—a metric that quantifies how much 'information' each principal component captures.

This page develops the mathematical framework for variance explained, shows how to interpret and visualize it, and provides principled approaches for choosing the number of components to retain.

Let's establish the precise definitions and understand what variance explained really measures.

Total Variance

For centered data, the total variance is the sum of all eigenvalues:

$$\text{Total Variance} = \sum_{j=1}^{d} \lambda_j = \text{tr}(\mathbf{S}) = \sum_{j=1}^{d} \text{Var}(X_j)$$

This equals the trace of the covariance matrix, which also equals the sum of individual feature variances. It's a fixed property of the data—the 'total information' we start with.

Variance Explained by Component $j$

Each principal component captures variance equal to its eigenvalue:

$$\text{Variance of PC}_j = \lambda_j$$

The proportion of variance explained by component $j$ is:

$$\text{PVE}j = \frac{\lambda_j}{\sum{k=1}^{d} \lambda_k}$$

This is always between 0 and 1, and all proportions sum to 1:

$$\sum_{j=1}^{d} \text{PVE}_j = 1$$

Cumulative Variance Explained

More practically, we care about how much total variance is explained by the first $k$ components:

$$\text{CVE}k = \sum{j=1}^{k} \text{PVE}j = \frac{\sum{j=1}^{k} \lambda_j}{\sum_{j=1}^{d} \lambda_j}$$

This tells us: if we keep $k$ components, we preserve CVE$_k$ fraction of the total variance.

The cumulative variance sequence is:

• CVE$_1 = $ PVE$_1$
• CVE$_2 = $ PVE$_1 + $ PVE$_2$
• ...
• CVE$_d = 1$ (keeping all components preserves everything)

The distribution of eigenvalues—the eigenvalue spectrum—reveals important information about the data's structure. Different patterns have different implications for dimensionality reduction.

Pattern 1: Dominant Eigenvalue(s)

If $\lambda_1 \gg \lambda_2 \gg \cdots \gg \lambda_d$:

• The data is approximately low-dimensional
• A small number of components captures most variance
• Strong correlations exist between original features
• Dimensionality reduction is highly effective

Example: Stock returns often show this pattern—most variance is explained by market-wide factors.

Pattern 2: Slow Decay

If eigenvalues decrease gradually: $\lambda_1 \approx \lambda_2 \approx \cdots \approx \lambda_d$:

• Data is truly high-dimensional
• Many components needed to capture most variance
• Features are relatively independent
• Dimensionality reduction has limited benefit

Example: Well-designed experiments with orthogonal factors show this pattern.

Pattern 3: Cliff Edge

If eigenvalues are large for some components then suddenly drop:

• Data lies near a lower-dimensional subspace
• The 'cliff' indicates the true dimensionality
• Components after the cliff capture noise
• Clear choice of $k$ (components before the cliff)

Example: PCA on noisy observations of a truly low-dimensional phenomenon.

There is no single 'correct' number of components—the right choice depends on your application, goals, and constraints. Here are the main approaches:

Method 1: Variance Threshold

Choose $k$ such that CVE$_k \geq \tau$ for some threshold $\tau$:

$$k^* = \min\{k : \text{CVE}_k \geq \tau\}$$

Common thresholds:

• $\tau = 0.80$: Keep components explaining 80% of variance
• $\tau = 0.90$: Keep components explaining 90% of variance
• $\tau = 0.95$: Keep components explaining 95% of variance

Pros: Simple, interpretable, provides a guarantee on information preserved Cons: Arbitrary threshold choice, may not align with downstream task needs

Method 2: Elbow Method (Scree Plot Analysis)

Plot eigenvalues or PVE against component index. Look for an 'elbow'—a point where the rate of decrease changes sharply.

Procedure:

1. Plot $\lambda_j$ vs $j$ (or PVE$_j$ vs $j$)
2. Identify where the curve 'bends' from steep to flat
3. Choose $k$ at or before the elbow

Pros: Data-driven, captures structure in eigenvalue decay Cons: Subjective (no unique definition of 'elbow'), may be ambiguous

Method 3: Kaiser Criterion

Keep components with eigenvalue greater than the average:

$$k^* = |\{j : \lambda_j > \bar{\lambda}\}|$$

For standardized data (each feature has variance 1), $\bar{\lambda} = 1$, so this becomes: keep components with $\lambda_j > 1$.

Intuition: A component with $\lambda_j = 1$ explains the same variance as a single original (standardized) feature. If it explains less, why bother?

Pros: Objective, no parameter to choose Cons: Can over- or under-estimate, not connected to downstream task

Method 4: Cross-Validation

Use the reconstruction error on held-out data to choose $k$:

1. Split data into train and validation sets
2. Fit PCA with $k$ components on training data
3. Compute reconstruction error on validation data
4. Choose $k$ that minimizes validation reconstruction error (plus complexity penalty if desired)

This is the most principled approach when reconstruction accuracy matters.

Method 5: Downstream Task Performance

If PCA is a preprocessing step for classification, regression, or clustering:

1. Fit PCA with various $k$ values
2. Train downstream model on PCA-transformed data
3. Evaluate downstream model on validation set
4. Choose $k$ that optimizes downstream performance

This directly optimizes what you care about.

Let's develop a deeper mathematical understanding of how variance is distributed across components.

Effective Dimensionality

The eigenvalue distribution contains information about the data's 'effective' dimensionality. One formal measure is the participation ratio:

$$d_{\text{eff}} = \frac{\left(\sum_{j=1}^{d} \lambda_j\right)^2}{\sum_{j=1}^{d} \lambda_j^2}$$

This quantity:

• Equals $d$ when all eigenvalues are equal (truly $d$-dimensional)
• Equals 1 when only one eigenvalue is non-zero (1-dimensional)
• Generally falls between: extent of variance concentration

Interpretation: $d_{\text{eff}}$ estimates how many 'effective' dimensions the data has, accounting for unequal importance of components.

Entropy-Based Measure

Another measure treats the normalized eigenvalues as a probability distribution and computes entropy:

$$p_j = \frac{\lambda_j}{\sum_k \lambda_k}$$

$$H = -\sum_j p_j \log p_j$$

The effective dimensionality is then $d_{\text{entropy}} = e^H$.

• Maximum entropy $H = \log d$ yields $d_{\text{entropy}} = d$
• Minimum entropy (single dominant component) yields $d_{\text{entropy}} \approx 1$

This has information-theoretic motivation: it measures the 'surprise' in encountering variance in different directions.

Bounds on Cumulative Variance

For the first $k$ components, we can establish bounds and approximations:

Upper bound: CVE$_k \leq k/d$ with equality when eigenvalues are equal.

Lower bound: If $\lambda_1 \geq \lambda_2 \geq \cdots$, then:

$$\text{CVE}_k \geq \frac{k \lambda_k}{\sum_j \lambda_j}$$

Decay rate connection: If eigenvalues follow power law $\lambda_j \sim j^{-\alpha}$:

$$\text{CVE}_k \approx \begin{cases} 1 - (k/d)^{1-\alpha} & \alpha < 1 \\ \log(k)/\log(d) & \alpha = 1 \\ (k/d)^{1-1/\alpha} & \alpha > 1 \end{cases}$$

These asymptotics help predict how many components are needed for a given variance target.

Understanding variance explained in context is crucial for making good decisions. Here are guidelines for interpreting these metrics in practice.

Context Matters

A given PVE value means different things in different contexts:

• Image data: 80% variance in 10 components (from 10,000 pixels) is impressive
• Survey data: 80% variance in 10 components (from 20 questions) is less impressive
• Genomics: 50% variance in 5 components (from 20,000 genes) is remarkable

The ratio of kept components to original features provides context for interpreting variance explained.

Variance Explained ≠ Information Preserved

Critical caveat: variance is not the same as task-relevant information.

Example: In a classification task, the class-discriminative information might be in the low-variance components. High-variance components might capture within-class variation that's irrelevant for classification.

PCA optimizes for variance, not for any specific downstream task. A component with 1% variance might be more important for your task than one with 30%.

While variance explained is the most common criterion, alternative approaches may be more appropriate for specific applications.

Reconstruction Error on Held-Out Data

Instead of training-set variance, measure reconstruction error on validation data:

$$\mathcal{E}{\text{val}}(k) = \frac{1}{n{\text{val}}} \sum_{i \in \text{val}} \|\mathbf{x}^{(i)} - \hat{\mathbf{x}}^{(i)}\|^2$$

This accounts for overfitting: more components always reduce training error but may not generalize.

Key insight: Very small eigenvalue components may capture noise rather than signal. On held-out data, reconstructing noise hurts.

Information Criteria

Model selection criteria like AIC and BIC can be applied:

$$\text{AIC}(k) = n \log(\mathcal{E}(k)) + 2 \cdot \text{params}(k)$$ $$\text{BIC}(k) = n \log(\mathcal{E}(k)) + \log(n) \cdot \text{params}(k)$$

where params$(k)$ is the effective number of parameters with $k$ components.

BIC typically selects fewer components (stronger complexity penalty), suitable when parsimony is valued.

Parallel Analysis (Random Matrix Theory)

Compare eigenvalues to those from random data with the same dimensions:

1. Generate random data of same size $(n \times d)$
2. Compute its eigenvalues
3. Repeat multiple times to get null distribution
4. Keep components with eigenvalues exceeding the 95th percentile of random eigenvalues

This accounts for 'spurious' eigenvalues that arise from finite sample effects.

Effective visualization is essential for understanding variance distribution and communicating results. Here are the key visualization types.

The Scree Plot

The most common visualization: eigenvalues (or PVE) plotted against component index.

Construction:

• X-axis: Component index ($j = 1, 2, \ldots, d$)
• Y-axis: Eigenvalue $\lambda_j$ (or PVE$_j$)
• Optional: Add horizontal line at mean eigenvalue (Kaiser criterion)
• Optional: Mark the elbow point

Interpretation: The 'elbow' indicates where additional components contribute diminishing returns. Components before the elbow capture structure; components after may capture noise.

Cumulative Variance Plot

Shows CVE$_k$ against $k$, revealing how variance accumulates.

Construction:

• X-axis: Number of components ($k$)
• Y-axis: Cumulative variance explained (0 to 1 or 0% to 100%)
• Optional: Add horizontal lines at common thresholds (80%, 90%, 95%)

Reading the plot: Steep initial rise indicates concentrated variance; gradual rise indicates distributed variance. Intersections with threshold lines indicate how many components needed.

Combined Dashboard

Best practice: show both individual and cumulative variance.