Scree Plots

The term "scree" comes from geology—it refers to the loose rocks that accumulate at the base of a cliff. In PCA, the scree plot got its name because its shape often resembles a cliff face with scree at the bottom: steep descent followed by a flat tail of 'rubble.'

The scree plot is arguably the most important diagnostic tool in PCA. It visualizes the eigenvalue spectrum, revealing patterns that determine how many components to retain. A quick glance at a scree plot tells experienced practitioners whether PCA will be effective, how many components make sense, and whether the data has unusual structure.

This page develops a deep understanding of scree plots—from construction fundamentals to sophisticated interpretation techniques and automatic elbow detection algorithms.

Let's establish the fundamentals of scree plot construction, including different variants and their respective uses.

Basic Scree Plot

The standard scree plot visualizes eigenvalues against component index:

• X-axis: Component number ($j = 1, 2, \ldots, d$)
• Y-axis: Eigenvalue $\lambda_j$
• Plot elements: Points connected by lines, showing the decay pattern

The eigenvalues are always plotted in descending order: $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_d$.

Proportion Variant

Instead of raw eigenvalues, plot the proportion of variance explained:

$$\text{PVE}j = \frac{\lambda_j}{\sum{k=1}^{d} \lambda_k}$$

This normalizes the plot to the [0, 1] range, making it easier to interpret and compare across datasets.

Log-Scale Variant

For data with eigenvalues spanning many orders of magnitude, a log-scale y-axis is essential:

• Y-axis: $\log(\lambda_j)$ or use logarithmic scale
• Reveals patterns invisible in linear scale
• Power-law decay appears as a straight line

When to use log scale:

• When $\lambda_1 / \lambda_d > 100$
• When you want to detect power-law structure
• When small eigenvalues matter and would otherwise be invisible

Combined Scree-Cumulative Plot

A highly informative variant shows both individual and cumulative variance:

• Bar chart: Individual PVE$_j$ for each component
• Line overlay: Cumulative variance CVE$k = \sum{j=1}^{k}$ PVE$_j$
• Dual axes: Left for individual, right for cumulative (or same scale 0-100%)

Different data structures produce characteristic scree plot shapes. Learning to read these patterns is a valuable diagnostic skill.

The Classic Elbow

The ideal scree plot shows a clear elbow—a sharp bend from steep descent to flat plateau:

|*
|  *
|    *
|      * * * * * * * * *
+----------------------→

Interpretation:

• Components before the elbow capture structure
• Components after the elbow capture noise
• The elbow location indicates optimal $k$

Typical sources: Signal-plus-noise models, low-rank structure corrupted by measurement error.

Gradual Decay (No Clear Elbow)

|*
| *
|  *
|   *
|    *
|     *
|      *
+----------------------→

Interpretation:

• No obvious cutoff point
• Data is genuinely high-dimensional or has complex structure
• PCA may not provide dramatic dimensionality reduction

Typical sources: Well-designed experiments, truly independent features, complex natural phenomena.

Multiple Elbows

|*
|  *
|    * *
|        *
|          * * *
|                * * * *
+----------------------→

Interpretation:

• Hierarchical or multi-scale structure
• Different groups of components serve different purposes
• May indicate multiple underlying factors at different scales

Typical sources: Multi-resolution data, nested structure, mixtures of processes.

The 'elbow method' is widely cited but often poorly understood. Let's dissect what we're actually looking for and why.

What is an Elbow?

Mathematically, an elbow occurs where the second derivative of the eigenvalue sequence changes sign or magnitude significantly. It represents a transition between two different decay regimes:

1. Fast decay regime (before elbow): Each component explains substantially more than the next
2. Slow decay regime (after elbow): Components explain similar (small) amounts

The elbow is where marginal value drops sharply.

Why Elbows Exist

Signal-plus-noise model: If true data lies near a $k$-dimensional subspace but is observed with noise:

$$\mathbf{x}{\text{observed}} = \mathbf{x}{\text{signal}} + \boldsymbol{\epsilon}_{\text{noise}}$$

The eigenvalues will be:

• $\lambda_1, \ldots, \lambda_k$: Large (signal variances + noise)
• $\lambda_{k+1}, \ldots, \lambda_d$: Small and similar (noise only)

The elbow occurs at $k$, the true signal dimensionality.

Why the elbow might be fuzzy:

• Noise level varies across components
• Signal eigenvalues themselves decay
• True dimensionality isn't integer-valued
• Non-linear structure doesn't fit the model

Guidelines for Visual Elbow Detection

1. Look at multiple scales: Check both linear and log scales
2. Consider the context: Prior knowledge about expected dimensionality
3. Examine both eigenvalues and cumulative variance: They may suggest different elbows
4. Be skeptical of 'obvious' elbows: They may be artifacts of visualization choices
5. When in doubt, try multiple values: See how downstream analysis changes

To remove subjectivity, we can use algorithmic methods for elbow detection. Several approaches have been developed, each with different assumptions.

Method 1: Maximum Curvature

Find the point of maximum curvature in the eigenvalue curve. For discrete data, approximate curvature as:

$$\kappa_j \approx |\lambda_{j-1} - 2\lambda_j + \lambda_{j+1}|$$

The elbow is at $k^* = \arg\max_j \kappa_j$.

Pros: Intuitive, matches visual perception Cons: Sensitive to noise in eigenvalues, requires smoothing

Method 2: Perpendicular Distance (Kneedle Algorithm)

Draw a line from the first to the last point. Find the point with maximum perpendicular distance from this line:

1. Normalize eigenvalues to [0, 1] range
2. Draw line from $(1, \lambda_1)$ to $(d, \lambda_d)$
3. Compute perpendicular distance from each point to the line
4. Elbow is the point with maximum distance

Pros: Robust, handles various decay patterns Cons: Depends on normalization, may miss elbows at edges

Method 3: Log-Ratio Analysis

For power-law decays, analyze the ratio of consecutive eigenvalues:

$$r_j = \frac{\lambda_j}{\lambda_{j+1}}$$

In a pure power-law decay, $r_j$ is constant. The elbow is where $r_j$ changes significantly:

$$k^* = \arg\max_j |r_j - r_{j+1}|$$

Pros: Scale-invariant, good for power-law data Cons: Assumes specific decay structure

Method 4: Broken Stick Model

Compare eigenvalues to those expected if total variance were randomly distributed. Under the broken stick model, the expected proportion for component $j$ is:

$$b_j = \frac{1}{d} \sum_{k=j}^{d} \frac{1}{k}$$

Keep components where observed PVE exceeds this null expectation.

Pros: Provides statistical baseline Cons: Very conservative, may underestimate $k$

A fundamentally different approach asks: which eigenvalues are statistically distinguishable from noise? This connects scree plot interpretation to hypothesis testing.

The Random Matrix Theory Perspective

Even for completely random data, eigenvalues are not all equal—there's sampling variation. The largest eigenvalues of random matrices follow the Marchenko-Pastur distribution (in the limit of large $n$ and $d$).

For a $n \times d$ random matrix with $\gamma = d/n < 1$, the eigenvalue distribution has support:

$$[(1-\sqrt{\gamma})^2, (1+\sqrt{\gamma})^2]$$

Eigenvalues outside this range are significant; those inside could be noise.

Parallel Analysis

A simulation-based approach that doesn't require asymptotic formulas:

1. Generate $B$ random datasets of same size $(n \times d)$ from $\mathcal{N}(0, 1)$
2. Compute eigenvalues of each
3. For each component $j$, compute 95th percentile of eigenvalue $j$ across simulations
4. Keep real components where $\lambda_j^{\text{real}} > \lambda_j^{95\%}$

This accounts for finite-sample effects that the Marchenko-Pastur approximation might miss.

Scree plots are powerful but can mislead. Understanding common pitfalls helps you avoid drawing incorrect conclusions.

Pitfall 1: Scale Illusions

The same data plotted at different scales can suggest different elbows:

• Zooming in on small eigenvalues may create apparent structure that's really noise
• Linear scale may hide elbows among small eigenvalues
• Log scale may exaggerate differences in the tail

Mitigation: Always show the full range and multiple scales. Be explicit about scale choices.

Pitfall 2: The First Component Dominance

Sometimes the first eigenvalue is so large that it dominates the plot, making all other structure invisible:

|*
|
|
|
|
|* * * * * * * * *
+----------------------→

This can happen when:

• Data is not centered properly (first PC captures the mean)
• One feature has vastly larger variance (forgot to standardize)
• There's a genuine dominant factor

Solutions: Check centering, consider standardization, use log scale, plot components 2-d separately.

Beyond basic scree plots, advanced techniques provide deeper insights for complex situations.

Bootstrap Confidence Intervals

Eigenvalues have sampling uncertainty. Visualize this with bootstrap confidence intervals:

1. Resample data with replacement $B$ times
2. Compute PCA and eigenvalues for each bootstrap sample
3. Plot percentile intervals (e.g., 5th-95th) around each eigenvalue

This reveals which eigenvalues are distinguishable from each other. If confidence intervals for $\lambda_3$ and $\lambda_4$ overlap, they may represent the same underlying dimensionality.

Stability Analysis

Check how stable the scree plot is to data perturbations:

1. Hold-out stability: Compute eigenvalues on random subsets of observations
2. Feature stability: Compute eigenvalues on random subsets of features
3. Noise injection: Add small noise and check eigenvalue changes

Unstable eigenvalues (large variance across perturbations) should be treated cautiously—they may be fitting noise rather than signal.

Comparative Scree Plots

When analyzing multiple related datasets, overlay their scree plots:

• Helps identify shared structure (similar decay patterns)
• Reveals dataset-specific components
• Supports transfer learning decisions

Example use cases: Comparing PCA across different patient cohorts, analyzing variance structure pre- and post-treatment, comparing training and test set structure.

Residual Diagnostics

After choosing $k$ components, analyze the residuals:

$$\mathbf{E} = \mathbf{X} - \mathbf{X}_k = \mathbf{X} - \mathbf{X}\mathbf{W}_k\mathbf{W}_k^T$$

Plot:

• Distribution of residual norms: should be uniform if $k$ captures all structure
• Residual vs. fitted plots: should show no patterns
• Residual spatial structure: should appear random

Patterns in residuals suggest additional structure not captured by $k$ components.