Inner and Outer Loops

Having understood why model selection bias corrupts performance estimates, we now turn to how nested cross-validation solves this problem through its ingenious two-loop structure.

Nested cross-validation (nested CV, double cross-validation, or NCV) isn't merely running cross-validation twice—it's a carefully designed separation of concerns that ensures the data used for model selection is completely independent from the data used for performance evaluation. This separation is what makes unbiased estimation possible.

Nested CV consists of two distinct, hierarchically organized cross-validation procedures:

The Outer Loop (Evaluation Loop)

• Purpose: Estimate the generalization performance of the entire model selection procedure
• Splits the full dataset into K_outer folds
• Each iteration holds out one fold as the outer test set
• The remaining K_outer - 1 folds form the outer training set
• Never touches hyperparameter tuning directly—it just evaluates whatever the inner loop produces

The Inner Loop (Selection Loop)

• Purpose: Select the best hyperparameters using only the outer training set
• Runs inside each outer loop iteration
• Splits the outer training set into K_inner folds
• Evaluates each hyperparameter configuration via inner CV
• Returns the best configuration to the outer loop

Visual representation of one outer iteration:

Full Dataset
├── Outer Training Set (K_outer - 1 folds) ──────────────────────┐
│   ├── Inner Fold 1 (test) ←─┐                                  │
│   ├── Inner Fold 2 (test) ←─┤                                  │
│   ├── Inner Fold 3 (test) ←─┼── Inner CV evaluates             │
│   ├── Inner Fold 4 (test) ←─┤   each hyperparameter config     │
│   └── Inner Fold 5 (test) ←─┘                                  │
│                              ↓                                  │
│          Best hyperparameters selected                         │
│                              ↓                                  │
│          Train final model on ALL outer training data          │
│                              │                                  │
└──────────────────────────────│──────────────────────────────────┘
                               ↓
└── Outer Test Set (1 fold) ← Evaluate final model (this is one outer score)

This process repeats K_outer times, each time with a different fold held out as the outer test set. The K_outer outer test scores are averaged to produce the final nested CV estimate.

Let's formalize the nested CV algorithm with precise pseudocode that can be directly translated to implementation.

Key algorithmic details:

1. Outer split before inner: The outer test set is separated before any inner loop processing occurs
2. Inner loop is self-contained: It only sees outer training data and makes no reference to outer test
3. Retrain after selection: After selecting best hyperparameters, we retrain on the entire outer training set—not just the inner training folds
4. One score per outer fold: Each outer iteration produces exactly one test score
5. Different fold randomization: Inner and outer splitters can use different random states

Understanding precisely what data flows where is crucial for implementing nested CV correctly and avoiding subtle bugs that reintroduce selection bias.

Data sizes through the pipeline (example):

Note how inner validation sets are quite small. With 1000 samples, inner validation uses only 160 samples—potentially high variance. This is one reason nested CV is most valuable for moderately-sized datasets where variance matters but is manageable.

Despite their structural similarity (both are cross-validation), the inner and outer loops serve fundamentally different purposes. Understanding this distinction is key to proper interpretation.

Why inner scores don't matter for reporting:

Inner CV scores are computed during the selection process. They're used to rank hyperparameter configurations, and the best-scoring configuration is selected. This means inner scores suffer from exactly the selection bias we discussed in the previous page.

However, this is fine because we don't use inner scores for reporting. The outer loop provides unbiased evaluation by testing on data the inner loop never saw. The inner loop's job is simply to make reasonable selections—it doesn't need to be unbiased, just effective.

Analogy:

Imagine a sports team holding tryouts (inner loop) and then playing in a tournament (outer loop). The tryout scores help select the roster but don't predict tournament performance. A player might score high in tryouts partly due to lucky matchups against other candidates. The tournament games (against external opponents) give a true measure of the selected team's ability.

The choice of fold numbers for inner and outer loops involves tradeoffs between bias, variance, and computational cost.

K_outer (Outer Folds):

The outer loop determines how many independent performance estimates you'll average. Standard choices:

For the outer loop, K_outer = 5 is the most common choice over even K_outer = 10 because:

• Outer test sets need sufficient size for reliable performance measurement
• With K_outer = 10, each outer test set is only 10% of data—potentially high variance
• The computational cost multiplier of K_outer = 10 is rarely justified

K_inner (Inner Folds):

The inner loop is for model selection, and its requirements differ:

For inner loops, there's more flexibility because:

• Inner scores are only used for ranking, not reporting
• A slightly suboptimal selection isn't catastrophic
• Computational savings multiply across the parameter grid

Asymmetric choices:

There's no requirement for K_outer = K_inner. In fact, different values are often appropriate:

• Larger K_outer, smaller K_inner: Prioritizes evaluation quality over selection precision
• Smaller K_outer, larger K_inner: Prioritizes selection precision (unusual but valid for very expensive models)

Leave-One-Out considerations:

LOOCV (K = N) is almost never appropriate for nested CV:

• Outer LOOCV provides unstable, high-variance estimates
• Inner LOOCV is computationally prohibitive
• The combination is essentially never practical

For classification problems, stratified nested CV maintains class proportions in both inner and outer folds. This is especially important for imbalanced datasets.

When stratification is critical:

• Imbalanced classes: Minority class might be missing from some folds without stratification
• Small datasets: Random variation in class proportions is high
• Multi-class problems: Many classes increase the importance of balanced representation

Regression stratification:

For regression problems, you can create synthetic strata by binning the target variable:

from sklearn.model_selection import StratifiedKFold
import pandas as pd

# Create bins for stratification
y_binned = pd.cut(y, bins=5, labels=False)

# Use binned y for stratification
outer_cv = StratifiedKFold(n_splits=5)
for train_idx, test_idx in outer_cv.split(X, y_binned):
    # ... rest of nested CV

This ensures each fold has a similar distribution of target values.

An important feature of nested CV: the best hyperparameters can differ across outer folds. This behavior is expected and informative.

Why configurations vary:

1. Statistical variation: With limited data, different train/test splits favor different configurations
2. Flat optima: Often many configurations perform similarly; the 'winner' is determined by noise
3. Data heterogeneity: Different folds may emphasize different data regions

What to do about it:

• Don't force consistency: Different configurations are allowed and expected
• Look for patterns: If configurations vary wildly, the choice may not matter much
• Final model selection: For deployment, run a final non-nested CV on all data to pick one configuration

Final model training process:

1. Report nested CV score as your unbiased performance estimate
2. Run standard (non-nested) CV on full dataset to select final hyperparameters
3. Train final model with selected hyperparameters on all available data
4. Deploy this model