AutoML Overview

The search space is the universe of configurations that an AutoML system can explore. It defines the boundaries of what's achievable—configurations outside the search space can never be discovered, no matter how effective the search strategy.

Search space design is a critical skill that separates effective AutoML from wasted computation. Too narrow a space may exclude the optimal solution; too broad wastes resources exploring irrelevant configurations.

This page covers the theory and practice of search space design: parameter types, space structure, conditioning, and the art of balancing expressiveness with tractability.

A search space $\mathcal{X}$ is a set of possible configurations. Each configuration $x \in \mathcal{X}$ specifies values for all hyperparameters:

$$x = (x_1, x_2, ..., x_n)$$

where each $x_i$ is drawn from its corresponding domain $\mathcal{X}_i$.

Parameter Types:

Different parameters have fundamentally different characteristics that affect how they should be searched:

Scale Matters:

Many parameters are best searched on logarithmic scale. Learning rates, regularization strengths, and other multiplicative factors vary over orders of magnitude:

• Learning rate: $10^{-5}$ to $10^{-1}$ (5 orders of magnitude)
• L2 regularization: $10^{-6}$ to $10^{0}$ (6 orders of magnitude)

Searching uniformly in linear space wastes most samples in the upper ranges. Log-scale sampling distributes samples evenly across magnitudes:

Real ML configurations have conditional structure: some parameters only matter when others take specific values. This creates a hierarchical, tree-structured search space.

Example: Neural Network Configuration

Consider configuring a neural network optimizer:

• If optimizer = 'SGD': momentum matters, beta1/beta2 don't
• If optimizer = 'Adam': beta1/beta2 matter, momentum doesn't
• If use_dropout = True: dropout_rate matters
• If use_dropout = False: dropout_rate is irrelevant

Naively treating all parameters as independent wastes search effort on meaningless combinations.

Why Conditional Spaces Matter:

1. Reduced Effective Dimensionality: A 20-parameter space with conditions may have only 8-10 active parameters for any given configuration.

2. Meaningful Configurations Only: Prevents wasting evaluations on nonsensical combinations.

3. Better Surrogate Models: Optimization algorithms can learn structure rather than treating inactive parameters as noise.

4. Hardware/Memory Constraints: Can encode constraints like "if model_size = large, then batch_size ≤ 32".

Combined Algorithm Selection and Hyperparameter optimization (CASH) unifies algorithm choice with parameter tuning into a single hierarchical search space.

Structure:

Root: algorithm ∈ {RandomForest, GradientBoosting, SVM, NeuralNet, ...}
│
├── If RandomForest:
│   ├── n_estimators ∈ [50, 500]
│   ├── max_depth ∈ [None, 5, 10, 20, 50]
│   └── min_samples_split ∈ [2, 20]
│
├── If GradientBoosting:
│   ├── n_estimators ∈ [50, 500]  
│   ├── learning_rate ∈ [0.001, 0.3] (log)
│   └── max_depth ∈ [3, 10]
│
├── If SVM:
│   ├── C ∈ [0.001, 1000] (log)
│   ├── kernel ∈ {linear, rbf, poly}
│   └── If kernel = rbf: gamma ∈ [1e-5, 10] (log)
│
└── If NeuralNet:
    ├── n_layers ∈ [1, 5]
    ├── hidden_size ∈ [16, 512]
    └── activation ∈ {relu, tanh, elu}

This hierarchical structure naturally encodes that different algorithms have different hyperparameters.

NAS search spaces define the universe of possible neural network architectures. The design of this space profoundly affects both the quality of discovered architectures and the computational cost of search.

Cell-Based Search Spaces:

Modern NAS typically searches for cells—small repeating units—rather than entire networks. A cell contains nodes connected by operations. The network is constructed by stacking cells.

Search Space Size Calculation:

For a cell with B nodes, each receiving 2 inputs from prior nodes, with K operations:

$$|\mathcal{X}| = \prod_{i=2}^{B+1} K^2 \cdot \binom{i}{2} = K^{2B} \cdot \frac{(B+1)!}{2}$$

With B=4 nodes and K=8 operations: $$|\mathcal{X}| = 8^8 \cdot 60 \approx 10^9 \text{ architectures}$$

Exhaustive search is clearly impossible.

Constraining the Space:

Effective NAS requires balancing expressiveness with tractability:

• Reduce operations: Fewer primitive operations shrink the space exponentially
• Limit connections: Force each node to select exactly 2 inputs rather than any subset
• Share cells: Use same cell throughout rather than unique cells per layer
• Pre-define macro: Fix overall network structure, search only within cells

Real-world AutoML must optimize not just accuracy but multiple objectives subject to constraints.

Common Objectives:

• Prediction accuracy/error
• Inference latency
• Model size (memory footprint)
• Training time
• Energy consumption
• Interpretability

Common Constraints:

• Maximum inference time per prediction
• Maximum model size for deployment
• Minimum accuracy threshold
• Hardware-specific limits (TPU/GPU memory)

Effective search space design balances competing concerns. These principles guide the process:

Common Mistakes:

What's Next:

With the search space defined, we need search strategies to explore it efficiently. The next page covers methods from random search through Bayesian optimization to evolutionary algorithms—the engines that power AutoML exploration.