Maximum Depth:… — Pruning and Regularization | OneNoughtOne

0/106

Machine LearningDecision Trees

Pruning and Regularization

LevelIntermediate

Duration75 mins

TopicDecision Trees

4 / 5

Maximum Depth: Controlling Tree Complexity

Tree Depth and Model Complexity

The maximum depth parameter (max_depth) is perhaps the most intuitive regularization control for decision trees. It directly limits how many sequential decisions the tree can make from root to any leaf. Depth has a profound relationship with model complexity: Depth 1 (decision stump) : A single split, 2 leaves, essentially a threshold on one feature Depth 3 : Up to 8 leaves, can model moderate nonlinearities Depth 10 : Up to 1024 leaves, can capture complex patterns Depth 20 : Up to ~1 million leaves, can memorize most datasets This exponential relationship between depth and capacity makes max_depth a powerful complexity control—small changes have large effects.

What You Will Learn

This page explores maximum depth as a complexity control: the mathematical relationship between depth and capacity, how depth affects the bias-variance tradeoff, practical guidelines for depth selection, and the interaction between depth and decision boundary complexity.

The Mathematics of Depth and Capacity

Understanding the mathematical relationship between tree depth and model capacity is essential for effective regularization. Maximum Number of Leaves: A binary tree of depth $d$ has at most $2^d$ leaves: $$|\text{leaves}| \leq 2^d$$ This is achieved when the tree is complete (every non-leaf node has exactly two children at every level up to $d$). Maximum Number of Internal Nodes: Internal nodes (splits) number at most $2^d - 1$: $$|\text{internal nodes}| \leq 2^d - 1$$ Decision Boundary Complexity: Each split introduces an axis-aligned hyperplane. A depth-$d$ tree can have up to $2^d - 1$ hyperplanes, enabling increasingly complex decision boundaries:

Depth	Max Leaves	Max Splits	Boundary Complexity
1	2	1	Single threshold
2	4	3	Simple rectangles
3	8	7	Nested rectangles
5	32	31	Moderate complexity
10	1024	1023	High complexity
20	1M	1M-1	Extreme complexity

Practical Tree Shapes

Real trees are rarely complete. Data-driven splits create imbalanced trees where some branches go deeper than others. The max_depth constraint limits the deepest branch; most leaves will be at shallower depths.

Depth and the Bias-Variance Tradeoff

Maximum depth directly controls the bias-variance tradeoff in decision trees. Shallow Trees (Low Depth): High Bias: Limited capacity constrains what patterns can be learned Low Variance: Simple model is stable across different training sets Underfitting Risk: May miss important patterns in the data Deep Trees (High Depth): Low Bias: High capacity allows learning complex patterns High Variance: Complex model is sensitive to training data specifics Overfitting Risk: May memorize noise in the training data The Optimal Depth: The optimal depth balances these forces. It's typically found via cross-validation: $$d^* = \arg\min_d \mathbb{E}[\text{Test Error}(d)]$$ In practice, expected test error first decreases (bias reduction) then increases (variance increase) as depth grows.

depth_bias_variance.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
 
def analyze_depth_tradeoff(X, y, max_depth_range=range(1, 25)):
    """
    Analyze bias-variance tradeoff as depth increases.
    
    Shows the characteristic U-shaped test error curve.
    """
    train_errors = []
    test_errors = []
    test_stds = []
    
    for depth in max_depth_range:
        clf = DecisionTreeClassifier(
            max_depth=depth,
            random_state=42
        )
        
        # Training error (measure of bias)
        clf.fit(X, y)
        train_error = 1 - clf.score(X, y)
        train_errors.append(train_error)
        
        # CV error (estimate of test error)
        cv_scores = cross_val_score(clf, X, y, cv=5)
        test_error = 1 - cv_scores.mean()
        test_errors.append(test_error)
        test_stds.append(cv_scores.std())
    
    # Find optimal depth
    optimal_idx = np.argmin(test_errors)
    optimal_depth = list(max_depth_range)[optimal_idx]
    
    return {
        'depths': list(max_depth_range),
        'train_errors': train_errors,
        'test_errors': test_errors,
        'test_stds': test_stds,
        'optimal_depth': optimal_depth,
        'optimal_test_error': test_errors[optimal_idx]
    }
 
# Typical pattern:
# - Train error: Decreases monotonically to 0
# - Test error: Decreases, then increases (U-shape)
# - Gap: (Train - Test) increases with depth (overfitting)

The Overfitting Gap

The gap between training accuracy and CV accuracy is a direct measure of overfitting. When this gap exceeds 5-10%, you're likely overfit. Reducing max_depth shrinks this gap. If the gap is near zero, you might have room to increase depth.

Effective Depth vs Maximum Depth

An important distinction exists between the maximum depth parameter and the effective depth of the tree. Maximum Depth (max_depth): The constraint you set Specifies the deepest any branch can grow A hard upper bound Effective Depth (tree.get_depth()): The actual depth achieved May be less than max_depth if other constraints trigger first The depth of the deepest leaf When Effective Depth < Max Depth: Several factors can cause trees to stop before reaching max_depth: Pure nodes : All samples in a node belong to one class min_samples_split : Not enough samples to split further min_samples_leaf : No valid split maintains minimum leaf size min_impurity_decrease : No split provides sufficient impurity reduction Data exhaustion : All features constant within the node

effective_depth_analysis.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from sklearn.tree import DecisionTreeClassifier
import numpy as np
 
def compare_max_vs_effective_depth(X, y):
    """
    Compare max_depth setting to achieved effective depth.
    
    Demonstrates that max_depth is an upper bound, not a target.
    """
    results = []
    
    for max_depth in [5, 10, 15, 20, 25, 30, None]:
        clf = DecisionTreeClassifier(
            max_depth=max_depth,
            random_state=42
        )
        clf.fit(X, y)
        
        results.append({
            'max_depth': max_depth if max_depth else 'None',
            'effective_depth': clf.get_depth(),
            'n_leaves': clf.get_n_leaves(),
            'n_features_used': len(np.unique(clf.tree_.feature[
                clf.tree_.feature >= 0
            ])),
            'train_accuracy': clf.score(X, y)
        })
    
    return results
 
# Example output:
# max_depth=None: effective=15, leaves=87, acc=1.00
# max_depth=20:   effective=15, leaves=87, acc=1.00  (same!)
# max_depth=10:   effective=10, leaves=45, acc=0.97
# max_depth=5:    effective=5,  leaves=18, acc=0.89

Practical Implication

Setting max_depth=100 isn't meaningfully different from max_depth=None for most datasets. The tree will naturally stop well before that depth. However, very large explicit values can slow down tree construction due to deeper recursive calls.

Practical Depth Selection Guidelines

Selecting the right maximum depth involves balancing multiple considerations: model accuracy, interpretability, computational cost, and generalization. Interpretability-First Approach: When the tree itself needs to be explained to stakeholders: Depth 1-3 : Excellent interpretability, can be drawn on a whiteboard Depth 4-5 : Good interpretability, can be followed with effort Depth 6+ : Difficult to interpret as a whole Accuracy-First Approach: When performance is paramount (though you might consider ensembles): Use cross-validation to find optimal depth Plot CV accuracy vs depth to visualize the sweet spot Consider the 1-SE rule: simplest depth within 1 SE of best

Depth Guidelines by Use Case
Use Case	Recommended Depth	Rationale
Decision rules for business	3-5	Must be human-interpretable
Feature importance analysis	5-8	Moderate depth captures relationships
Standalone prediction model	8-15 (CV-tuned)	Balance accuracy and generalization
Random Forest base learner	None (unlimited)	Ensemble handles overfitting
Gradient Boosting base learner	3-8	Shallow trees, sequential correction
Quick baseline model	5-10	Reasonable default range

The log₂(n) Heuristic

A rough heuristic for maximum useful depth is log₂(n), where n is training set size. With 1000 samples, this suggests depth \approx 10. Beyond this, leaves average less than 1 sample—clear overfitting. This is just a heuristic; CV should guide final selection.

Depth and Decision Boundary Geometry

Understanding how depth affects decision boundaries provides geometric intuition for regularization. Axis-Aligned Boundaries: Decision trees create axis-aligned rectangular boundaries. Each split adds a horizontal or vertical line (in 2D) dividing the feature space. Depth 1: One split creates two half-spaces Depth 2: Up to 3 splits create at most 4 rectangles Depth d: Up to $2^d$ rectangles Approximating Nonlinear Boundaries: Trees approximate smooth/diagonal boundaries with staircase patterns: Shallow trees: Coarse staircase, may miss nuances Deep trees: Fine staircase, better approximation Very deep trees: Overly fine, starts fitting noise The Staircase Effect: For a diagonal decision boundary $y = x$, a tree needs depth $d$ to achieve approximation error $O(2^{-d})$. This is why trees struggle with smooth diagonal boundaries without sufficient depth—but too much depth leads to overfitting.

Geometric Insights

•Linear boundaries: Depth 1-2 often sufficient if boundary is axis-aligned
•Diagonal boundaries: Require moderate depth (5-10) for reasonable approximation
•Circular/curved boundaries: Require higher depth for staircase approximation
•High-dimensional data: Depth needs increase less dramatically (many features to split on)
•Clustered data: Natural clusters may be separable at low depth

Cross-Validation for Depth Selection

Cross-validation is the gold standard for selecting optimal depth. Here's a complete workflow.

cv_depth_selection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV
import numpy as np
 
def select_optimal_depth(X, y, max_search_depth=20, cv=5):
    """
    Select optimal max_depth using cross-validation.
    
    Implements both exhaustive search and 1-SE rule.
    """
    depths = range(1, max_search_depth + 1)
    
    cv_means = []
    cv_stds = []
    
    for depth in depths:
        clf = DecisionTreeClassifier(
            max_depth=depth,
            random_state=42
        )
        scores = cross_val_score(clf, X, y, cv=cv)
        cv_means.append(scores.mean())
        cv_stds.append(scores.std())
    
    cv_means = np.array(cv_means)
    cv_stds = np.array(cv_stds)
    
    # Best depth (max rule)
    best_idx = np.argmax(cv_means)
    best_depth = depths[best_idx]
    
    # 1-SE rule: simplest depth within 1 SE of best
    best_mean = cv_means[best_idx]
    best_se = cv_stds[best_idx] / np.sqrt(cv)
    threshold = best_mean - best_se
    
    # Find smallest depth meeting threshold
    depth_1se = best_depth
    for i, (depth, mean) in enumerate(zip(depths, cv_means)):
        if mean >= threshold:
            depth_1se = depth
            break
    
    return {
        'best_depth': best_depth,
        'best_cv_score': cv_means[best_idx],
        'depth_1se': depth_1se,
        'cv_score_1se': cv_means[depth_1se - 1],
        'cv_means': cv_means.tolist(),
        'cv_stds': cv_stds.tolist()
    }
 
# Usage:
# result = select_optimal_depth(X_train, y_train)
# print(f"Best depth: {result['best_depth']}")
# print(f"1-SE depth: {result['depth_1se']}")  # Often simpler, nearly as good

Summary

Key Takeaways

•Exponential capacity: A depth-d tree can have up to 2^d leaves; small depth changes have large effects
•Bias-variance tradeoff: Shallow trees have high bias/low variance; deep trees have low bias/high variance
•Effective vs maximum depth: Trees often stop before max_depth due to other constraints
•Interpretability threshold: Depth 5-6 is typically the limit for human comprehension
•Decision boundaries: Greater depth enables finer staircase approximation of smooth boundaries
•CV for selection: Cross-validation with 1-SE rule balances performance and parsimony

Page Complete

You now understand maximum depth as a complexity control mechanism, its mathematical foundations, and practical tuning approaches. Next, we'll explore minimum impurity decrease—a parameter that directly controls the quality threshold for splits.

Machine LearningDecision Trees

Pruning and Regularization

LevelIntermediate

Duration75 mins

TopicDecision Trees

4 / 5

Maximum Depth: Controlling Tree Complexity

Tree Depth and Model Complexity

The maximum depth parameter (max_depth) is perhaps the most intuitive regularization control for decision trees. It directly limits how many sequential decisions the tree can make from root to any leaf. Depth has a profound relationship with model complexity: Depth 1 (decision stump) : A single split, 2 leaves, essentially a threshold on one feature Depth 3 : Up to 8 leaves, can model moderate nonlinearities Depth 10 : Up to 1024 leaves, can capture complex patterns Depth 20 : Up to ~1 million leaves, can memorize most datasets This exponential relationship between depth and capacity makes max_depth a powerful complexity control—small changes have large effects.

What You Will Learn

This page explores maximum depth as a complexity control: the mathematical relationship between depth and capacity, how depth affects the bias-variance tradeoff, practical guidelines for depth selection, and the interaction between depth and decision boundary complexity.

The Mathematics of Depth and Capacity

Understanding the mathematical relationship between tree depth and model capacity is essential for effective regularization. Maximum Number of Leaves: A binary tree of depth $d$ has at most $2^d$ leaves: $$|\text{leaves}| \leq 2^d$$ This is achieved when the tree is complete (every non-leaf node has exactly two children at every level up to $d$). Maximum Number of Internal Nodes: Internal nodes (splits) number at most $2^d - 1$: $$|\text{internal nodes}| \leq 2^d - 1$$ Decision Boundary Complexity: Each split introduces an axis-aligned hyperplane. A depth-$d$ tree can have up to $2^d - 1$ hyperplanes, enabling increasingly complex decision boundaries:

Depth	Max Leaves	Max Splits	Boundary Complexity
1	2	1	Single threshold
2	4	3	Simple rectangles
3	8	7	Nested rectangles
5	32	31	Moderate complexity
10	1024	1023	High complexity
20	1M	1M-1	Extreme complexity

Practical Tree Shapes

Real trees are rarely complete. Data-driven splits create imbalanced trees where some branches go deeper than others. The max_depth constraint limits the deepest branch; most leaves will be at shallower depths.

Depth and the Bias-Variance Tradeoff

Maximum depth directly controls the bias-variance tradeoff in decision trees. Shallow Trees (Low Depth): High Bias: Limited capacity constrains what patterns can be learned Low Variance: Simple model is stable across different training sets Underfitting Risk: May miss important patterns in the data Deep Trees (High Depth): Low Bias: High capacity allows learning complex patterns High Variance: Complex model is sensitive to training data specifics Overfitting Risk: May memorize noise in the training data The Optimal Depth: The optimal depth balances these forces. It's typically found via cross-validation: $$d^* = \arg\min_d \mathbb{E}[\text{Test Error}(d)]$$ In practice, expected test error first decreases (bias reduction) then increases (variance increase) as depth grows.

depth_bias_variance.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
 
def analyze_depth_tradeoff(X, y, max_depth_range=range(1, 25)):
    """
    Analyze bias-variance tradeoff as depth increases.
    
    Shows the characteristic U-shaped test error curve.
    """
    train_errors = []
    test_errors = []
    test_stds = []
    
    for depth in max_depth_range:
        clf = DecisionTreeClassifier(
            max_depth=depth,
            random_state=42
        )
        
        # Training error (measure of bias)
        clf.fit(X, y)
        train_error = 1 - clf.score(X, y)
        train_errors.append(train_error)
        
        # CV error (estimate of test error)
        cv_scores = cross_val_score(clf, X, y, cv=5)
        test_error = 1 - cv_scores.mean()
        test_errors.append(test_error)
        test_stds.append(cv_scores.std())
    
    # Find optimal depth
    optimal_idx = np.argmin(test_errors)
    optimal_depth = list(max_depth_range)[optimal_idx]
    
    return {
        'depths': list(max_depth_range),
        'train_errors': train_errors,
        'test_errors': test_errors,
        'test_stds': test_stds,
        'optimal_depth': optimal_depth,
        'optimal_test_error': test_errors[optimal_idx]
    }
 
# Typical pattern:
# - Train error: Decreases monotonically to 0
# - Test error: Decreases, then increases (U-shape)
# - Gap: (Train - Test) increases with depth (overfitting)

The Overfitting Gap

The gap between training accuracy and CV accuracy is a direct measure of overfitting. When this gap exceeds 5-10%, you're likely overfit. Reducing max_depth shrinks this gap. If the gap is near zero, you might have room to increase depth.

Effective Depth vs Maximum Depth

An important distinction exists between the maximum depth parameter and the effective depth of the tree. Maximum Depth (max_depth): The constraint you set Specifies the deepest any branch can grow A hard upper bound Effective Depth (tree.get_depth()): The actual depth achieved May be less than max_depth if other constraints trigger first The depth of the deepest leaf When Effective Depth < Max Depth: Several factors can cause trees to stop before reaching max_depth: Pure nodes : All samples in a node belong to one class min_samples_split : Not enough samples to split further min_samples_leaf : No valid split maintains minimum leaf size min_impurity_decrease : No split provides sufficient impurity reduction Data exhaustion : All features constant within the node

effective_depth_analysis.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from sklearn.tree import DecisionTreeClassifier
import numpy as np
 
def compare_max_vs_effective_depth(X, y):
    """
    Compare max_depth setting to achieved effective depth.
    
    Demonstrates that max_depth is an upper bound, not a target.
    """
    results = []
    
    for max_depth in [5, 10, 15, 20, 25, 30, None]:
        clf = DecisionTreeClassifier(
            max_depth=max_depth,
            random_state=42
        )
        clf.fit(X, y)
        
        results.append({
            'max_depth': max_depth if max_depth else 'None',
            'effective_depth': clf.get_depth(),
            'n_leaves': clf.get_n_leaves(),
            'n_features_used': len(np.unique(clf.tree_.feature[
                clf.tree_.feature >= 0
            ])),
            'train_accuracy': clf.score(X, y)
        })
    
    return results
 
# Example output:
# max_depth=None: effective=15, leaves=87, acc=1.00
# max_depth=20:   effective=15, leaves=87, acc=1.00  (same!)
# max_depth=10:   effective=10, leaves=45, acc=0.97
# max_depth=5:    effective=5,  leaves=18, acc=0.89

Practical Implication

Setting max_depth=100 isn't meaningfully different from max_depth=None for most datasets. The tree will naturally stop well before that depth. However, very large explicit values can slow down tree construction due to deeper recursive calls.

Practical Depth Selection Guidelines

Selecting the right maximum depth involves balancing multiple considerations: model accuracy, interpretability, computational cost, and generalization. Interpretability-First Approach: When the tree itself needs to be explained to stakeholders: Depth 1-3 : Excellent interpretability, can be drawn on a whiteboard Depth 4-5 : Good interpretability, can be followed with effort Depth 6+ : Difficult to interpret as a whole Accuracy-First Approach: When performance is paramount (though you might consider ensembles): Use cross-validation to find optimal depth Plot CV accuracy vs depth to visualize the sweet spot Consider the 1-SE rule: simplest depth within 1 SE of best

Depth Guidelines by Use Case
Use Case	Recommended Depth	Rationale
Decision rules for business	3-5	Must be human-interpretable
Feature importance analysis	5-8	Moderate depth captures relationships
Standalone prediction model	8-15 (CV-tuned)	Balance accuracy and generalization
Random Forest base learner	None (unlimited)	Ensemble handles overfitting
Gradient Boosting base learner	3-8	Shallow trees, sequential correction
Quick baseline model	5-10	Reasonable default range

The log₂(n) Heuristic

A rough heuristic for maximum useful depth is log₂(n), where n is training set size. With 1000 samples, this suggests depth \approx 10. Beyond this, leaves average less than 1 sample—clear overfitting. This is just a heuristic; CV should guide final selection.

Depth and Decision Boundary Geometry

Understanding how depth affects decision boundaries provides geometric intuition for regularization. Axis-Aligned Boundaries: Decision trees create axis-aligned rectangular boundaries. Each split adds a horizontal or vertical line (in 2D) dividing the feature space. Depth 1: One split creates two half-spaces Depth 2: Up to 3 splits create at most 4 rectangles Depth d: Up to $2^d$ rectangles Approximating Nonlinear Boundaries: Trees approximate smooth/diagonal boundaries with staircase patterns: Shallow trees: Coarse staircase, may miss nuances Deep trees: Fine staircase, better approximation Very deep trees: Overly fine, starts fitting noise The Staircase Effect: For a diagonal decision boundary $y = x$, a tree needs depth $d$ to achieve approximation error $O(2^{-d})$. This is why trees struggle with smooth diagonal boundaries without sufficient depth—but too much depth leads to overfitting.

Geometric Insights

•Linear boundaries: Depth 1-2 often sufficient if boundary is axis-aligned
•Diagonal boundaries: Require moderate depth (5-10) for reasonable approximation
•Circular/curved boundaries: Require higher depth for staircase approximation
•High-dimensional data: Depth needs increase less dramatically (many features to split on)
•Clustered data: Natural clusters may be separable at low depth

Cross-Validation for Depth Selection

Cross-validation is the gold standard for selecting optimal depth. Here's a complete workflow.

cv_depth_selection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV
import numpy as np
 
def select_optimal_depth(X, y, max_search_depth=20, cv=5):
    """
    Select optimal max_depth using cross-validation.
    
    Implements both exhaustive search and 1-SE rule.
    """
    depths = range(1, max_search_depth + 1)
    
    cv_means = []
    cv_stds = []
    
    for depth in depths:
        clf = DecisionTreeClassifier(
            max_depth=depth,
            random_state=42
        )
        scores = cross_val_score(clf, X, y, cv=cv)
        cv_means.append(scores.mean())
        cv_stds.append(scores.std())
    
    cv_means = np.array(cv_means)
    cv_stds = np.array(cv_stds)
    
    # Best depth (max rule)
    best_idx = np.argmax(cv_means)
    best_depth = depths[best_idx]
    
    # 1-SE rule: simplest depth within 1 SE of best
    best_mean = cv_means[best_idx]
    best_se = cv_stds[best_idx] / np.sqrt(cv)
    threshold = best_mean - best_se
    
    # Find smallest depth meeting threshold
    depth_1se = best_depth
    for i, (depth, mean) in enumerate(zip(depths, cv_means)):
        if mean >= threshold:
            depth_1se = depth
            break
    
    return {
        'best_depth': best_depth,
        'best_cv_score': cv_means[best_idx],
        'depth_1se': depth_1se,
        'cv_score_1se': cv_means[depth_1se - 1],
        'cv_means': cv_means.tolist(),
        'cv_stds': cv_stds.tolist()
    }
 
# Usage:
# result = select_optimal_depth(X_train, y_train)
# print(f"Best depth: {result['best_depth']}")
# print(f"1-SE depth: {result['depth_1se']}")  # Often simpler, nearly as good

Summary

Key Takeaways

•Exponential capacity: A depth-d tree can have up to 2^d leaves; small depth changes have large effects
•Bias-variance tradeoff: Shallow trees have high bias/low variance; deep trees have low bias/high variance
•Effective vs maximum depth: Trees often stop before max_depth due to other constraints
•Interpretability threshold: Depth 5-6 is typically the limit for human comprehension
•Decision boundaries: Greater depth enables finer staircase approximation of smooth boundaries
•CV for selection: Cross-validation with 1-SE rule balances performance and parsimony

Page Complete

You now understand maximum depth as a complexity control mechanism, its mathematical foundations, and practical tuning approaches. Next, we'll explore minimum impurity decrease—a parameter that directly controls the quality threshold for splits.

Maximum Depth: Controlling Tree Complexity

Tree Depth and Model Complexity

The Mathematics of Depth and Capacity

Depth and the Bias-Variance Tradeoff

Effective Depth vs Maximum Depth

Practical Depth Selection Guidelines

Depth and Decision Boundary Geometry

Cross-Validation for Depth Selection

Summary

There’s more to understand.

Maximum Depth: Controlling Tree Complexity

Tree Depth and Model Complexity

The Mathematics of Depth and Capacity

Depth and the Bias-Variance Tradeoff

Effective Depth vs Maximum Depth

Practical Depth Selection Guidelines

Depth and Decision Boundary Geometry

Cross-Validation for Depth Selection

Summary

There’s more to understand.