Multi-objective Evaluation — Custom and Business… | OneNoughtOne

0/106

Machine LearningCustom and Business Metrics

Custom and Business Metrics

LevelIntermediate

Duration90 mins

TopicCustom and Business Metrics

4 / 5

Multi-objective Evaluation

When One Metric Isn't Enough

Real-world ML deployments rarely optimize a single objective. Consider a recommendation system that must simultaneously: Maximize relevance (user satisfaction) Maximize diversity (content discovery) Minimize computational cost (latency, resources) Ensure fairness (equitable exposure) These objectives often conflict . More diversity may reduce relevance. Lower latency may require simpler models with worse accuracy. Multi-objective evaluation provides frameworks for understanding, comparing, and selecting models when multiple objectives matter—even when they can't all be maximized simultaneously.

What You Will Learn

By the end of this page, you will understand Pareto optimality and dominance, apply scalarization techniques to combine objectives, construct and interpret Pareto frontiers, and make principled decisions when objectives conflict.

Understanding Objective Conflicts

Common Objective Conflicts in ML: Precision vs. Recall: Fundamental trade-off in classification Accuracy vs. Fairness: Optimal predictors may encode biases Performance vs. Interpretability: Complex models often perform better but explain worse Performance vs. Efficiency: Better models often require more compute Short-term vs. Long-term: Optimizing immediate metrics may harm long-term outcomes Mathematical Representation: Let $f_1(\theta), f_2(\theta), ..., f_k(\theta)$ be k objective functions over model/threshold space $\theta$. Multi-objective optimization seeks: $$\max_{\theta} {f_1(\theta), f_2(\theta), ..., f_k(\theta)}$$ When objectives conflict, no single $\theta$ maximizes all objectives simultaneously.

Example: Three Models with Conflicting Objectives
Model	Accuracy	Inference Time	Interpretability
Logistic Regression	78%	1ms	High
Random Forest	85%	15ms	Medium
Deep Neural Network	89%	50ms	Low

No model dominates on all three objectives. The "best" model depends on how you weight accuracy vs. speed vs. explainability—which is a business decision, not a technical one.

Pareto Optimality and Dominance

Pareto Dominance: Solution A dominates solution B if: A is at least as good as B on all objectives, AND A is strictly better than B on at least one objective Pareto Optimality: A solution is Pareto optimal (or Pareto efficient) if no other solution dominates it. The set of all Pareto optimal solutions forms the Pareto frontier . Key Insight: Any model on the Pareto frontier represents a valid trade-off—improving any objective requires sacrificing another. Models not on the frontier are strictly inferior: you could do better on at least one metric without losing on others.

pareto_frontier.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
import numpy as np
 
def is_pareto_dominated(point, other_points, maximize=True):
    """
    Check if a point is dominated by any other point.
    
    For maximization: dominated if another point is >= on all and > on at least one.
    """
    for other in other_points:
        if np.array_equal(point, other):
            continue
        if maximize:
            if all(other >= point) and any(other > point):
                return True
        else:
            if all(other <= point) and any(other < point):
                return True
    return False
 
def find_pareto_frontier(points, maximize=True):
    """
    Find Pareto frontier from a set of points.
    
    Parameters
    ----------
    points : array-like, shape (n_points, n_objectives)
        Each row is a solution, each column is an objective value.
    maximize : bool or list of bool
        Whether to maximize each objective.
        
    Returns
    -------
    frontier_indices : list
        Indices of Pareto-optimal solutions.
    """
    points = np.array(points)
    n_points = len(points)
    
    frontier_indices = []
    for i in range(n_points):
        if not is_pareto_dominated(points[i], points, maximize):
            frontier_indices.append(i)
    
    return frontier_indices
 
 
# Example: Model selection with accuracy vs latency
models = [
    {'name': 'Linear', 'accuracy': 0.78, 'latency_ms': 2},
    {'name': 'RF-small', 'accuracy': 0.82, 'latency_ms': 8},
    {'name': 'RF-large', 'accuracy': 0.85, 'latency_ms': 25},
    {'name': 'XGBoost', 'accuracy': 0.86, 'latency_ms': 15},
    {'name': 'DNN-small', 'accuracy': 0.84, 'latency_ms': 20},
    {'name': 'DNN-large', 'accuracy': 0.89, 'latency_ms': 60},
    {'name': 'Ensemble', 'accuracy': 0.90, 'latency_ms': 100},
]
 
# For Pareto: maximize accuracy, minimize latency
# Convert to: maximize accuracy, maximize -latency
points = np.array([[m['accuracy'], -m['latency_ms']] for m in models])
 
frontier_idx = find_pareto_frontier(points, maximize=True)
 
print("Model Comparison: Accuracy vs Latency")
print("=" * 55)
print(f"{'Model':<12} {'Accuracy':>10} {'Latency':>10} {'Pareto':>10}")
print("-" * 55)
 
for i, m in enumerate(models):
    pareto = "✓ Optimal" if i in frontier_idx else ""
    print(f"{m['name']:<12} {m['accuracy']:>10.2%} {m['latency_ms']:>8}ms {pareto:>10}")
 
print(f"
Pareto frontier contains {len(frontier_idx)} models")

Pareto Frontier Interpretation

The Pareto frontier shows the achievable trade-offs. Moving along the frontier, you trade objective performance: more accuracy requires accepting higher latency. The frontier shape reveals the cost of trade-offs—steep regions mean expensive trade-offs; flat regions mean cheap gains.

Scalarization: Combining Multiple Objectives

Scalarization converts multi-objective optimization into single-objective optimization by combining objectives into a scalar value. 1. Weighted Sum (Linear Scalarization): $$S_{\text{linear}} = \sum_{i=1}^{k} w_i \cdot f_i(\theta)$$ where $w_i \geq 0$ and $\sum w_i = 1$. Pros: Simple, interpretable weights Cons: Cannot find solutions in non-convex regions of Pareto frontier 2. Weighted Product (Geometric Scalarization): $$S_{\text{product}} = \prod_{i=1}^{k} f_i(\theta)^{w_i}$$ Pros: Penalizes solutions that are very poor on any objective Cons: Requires all objectives to be positive 3. Chebyshev (Min-Max) Scalarization: $$S_{\text{cheby}} = \min_i \left( w_i \cdot |f_i(\theta) - f_i^*| \right)$$ where $f_i^*$ is the ideal value for objective $i$. Pros: Can find any Pareto-optimal solution Cons: Requires knowing ideal values

scalarization.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import numpy as np
 
def linear_scalarization(objectives, weights):
    """Weighted sum of objectives."""
    return np.dot(objectives, weights)
 
def product_scalarization(objectives, weights):
    """Weighted product (geometric mean) of objectives."""
    return np.prod(np.power(objectives, weights))
 
def chebyshev_scalarization(objectives, ideal, weights):
    """
    Chebyshev scalarization (minimize worst deviation from ideal).
    Returns negative to convert to maximization.
    """
    deviations = weights * np.abs(ideal - objectives)
    return -np.max(deviations)  # Negative because we want to minimize
 
def rank_models_by_scalarization(models, weights, method='linear'):
    """Rank models using specified scalarization method."""
    scores = []
    
    for m in models:
        objectives = np.array([m['accuracy'], 1.0 / m['latency_ms']])  # Normalize latency
        
        if method == 'linear':
            score = linear_scalarization(objectives, weights)
        elif method == 'product':
            score = product_scalarization(objectives, weights)
        elif method == 'chebyshev':
            ideal = np.array([1.0, 1.0])  # Perfect accuracy, instant latency
            score = chebyshev_scalarization(objectives, ideal, weights)
        
        scores.append((m['name'], score))
    
    return sorted(scores, key=lambda x: -x[1])
 
 
models = [
    {'name': 'Linear', 'accuracy': 0.78, 'latency_ms': 2},
    {'name': 'XGBoost', 'accuracy': 0.86, 'latency_ms': 15},
    {'name': 'DNN-large', 'accuracy': 0.89, 'latency_ms': 60},
]
 
# Different weight scenarios
scenarios = [
    ("Accuracy-focused", [0.9, 0.1]),
    ("Balanced", [0.5, 0.5]),
    ("Latency-focused", [0.1, 0.9]),
]
 
print("Model Rankings by Scalarization")
print("=" * 50)
 
for scenario_name, weights in scenarios:
    print(f"
{scenario_name} (weights: {weights}):")
    rankings = rank_models_by_scalarization(models, weights, 'linear')
    for rank, (name, score) in enumerate(rankings, 1):
        print(f"  {rank}. {name}: {score:.4f}")

Practical Decision Strategies

When facing multi-objective decisions, several practical strategies help navigate trade-offs:

Decision Strategies

•Constraint Satisfaction: Set minimum thresholds for secondary objectives, optimize primary objective subject to constraints. Example: "Maximize accuracy such that latency < 20ms and fairness gap < 5%"
•Lexicographic Ordering: Prioritize objectives hierarchically. First optimize most important; among ties, optimize second-most important, etc.
•Reference Point Method: Stakeholders specify desired target for each objective. Select solution closest to reference point.
•Elicitation from Frontier: Show stakeholders the Pareto frontier; let them choose based on visualized trade-offs.
•Regret Minimization: Select solution that minimizes maximum regret across plausible weight scenarios.

constraint_optimization.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def select_with_constraints(models, primary_obj, constraints):
    """
    Select best model on primary objective subject to constraints.
    
    Parameters
    ----------
    models : list of dicts
        Each dict has objective values as keys.
    primary_obj : str
        Key of objective to maximize.
    constraints : dict
        {objective: (op, value)} where op is 'min' or 'max'.
        
    Returns
    -------
    Selected model or None if no feasible model.
    """
    feasible = []
    
    for m in models:
        satisfies = True
        for obj, (op, threshold) in constraints.items():
            if op == 'min' and m[obj] < threshold:
                satisfies = False
            elif op == 'max' and m[obj] > threshold:
                satisfies = False
        
        if satisfies:
            feasible.append(m)
    
    if not feasible:
        return None
    
    return max(feasible, key=lambda x: x[primary_obj])
 
 
# Example
models = [
    {'name': 'A', 'accuracy': 0.78, 'latency_ms': 5, 'fairness_gap': 0.02},
    {'name': 'B', 'accuracy': 0.85, 'latency_ms': 25, 'fairness_gap': 0.04},
    {'name': 'C', 'accuracy': 0.88, 'latency_ms': 50, 'fairness_gap': 0.08},
    {'name': 'D', 'accuracy': 0.84, 'latency_ms': 15, 'fairness_gap': 0.03},
]
 
# Maximize accuracy with constraints
constraints = {
    'latency_ms': ('max', 30),       # Latency must be <= 30ms
    'fairness_gap': ('max', 0.05),   # Fairness gap must be <= 5%
}
 
selected = select_with_constraints(models, 'accuracy', constraints)
 
print("Constraint-Based Selection")
print(f"Primary: Maximize accuracy")
print(f"Constraints: latency <= 30ms, fairness_gap <= 5%")
print(f"
Selected: {selected['name'] if selected else 'No feasible solution'}")
if selected:
    print(f"  Accuracy: {selected['accuracy']:.2%}")
    print(f"  Latency: {selected['latency_ms']}ms")
    print(f"  Fairness gap: {selected['fairness_gap']:.1%}")

Multi-Objective Model Development

Beyond evaluation, multi-objective thinking affects model development: Generating Diverse Solutions: Instead of training one model, generate a portfolio of models along the Pareto frontier: Train with different hyperparameter configurations Train with different regularization strengths Use different model architectures Apply different threshold settings to the same model Multi-Objective Hyperparameter Tuning: Modern HPO tools support multi-objective optimization: Optuna : Built-in multi-objective optimization Ray Tune : Multi-objective search with NSGA-II DEHB : Multi-fidelity multi-objective optimization

Threshold-Based Pareto Frontiers

A single probabilistic classifier generates an entire Pareto frontier by varying its decision threshold. This is cheaper than training multiple models. The precision-recall curve IS a Pareto frontier for precision vs. recall objectives.

Fairness as an Objective: Fairness constraints are naturally represented as additional objectives: Maximize accuracy Minimize demographic parity gap Minimize equalized odds gap The Pareto frontier reveals the accuracy-fairness trade-off explicitly, enabling informed decisions about acceptable trade-offs.

Summary

Key Takeaways

•Acknowledge conflicts: Real deployments involve multiple competing objectives that can't all be maximized
•Use Pareto analysis: Identify Pareto-optimal solutions; eliminate dominated ones
•Scalarize when needed: Weighted sums, products, or Chebyshev methods combine objectives into single scores
•Apply practical strategies: Constraints, lexicographic ordering, and reference points help make decisions
•Involve stakeholders: Trade-off decisions are business decisions; present options, don't just pick one

Page Complete

You now understand how to evaluate and select models when multiple objectives matter. Next, we'll explore metric selection strategy—how to choose the right metrics for your specific problem context.

Machine LearningCustom and Business Metrics

Custom and Business Metrics

LevelIntermediate

Duration90 mins

TopicCustom and Business Metrics

4 / 5

Multi-objective Evaluation

When One Metric Isn't Enough

Real-world ML deployments rarely optimize a single objective. Consider a recommendation system that must simultaneously: Maximize relevance (user satisfaction) Maximize diversity (content discovery) Minimize computational cost (latency, resources) Ensure fairness (equitable exposure) These objectives often conflict . More diversity may reduce relevance. Lower latency may require simpler models with worse accuracy. Multi-objective evaluation provides frameworks for understanding, comparing, and selecting models when multiple objectives matter—even when they can't all be maximized simultaneously.

What You Will Learn

By the end of this page, you will understand Pareto optimality and dominance, apply scalarization techniques to combine objectives, construct and interpret Pareto frontiers, and make principled decisions when objectives conflict.

Understanding Objective Conflicts

Common Objective Conflicts in ML: Precision vs. Recall: Fundamental trade-off in classification Accuracy vs. Fairness: Optimal predictors may encode biases Performance vs. Interpretability: Complex models often perform better but explain worse Performance vs. Efficiency: Better models often require more compute Short-term vs. Long-term: Optimizing immediate metrics may harm long-term outcomes Mathematical Representation: Let $f_1(\theta), f_2(\theta), ..., f_k(\theta)$ be k objective functions over model/threshold space $\theta$. Multi-objective optimization seeks: $$\max_{\theta} {f_1(\theta), f_2(\theta), ..., f_k(\theta)}$$ When objectives conflict, no single $\theta$ maximizes all objectives simultaneously.

Example: Three Models with Conflicting Objectives
Model	Accuracy	Inference Time	Interpretability
Logistic Regression	78%	1ms	High
Random Forest	85%	15ms	Medium
Deep Neural Network	89%	50ms	Low

No model dominates on all three objectives. The "best" model depends on how you weight accuracy vs. speed vs. explainability—which is a business decision, not a technical one.

Pareto Optimality and Dominance

Pareto Dominance: Solution A dominates solution B if: A is at least as good as B on all objectives, AND A is strictly better than B on at least one objective Pareto Optimality: A solution is Pareto optimal (or Pareto efficient) if no other solution dominates it. The set of all Pareto optimal solutions forms the Pareto frontier . Key Insight: Any model on the Pareto frontier represents a valid trade-off—improving any objective requires sacrificing another. Models not on the frontier are strictly inferior: you could do better on at least one metric without losing on others.

pareto_frontier.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
import numpy as np
 
def is_pareto_dominated(point, other_points, maximize=True):
    """
    Check if a point is dominated by any other point.
    
    For maximization: dominated if another point is >= on all and > on at least one.
    """
    for other in other_points:
        if np.array_equal(point, other):
            continue
        if maximize:
            if all(other >= point) and any(other > point):
                return True
        else:
            if all(other <= point) and any(other < point):
                return True
    return False
 
def find_pareto_frontier(points, maximize=True):
    """
    Find Pareto frontier from a set of points.
    
    Parameters
    ----------
    points : array-like, shape (n_points, n_objectives)
        Each row is a solution, each column is an objective value.
    maximize : bool or list of bool
        Whether to maximize each objective.
        
    Returns
    -------
    frontier_indices : list
        Indices of Pareto-optimal solutions.
    """
    points = np.array(points)
    n_points = len(points)
    
    frontier_indices = []
    for i in range(n_points):
        if not is_pareto_dominated(points[i], points, maximize):
            frontier_indices.append(i)
    
    return frontier_indices
 
 
# Example: Model selection with accuracy vs latency
models = [
    {'name': 'Linear', 'accuracy': 0.78, 'latency_ms': 2},
    {'name': 'RF-small', 'accuracy': 0.82, 'latency_ms': 8},
    {'name': 'RF-large', 'accuracy': 0.85, 'latency_ms': 25},
    {'name': 'XGBoost', 'accuracy': 0.86, 'latency_ms': 15},
    {'name': 'DNN-small', 'accuracy': 0.84, 'latency_ms': 20},
    {'name': 'DNN-large', 'accuracy': 0.89, 'latency_ms': 60},
    {'name': 'Ensemble', 'accuracy': 0.90, 'latency_ms': 100},
]
 
# For Pareto: maximize accuracy, minimize latency
# Convert to: maximize accuracy, maximize -latency
points = np.array([[m['accuracy'], -m['latency_ms']] for m in models])
 
frontier_idx = find_pareto_frontier(points, maximize=True)
 
print("Model Comparison: Accuracy vs Latency")
print("=" * 55)
print(f"{'Model':<12} {'Accuracy':>10} {'Latency':>10} {'Pareto':>10}")
print("-" * 55)
 
for i, m in enumerate(models):
    pareto = "✓ Optimal" if i in frontier_idx else ""
    print(f"{m['name']:<12} {m['accuracy']:>10.2%} {m['latency_ms']:>8}ms {pareto:>10}")
 
print(f"
Pareto frontier contains {len(frontier_idx)} models")

Pareto Frontier Interpretation

The Pareto frontier shows the achievable trade-offs. Moving along the frontier, you trade objective performance: more accuracy requires accepting higher latency. The frontier shape reveals the cost of trade-offs—steep regions mean expensive trade-offs; flat regions mean cheap gains.

Scalarization: Combining Multiple Objectives

Scalarization converts multi-objective optimization into single-objective optimization by combining objectives into a scalar value. 1. Weighted Sum (Linear Scalarization): $$S_{\text{linear}} = \sum_{i=1}^{k} w_i \cdot f_i(\theta)$$ where $w_i \geq 0$ and $\sum w_i = 1$. Pros: Simple, interpretable weights Cons: Cannot find solutions in non-convex regions of Pareto frontier 2. Weighted Product (Geometric Scalarization): $$S_{\text{product}} = \prod_{i=1}^{k} f_i(\theta)^{w_i}$$ Pros: Penalizes solutions that are very poor on any objective Cons: Requires all objectives to be positive 3. Chebyshev (Min-Max) Scalarization: $$S_{\text{cheby}} = \min_i \left( w_i \cdot |f_i(\theta) - f_i^*| \right)$$ where $f_i^*$ is the ideal value for objective $i$. Pros: Can find any Pareto-optimal solution Cons: Requires knowing ideal values

scalarization.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import numpy as np
 
def linear_scalarization(objectives, weights):
    """Weighted sum of objectives."""
    return np.dot(objectives, weights)
 
def product_scalarization(objectives, weights):
    """Weighted product (geometric mean) of objectives."""
    return np.prod(np.power(objectives, weights))
 
def chebyshev_scalarization(objectives, ideal, weights):
    """
    Chebyshev scalarization (minimize worst deviation from ideal).
    Returns negative to convert to maximization.
    """
    deviations = weights * np.abs(ideal - objectives)
    return -np.max(deviations)  # Negative because we want to minimize
 
def rank_models_by_scalarization(models, weights, method='linear'):
    """Rank models using specified scalarization method."""
    scores = []
    
    for m in models:
        objectives = np.array([m['accuracy'], 1.0 / m['latency_ms']])  # Normalize latency
        
        if method == 'linear':
            score = linear_scalarization(objectives, weights)
        elif method == 'product':
            score = product_scalarization(objectives, weights)
        elif method == 'chebyshev':
            ideal = np.array([1.0, 1.0])  # Perfect accuracy, instant latency
            score = chebyshev_scalarization(objectives, ideal, weights)
        
        scores.append((m['name'], score))
    
    return sorted(scores, key=lambda x: -x[1])
 
 
models = [
    {'name': 'Linear', 'accuracy': 0.78, 'latency_ms': 2},
    {'name': 'XGBoost', 'accuracy': 0.86, 'latency_ms': 15},
    {'name': 'DNN-large', 'accuracy': 0.89, 'latency_ms': 60},
]
 
# Different weight scenarios
scenarios = [
    ("Accuracy-focused", [0.9, 0.1]),
    ("Balanced", [0.5, 0.5]),
    ("Latency-focused", [0.1, 0.9]),
]
 
print("Model Rankings by Scalarization")
print("=" * 50)
 
for scenario_name, weights in scenarios:
    print(f"
{scenario_name} (weights: {weights}):")
    rankings = rank_models_by_scalarization(models, weights, 'linear')
    for rank, (name, score) in enumerate(rankings, 1):
        print(f"  {rank}. {name}: {score:.4f}")

Practical Decision Strategies

When facing multi-objective decisions, several practical strategies help navigate trade-offs:

Decision Strategies

•Constraint Satisfaction: Set minimum thresholds for secondary objectives, optimize primary objective subject to constraints. Example: "Maximize accuracy such that latency < 20ms and fairness gap < 5%"
•Lexicographic Ordering: Prioritize objectives hierarchically. First optimize most important; among ties, optimize second-most important, etc.
•Reference Point Method: Stakeholders specify desired target for each objective. Select solution closest to reference point.
•Elicitation from Frontier: Show stakeholders the Pareto frontier; let them choose based on visualized trade-offs.
•Regret Minimization: Select solution that minimizes maximum regret across plausible weight scenarios.

constraint_optimization.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def select_with_constraints(models, primary_obj, constraints):
    """
    Select best model on primary objective subject to constraints.
    
    Parameters
    ----------
    models : list of dicts
        Each dict has objective values as keys.
    primary_obj : str
        Key of objective to maximize.
    constraints : dict
        {objective: (op, value)} where op is 'min' or 'max'.
        
    Returns
    -------
    Selected model or None if no feasible model.
    """
    feasible = []
    
    for m in models:
        satisfies = True
        for obj, (op, threshold) in constraints.items():
            if op == 'min' and m[obj] < threshold:
                satisfies = False
            elif op == 'max' and m[obj] > threshold:
                satisfies = False
        
        if satisfies:
            feasible.append(m)
    
    if not feasible:
        return None
    
    return max(feasible, key=lambda x: x[primary_obj])
 
 
# Example
models = [
    {'name': 'A', 'accuracy': 0.78, 'latency_ms': 5, 'fairness_gap': 0.02},
    {'name': 'B', 'accuracy': 0.85, 'latency_ms': 25, 'fairness_gap': 0.04},
    {'name': 'C', 'accuracy': 0.88, 'latency_ms': 50, 'fairness_gap': 0.08},
    {'name': 'D', 'accuracy': 0.84, 'latency_ms': 15, 'fairness_gap': 0.03},
]
 
# Maximize accuracy with constraints
constraints = {
    'latency_ms': ('max', 30),       # Latency must be <= 30ms
    'fairness_gap': ('max', 0.05),   # Fairness gap must be <= 5%
}
 
selected = select_with_constraints(models, 'accuracy', constraints)
 
print("Constraint-Based Selection")
print(f"Primary: Maximize accuracy")
print(f"Constraints: latency <= 30ms, fairness_gap <= 5%")
print(f"
Selected: {selected['name'] if selected else 'No feasible solution'}")
if selected:
    print(f"  Accuracy: {selected['accuracy']:.2%}")
    print(f"  Latency: {selected['latency_ms']}ms")
    print(f"  Fairness gap: {selected['fairness_gap']:.1%}")

Multi-Objective Model Development

Beyond evaluation, multi-objective thinking affects model development: Generating Diverse Solutions: Instead of training one model, generate a portfolio of models along the Pareto frontier: Train with different hyperparameter configurations Train with different regularization strengths Use different model architectures Apply different threshold settings to the same model Multi-Objective Hyperparameter Tuning: Modern HPO tools support multi-objective optimization: Optuna : Built-in multi-objective optimization Ray Tune : Multi-objective search with NSGA-II DEHB : Multi-fidelity multi-objective optimization

Threshold-Based Pareto Frontiers

A single probabilistic classifier generates an entire Pareto frontier by varying its decision threshold. This is cheaper than training multiple models. The precision-recall curve IS a Pareto frontier for precision vs. recall objectives.

Fairness as an Objective: Fairness constraints are naturally represented as additional objectives: Maximize accuracy Minimize demographic parity gap Minimize equalized odds gap The Pareto frontier reveals the accuracy-fairness trade-off explicitly, enabling informed decisions about acceptable trade-offs.

Summary

Key Takeaways

•Acknowledge conflicts: Real deployments involve multiple competing objectives that can't all be maximized
•Use Pareto analysis: Identify Pareto-optimal solutions; eliminate dominated ones
•Scalarize when needed: Weighted sums, products, or Chebyshev methods combine objectives into single scores
•Apply practical strategies: Constraints, lexicographic ordering, and reference points help make decisions
•Involve stakeholders: Trade-off decisions are business decisions; present options, don't just pick one

Page Complete

You now understand how to evaluate and select models when multiple objectives matter. Next, we'll explore metric selection strategy—how to choose the right metrics for your specific problem context.

Multi-objective Evaluation

When One Metric Isn't Enough

Understanding Objective Conflicts

Pareto Optimality and Dominance

Scalarization: Combining Multiple Objectives

Practical Decision Strategies

Multi-Objective Model Development

Summary

There’s more to understand.

Multi-objective Evaluation

When One Metric Isn't Enough

Understanding Objective Conflicts

Pareto Optimality and Dominance

Scalarization: Combining Multiple Objectives

Practical Decision Strategies

Multi-Objective Model Development

Summary

There’s more to understand.