Sources of Error

In the previous page, we derived the mathematical decomposition of prediction error into bias, variance, and irreducible noise. But knowing that error has these components is only half the battle—to actually improve our models, we need to understand where each type of error comes from.

Think of it like debugging: knowing that your program crashes is useful, but knowing why it crashes is essential for fixing it. The same principle applies to machine learning. A high-variance model behaves differently than a high-bias model, and the interventions that help one can actually hurt the other.

This page provides a comprehensive anatomy of error sources. We'll trace each component back to specific aspects of the learning problem: the model architecture, the training algorithm, the data distribution, and the fundamental limits of prediction. By the end, you'll have a diagnostic framework for understanding any model's failure mode.

Bias arises when the learning algorithm systematically misses the true function. Even with infinite training data, a biased algorithm will not converge to the truth. There are several distinct sources:

1. Model Misspecification (Approximation Error)

The most fundamental source of bias is choosing a hypothesis class that doesn't contain the true function or any good approximation to it.

Consider trying to fit a linear model to data generated by: $$f(x) = \sin(2\pi x)$$

No linear function can capture this sinusoidal pattern. The best linear approximation might be a horizontal line through the mean, but it will systematically over-predict near the troughs and under-predict near the peaks. This error is intrinsic to the model class—it's not about having too little data.

Mathematically:

Let $\mathcal{H}$ be the hypothesis class and $f^* = \arg\min_{h \in \mathcal{H}} \mathbb{E}[(f(X) - h(X))^2]$ be the best function in the class. The approximation error is:

$$\text{Approximation Error} = \mathbb{E}[(f(X) - f^*(X))^2]$$

If $f \in \mathcal{H}$, this is zero—the model class can perfectly represent the truth. Otherwise, this term is positive and constitutes irreducible bias relative to the chosen model.

2. Inadequate Features

Even if your model class is theoretically capable of representing complex functions, missing relevant features creates bias. A linear model with only feature $x_1$ cannot capture dependence on $x_2$, regardless of how much data you have.

Example: Predicting house prices with only square footage when location is a major determinant. The model will be biased—systematically over-predicting in cheap neighborhoods and under-predicting in expensive ones.

The Feature Space Perspective:

Consider the true function $f(x_1, x_2) = x_1 + x_2^2$. If we only observe $x_1$ and fit: $$\hat{f}(x_1) = w_0 + w_1 x_1$$

The best we can do is: $$\bar{f}(x_1) = \mathbb{E}[f(x_1, x_2) | x_1] = x_1 + \mathbb{E}[x_2^2]$$

This equals the true function only on average over $x_2$. For any specific $(x_1, x_2)$ pair, there's systematic error from the missing variable.

3. Regularization (Intentional Bias)

Regularization methods like Ridge and Lasso deliberately introduce bias to reduce variance. This is intentional bias—we accept systematic error in exchange for lower variance and better generalization.

In Ridge regression: $$\hat{\mathbf{w}}_{\text{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$

The regularization parameter $\lambda$ shrinks coefficients toward zero. As $\lambda \to \infty$, all coefficients go to zero (maximum bias, minimum variance). As $\lambda \to 0$, we recover OLS (minimum bias, maximum variance).

The Bias-Variance Interpretation:

Regularization biases the solution away from the maximum likelihood estimate, which is unbiased but potentially high-variance. By accepting some bias, we can dramatically reduce variance, especially when features are correlated or when $p$ (number of features) is close to $n$ (number of samples).

4. Wrong Inductive Bias

Every learning algorithm embodies an inductive bias—assumptions about what kinds of patterns are more likely. These biases are necessary (without them, learning is impossible), but wrong biases cause systematic errors.

Examples:

• Smoothness bias: Assuming the function changes gradually. Fails for functions with discontinuities.
• Linear separability: Assuming classes can be separated by a hyperplane. Fails for XOR-like patterns.
• Stationarity: Assuming the data-generating process doesn't change over time. Fails for trending or seasonal data.

Wrong inductive biases can cause models to ignore valid patterns that violate assumptions, or hallucinate patterns that conform to assumptions but don't exist.

Variance measures how much predictions change when we train on different datasets sampled from the same distribution. High variance means the model is overly sensitive to the particular training examples observed.

1. Model Complexity (Too Many Parameters)

The most common source of variance is having more parameters than the data can reliably estimate. Each parameter is like a degree of freedom—more degrees of freedom allow the model to contort itself to fit training noise.

Consider fitting a degree-20 polynomial to 25 data points. The polynomial has 21 parameters—nearly as many as observations. It can pass exactly through all training points, fitting not just the signal but also the noise. A different 25 points would yield a completely different polynomial.

2. Sample Size (Limited Data)

Variance is inversely related to sample size. With more data, estimates become more stable because noise averages out. This is a fundamental statistical principle.

Formal Result:

For many estimators, variance decreases as $O(1/n)$ where $n$ is sample size. For example, the variance of the sample mean is: $$\text{Var}(\bar{X}) = \frac{\sigma^2}{n}$$

Similarly, in linear regression, coefficient variances are proportional to $1/n$ (assuming fixed design). This is why collecting more data is one of the most reliable ways to reduce variance—it doesn't depend on model changes.

3. Training Noise Fitting

Flexible models can fit the noise in training labels, not just the signal. Since the noise is random and different in each training set, predictions vary wildly.

Consider a k-NN classifier with k=1. For any test point, the prediction is the label of the single nearest training point. If that training label is noisy (wrong), the prediction is wrong. With k=1, there's no averaging—noise passes directly to predictions.

Larger k reduces variance by averaging over more neighbors, but increases bias by smoothing over genuine local structure.

4. Sensitive Optimization Landscapes

Some models have rugged optimization landscapes with many local minima. Depending on random initialization or the order of training examples, the algorithm may converge to different solutions.

Deep neural networks exhibit this: the same architecture trained on the same data can yield different results depending on random seeds. This is a form of variance that doesn't come from the hypothesis class itself but from the optimization procedure.

Types of Optimization Variance:

• Initialization sensitivity: Different random initial weights → different solutions
• SGD stochasticity: Random minibatch sampling → different gradient trajectories
• Hyperparameter sensitivity: Small changes in learning rate → very different models

These sources of variance are often called "optimization variance" as opposed to "statistical variance."

5. Feature Collinearity

When features are highly correlated, it becomes impossible to determine which feature is responsible for an effect. Small changes in data cause large swings in the coefficients of correlated features.

Example:

Suppose both $x_1$ and $x_2$ are measures of house size (square feet vs. square meters). They're almost perfectly correlated. The model might learn:

• Training set A: $\hat{y} = 100x_1 + 0x_2$
• Training set B: $\hat{y} = 0x_1 + 929x_2$ (since 1 m² ≈ 10.76 ft²)

Both make similar predictions, but coefficients are wildly different. This is coefficient instability—a form of variance. Regularization (Ridge regression) helps by shrinking coefficients, distributing credit among correlated features.

Irreducible error, denoted $\sigma^2$, represents the fundamental limit of prediction accuracy. No model, no matter how perfect, can reduce error below this floor. Understanding its sources helps set realistic expectations and identify when efforts should shift from modeling to data quality.

1. Measurement Noise

Physical measurements always have some imprecision. Temperature sensors fluctuate. Survey responses include random errors. Medical tests have false positive/negative rates.

If the true relationship is $y = f(x)$ but we observe: $$\tilde{y} = y + \varepsilon_{\text{measurement}}$$

The measurement error $\varepsilon_{\text{measurement}}$ contributes to irreducible variance. Better sensors reduce this component but cannot eliminate it entirely.

2. Unobserved Variables (Hidden Factors)

Often, factors that influence the outcome are not captured in the feature set. Two houses with identical measured features may sell for different prices because of unmeasured factors: nearby school quality, road noise, aesthetic appeal of the neighborhood.

Mathematically, if the true function is $f(x_1, x_2, \ldots, x_k, z)$ where $z$ is unobserved: $$y | \mathbf{x} = \mathbb{E}[f(\mathbf{x}, z) | \mathbf{x}] + \underbrace{(f(\mathbf{x}, z) - \mathbb{E}[f(\mathbf{x}, z) | \mathbf{x}])}_{\text{irreducible given } \mathbf{x}}$$

The variation due to $z$ appears as noise when conditioning only on $\mathbf{x}$.

3. Inherent Stochasticity

Some processes are genuinely random. Quantum phenomena have irreducible uncertainty. Human behavior involves free will and chaotic sensitivity to initial conditions. Financial markets reflect the aggregate of unpredictable individual decisions.

For these processes, even with complete information about observable states, perfect prediction is impossible in principle. The best a model can do is predict expected values and quantify uncertainty.

4. Label Noise (Target Errors)

In supervised learning, the training labels themselves may be noisy. A human annotator might mislabel an image. A medical diagnosis might be incorrect. A recorded stock price might have data entry errors.

Label noise is particularly problematic because it corrupts the very signal we're trying to learn. Unlike feature noise, which adds uncertainty, label noise misleads the learning algorithm.

5. Temporal and Sampling Variability

Even for identical inputs, outcomes may vary over time or across supposedly identical units:

• Same patient, same treatment, different days → different blood pressure readings
• Same product, same price, different weeks → different sales
• Same ad, same user, different contexts → different click probability

This variability represents genuine uncertainty that no static model can capture fully.

Error sources don't exist in isolation—they interact in complex ways that can amplify or mitigate overall prediction error.

Bias-Variance Interaction:

The fundamental tradeoff arises because the same model changes that reduce bias often increase variance, and vice versa:

• Adding features: Reduces bias (captures more signal) but increases variance (more parameters to estimate)
• Increasing polynomial degree: Reduces bias (better approximation) but increases variance (fits noise)
• Reducing regularization strength: Reduces bias (less shrinkage) but increases variance (overfitting)

This is not a coincidence but a mathematical necessity. More flexible models can get closer to any target function (lower bias) but have more ways to fit noise (higher variance).

Noise Amplification:

High irreducible error can amplify variance. When labels are noisy, models struggle to distinguish signal from noise. A flexible model will fit the noise, increasing variance. The relationship is:

• Noisier data → Higher optimal regularization
• Noisier data → Simpler optimal model complexity
• Noisier data → Need for more training examples to achieve fixed accuracy

The Variance-Noise Interaction:

$$\text{Optimal Complexity} \propto \frac{n \cdot \text{Signal Strength}}{\sigma^2}$$

This informal relationship captures the intuition: with more data ($n$) or stronger signal, we can afford more complex models. With higher noise ($\sigma^2$), we must use simpler models.

Data Size and Error Source Dominance:

The dominant error source changes with sample size:

• Small $n$: Variance dominates. Models overfit. Simple models often win.
• Moderate $n$: Bias and variance are comparable. Tuning is critical.
• Large $n$: Bias dominates. Complex models can be reliably estimated.

This explains why rule-of-thumb recommendations often conflict—what works at one scale may fail at another. The "best" model complexity is sample-size dependent.

Given that we can't directly observe bias, variance, and irreducible error, how do we diagnose which is causing poor performance? The key is examining relationships between training/test error and various factors.

Learning Curves: Error vs. Training Set Size

Plot training and test error as a function of training set size:

• High Bias: Both training and test error are high and converge to a similar (high) value. More data doesn't help much.
• High Variance: Large gap between training error (low) and test error (high). Gap narrows with more data.
• Good Fit: Both errors are low and converge to a similar value close to irreducible error.

Validation Curves: Error vs. Model Complexity

Plot training and test error as a function of model complexity (e.g., polynomial degree, regularization strength, tree depth):

• Underfitting region: Both errors high. Model too simple.
• Optimal region: Test error minimized. Balance achieved.
• Overfitting region: Training error low, test error rises. Model too complex.

The gap between curves indicates variance. The level of the test error curve indicates total generalization error.

Once you've diagnosed the dominant error source, you can apply targeted remedies. Using the wrong remedy wastes effort and may worsen performance.

Remedies for High Bias:

Remedies for High Variance:

Remedies for High Irreducible Error:

Let's walk through a realistic debugging scenario to see how these principles apply in practice.

Scenario:

You're building a model to predict customer churn (whether a customer will leave in the next month). Your initial model achieves:

• Training accuracy: 92%
• Test accuracy: 68%

Step 1: Initial Diagnosis

The large gap (92% - 68% = 24%) suggests high variance. The model memorizes training data but doesn't generalize.

Step 2: Examine the Model

You're using a Random Forest with:

• 500 trees
• No max depth limit
• min_samples_leaf = 1

These settings allow trees to grow until pure, likely overfitting.

Step 3: Apply Variance Reduction

Attempt 1: Add max_depth=10.

• Training: 85%, Test: 72%
• Gap reduced, test improved. Progress!

Attempt 2: Also increase min_samples_leaf=20.

• Training: 80%, Test: 75%
• Further improvement.

Attempt 3: Add regularization (increase min_samples_split=50).

• Training: 75%, Test: 76%
• Now training and test are close—variance controlled.

Step 4: Check for Bias

With Training: 75% and Test: 76%, is 75% good enough? Perhaps not for business requirements.

• Try adding features: Include customer tenure, recent interactions.
• Training: 82%, Test: 81%
• Good! Added expressiveness without overfitting.

Step 5: Final Tuning

With the model now in a balanced regime, you can fine-tune:

• Cross-validation to find optimal regularization strength
• Feature selection to remove low-importance variables
• Ensemble size adjustment (maybe 200 trees suffice)

Final Result:

• Training: 80%, Test: 79%
• Gap: 1%—minimal overfitting
• Business requirement: 75% accuracy—exceeded

The key was diagnosing before treating. Random hyperparameter search might have found similar settings eventually, but understanding the error sources made the search targeted and interpretable.