Estimation Theory
Bias and Variance of Estimators
Read aloud is unavailable in this browser.
The Two Faces of Estimation Error
When you estimate a parameter from data, your estimate will almost certainly differ from the true value. But why does it differ? Understanding the sources of estimation error is crucial for designing better estimators and making informed statistical decisions.
It turns out that estimation error has two fundamentally distinct sources:
- Bias: The estimator systematically misses the target
- Variance: The estimator fluctuates due to random sampling
These concepts form the bedrock of statistical estimation theory and have profound implications for machine learning, from model selection to the famous bias-variance tradeoff in prediction.
By the end of this page, you will master formal definitions of bias and variance, understand the bias-variance decomposition of mean squared error, analyze bias and variance of common estimators (sample mean, MLE variance estimator), explore the bias-variance tradeoff and its implications, and develop intuition through the dartboard analogy and mathematical rigor.
Estimators as Random Variables
Before defining bias and variance, we must understand a crucial fact: estimators are random variables.
The setup:
- True parameter: θ* (fixed, unknown)
- Data: D = {X₁, ..., Xₙ} sampled from P(X|θ*)
- Estimator: θ̂ = g(D) — a function of the data
Since D is random (different samples give different data), θ̂ is also random. If we could repeat the sampling process many times, we'd get a distribution of estimates — the sampling distribution of θ̂.
Key insight:
We don't observe the sampling distribution directly—we only have one dataset and one estimate. But the theoretical properties of this distribution (its mean, variance, shape) tell us about the quality of our estimation procedure.
The frequentist perspective on quality:
A good estimator should have a sampling distribution that:
- Is centered at the true parameter (low bias)
- Is concentrated — not spread out (low variance)
- Converges to the true parameter as n increases (consistency)
Thought experiment:
Imagine 1000 statisticians, each given a different random sample of size n from the same population. Each computes θ̂ from their sample.
- The average of all 1000 estimates is E[θ̂]
- The spread among estimates reflects Var(θ̂)
- Bias = E[θ̂] - θ*
The expectation E[θ̂] is over the sampling distribution—averaging over all possible datasets, not over parameters. This is the frequentist perspective. In Bayesian terms, we'd instead look at the posterior distribution, which quantifies uncertainty about θ given a single fixed dataset.
Formal Definition of Bias
Definition:
The bias of an estimator θ̂ is the difference between its expected value and the true parameter:
$$\text{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta^*$$
Interpretation:
- Bias > 0: The estimator tends to overestimate (on average)
- Bias < 0: The estimator tends to underestimate (on average)
- Bias = 0: The estimator is unbiased
An unbiased estimator satisfies E[θ̂] = θ* for all possible values of θ*.
Example 1: Sample Mean is Unbiased
Let X₁, ..., Xₙ be i.i.d. with E[Xᵢ] = μ. The sample mean is:
$$\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$$
Its expected value:
$$E[\bar{X}] = E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n E[X_i] = \frac{1}{n} \cdot n\mu = \mu$$
Since E[X̄] = μ, the sample mean is an unbiased estimator of the population mean.
Example 2: MLE Variance Estimator is Biased
For data from N(μ, σ²), the MLE for variance is:
$$\hat{\sigma}^2_{MLE} = \frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^2$$
Its expected value:
$$E[\hat{\sigma}^2_{MLE}] = \frac{n-1}{n}\sigma^2 eq \sigma^2$$
Bias = E[σ̂²] - σ² = -σ²/n < 0
The MLE systematically underestimates the true variance! This motivates Bessel's correction: using n-1 in the denominator.
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697
import numpy as npimport matplotlib.pyplot as plt def demonstrate_mle_variance_bias(): """ Empirically demonstrate that MLE variance estimator is biased. """ np.random.seed(42) true_mu = 0 true_sigma2 = 4 # True variance sample_sizes = [5, 10, 20, 50, 100, 500] n_simulations = 10000 results = [] for n in sample_sizes: mle_estimates = [] unbiased_estimates = [] for _ in range(n_simulations): # Generate sample data = np.random.normal(true_mu, np.sqrt(true_sigma2), size=n) # MLE variance (divide by n) mle_var = np.mean((data - np.mean(data))**2) # Unbiased variance (divide by n-1) unbiased_var = np.var(data, ddof=1) mle_estimates.append(mle_var) unbiased_estimates.append(unbiased_var) results.append({ 'n': n, 'mle_mean': np.mean(mle_estimates), 'unbiased_mean': np.mean(unbiased_estimates), 'expected_mle': (n-1)/n * true_sigma2, 'mle_bias': np.mean(mle_estimates) - true_sigma2, 'unbiased_bias': np.mean(unbiased_estimates) - true_sigma2 }) # Print results print("Bias of Variance Estimators") print("=" * 70) print(f"True variance: σ² = {true_sigma2}") print(f"Number of simulations: {n_simulations}") print() print(f"{'n':>6} | {'E[MLE]':>10} | {'Theory':>10} | {'E[Unbiased]':>12} | {'MLE Bias':>10}") print("-" * 70) for r in results: print(f"{r['n']:>6} | {r['mle_mean']:>10.4f} | {r['expected_mle']:>10.4f} | " f"{r['unbiased_mean']:>12.4f} | {r['mle_bias']:>10.4f}") # Visualize fig, axes = plt.subplots(1, 2, figsize=(14, 5)) # Left: Expected values vs sample size ax1 = axes[0] ns = [r['n'] for r in results] ax1.plot(ns, [r['mle_mean'] for r in results], 'bo-', label='MLE (biased)', markersize=8) ax1.plot(ns, [r['unbiased_mean'] for r in results], 'gs-', label='Sample variance (unbiased)', markersize=8) ax1.axhline(y=true_sigma2, color='red', linestyle='--', label=f'True σ² = {true_sigma2}') ax1.set_xlabel('Sample Size n') ax1.set_ylabel('Expected Value of Estimator') ax1.set_title('The MLE Variance Estimator is Biased') ax1.legend() ax1.grid(True, alpha=0.3) ax1.set_xscale('log') # Right: Bias magnitude ax2 = axes[1] theoretical_bias = [(n-1)/n * true_sigma2 - true_sigma2 for n in ns] empirical_bias = [r['mle_bias'] for r in results] ax2.plot(ns, theoretical_bias, 'r-', lw=2, label='Theoretical: -σ²/n') ax2.plot(ns, empirical_bias, 'bo', markersize=8, label='Empirical MLE bias') ax2.plot(ns, [r['unbiased_bias'] for r in results], 'gs', markersize=8, label='Unbiased estimator bias ≈ 0') ax2.axhline(y=0, color='gray', linestyle=':') ax2.set_xlabel('Sample Size n') ax2.set_ylabel('Bias') ax2.set_title('Bias Decreases with Sample Size') ax2.legend() ax2.grid(True, alpha=0.3) ax2.set_xscale('log') plt.tight_layout() plt.show() return results results = demonstrate_mle_variance_bias()Formal Definition of Variance
Definition:
The variance of an estimator θ̂ measures its spread around its own mean:
$$\text{Var}(\hat{\theta}) = E[(\hat{\theta} - E[\hat{\theta}])^2] = E[\hat{\theta}^2] - (E[\hat{\theta}])^2$$
Interpretation:
- High variance: Estimates fluctuate widely across different samples
- Low variance: Estimates are stable and consistent
- Variance measures precision (not accuracy—that involves bias too)
Standard Error:
The standard error is the square root of variance:
$$\text{SE}(\hat{\theta}) = \sqrt{\text{Var}(\hat{\theta})}$$
It has the same units as θ, making it more interpretable.
Example: Variance of the Sample Mean
For X₁, ..., Xₙ i.i.d. with Var(Xᵢ) = σ²:
$$\text{Var}(\bar{X}) = \text{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right)$$
$$= \frac{1}{n^2}\sum_{i=1}^n \text{Var}(X_i) = \frac{1}{n^2} \cdot n\sigma^2 = \frac{\sigma^2}{n}$$
Key insight: Variance decreases with sample size!
- n = 10: Var(X̄) = σ²/10
- n = 100: Var(X̄) = σ²/100
- n = 10000: Var(X̄) = σ²/10000
The standard error scales as 1/√n:
$$\text{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}}$$
To halve the standard error, you need 4 times as much data!
Standard error decreases as 1/√n, not 1/n. This has profound implications: going from n=100 to n=10,000 (100× more data) only reduces standard error by a factor of 10 (√100 = 10). Diminishing returns set in quickly. This is why sample size calculations are so important in experimental design.
The Mean Squared Error Decomposition
The Mean Squared Error (MSE) combines bias and variance into a single measure of estimator quality:
$$\text{MSE}(\hat{\theta}) = E[(\hat{\theta} - \theta^*)^2]$$
This measures the expected squared distance from the estimate to the true parameter.
The Bias-Variance Decomposition:
A beautiful mathematical fact is that MSE decomposes perfectly:
$$\boxed{\text{MSE}(\hat{\theta}) = \text{Bias}(\hat{\theta})^2 + \text{Var}(\hat{\theta})}$$
MSE = Bias² + Variance
Proof of the decomposition:
Let μ = E[θ̂] for brevity. Then:
$$\text{MSE} = E[(\hat{\theta} - \theta^*)^2]$$
Add and subtract μ:
$$= E[(\hat{\theta} - \mu + \mu - \theta^*)^2]$$
$$= E[(\hat{\theta} - \mu)^2] + 2E[(\hat{\theta} - \mu)(\mu - \theta^)] + (\mu - \theta^)^2$$
The middle term vanishes because E[θ̂ - μ] = 0:
$$= E[(\hat{\theta} - \mu)^2] + (\mu - \theta^*)^2$$
$$= \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2 \quad \square$$
- •Both matter: An unbiased estimator with huge variance can have worse MSE than a biased estimator with low variance.
- •Tradeoff exists: Often, introducing bias can reduce variance enough to lower overall MSE.
- •Regularization explained: Adding regularization (bias) reduces variance, potentially improving MSE.
- •Unbiased ≠ best: Being unbiased is not a guarantee of quality—MSE is often more relevant.
Imagine throwing darts at a target (the true parameter θ*). High bias = consistently missing the center in one direction. High variance = darts scattered widely. MSE measures the average squared distance from bullseye. A slightly biased but precise thrower might outperform an unbiased but erratic one!
Visualizing Bias and Variance
The dartboard analogy is so useful that it deserves a detailed exploration. Four scenarios illustrate all combinations of high/low bias and variance:
- •Darts clustered tightly around bullseye
- •Ideal estimator
- •Low MSE = Low Bias² + Low Var
- •Example: Sample mean with large n
- •Darts clustered tightly, but off-center
- •Precise but inaccurate
- •Systematic error dominates
- •Example: Heavily regularized estimator
- •Darts scattered widely, but centered
- •Accurate on average, but imprecise
- •Random error dominates
- •Example: Sample mean with small n
- •Darts scattered widely and off-center
- •Worst case: neither accurate nor precise
- •Both error sources contribute
- •Example: Poorly designed estimator
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114
import numpy as npimport matplotlib.pyplot as plt def plot_dartboard_analogy(): """ Visualize bias and variance using the dartboard analogy. """ np.random.seed(42) n_darts = 50 fig, axes = plt.subplots(2, 2, figsize=(12, 12)) scenarios = [ ('Low Bias, Low Variance(Ideal)', 0, 0.1), ('Low Bias, High Variance', 0, 0.8), ('High Bias, Low Variance', 0.7, 0.1), ('High Bias, High Variance', 0.7, 0.6), ] for ax, (title, bias, std) in zip(axes.flatten(), scenarios): # Generate dart positions x = np.random.normal(bias, std, n_darts) y = np.random.normal(bias, std, n_darts) # Draw target circles for r in [0.25, 0.5, 0.75, 1.0, 1.25]: circle = plt.Circle((0, 0), r, fill=False, color='gray', alpha=0.5) ax.add_patch(circle) # Draw bullseye bullseye = plt.Circle((0, 0), 0.1, fill=True, color='red', alpha=0.5) ax.add_patch(bullseye) # Plot darts ax.scatter(x, y, c='blue', alpha=0.6, s=50, edgecolors='darkblue') # Mark center of mass (mean) mean_x, mean_y = np.mean(x), np.mean(y) ax.scatter([mean_x], [mean_y], c='green', s=200, marker='X', edgecolors='darkgreen', linewidths=2, label='Mean', zorder=5) # Calculate actual bias and variance actual_bias = np.sqrt(mean_x**2 + mean_y**2) actual_var = np.var(x) + np.var(y) mse = np.mean(x**2 + y**2) ax.set_xlim(-1.5, 1.5) ax.set_ylim(-1.5, 1.5) ax.set_aspect('equal') ax.axhline(y=0, color='gray', linestyle='-', alpha=0.3) ax.axvline(x=0, color='gray', linestyle='-', alpha=0.3) ax.set_title(f'{title}Bias²≈{actual_bias**2:.2f}, Var≈{actual_var:.2f}, MSE≈{mse:.2f}', fontsize=11) ax.legend(loc='upper right') plt.suptitle('The Dartboard Analogy: Bias vs Variance', fontsize=14, y=1.02) plt.tight_layout() plt.show() def plot_bias_variance_tradeoff(): """ Visualize the bias-variance tradeoff. """ # Simulated example: varying regularization strength regularization = np.linspace(0, 3, 100) # Bias increases with regularization (we shrink toward prior/zero) bias_squared = 0.1 * regularization**2 # Variance decreases with regularization (estimates become more stable) variance = 2 * np.exp(-regularization) # MSE = Bias² + Variance mse = bias_squared + variance # Find optimal regularization optimal_idx = np.argmin(mse) optimal_reg = regularization[optimal_idx] fig, ax = plt.subplots(figsize=(10, 6)) ax.plot(regularization, bias_squared, 'r-', lw=2, label='Bias²') ax.plot(regularization, variance, 'b-', lw=2, label='Variance') ax.plot(regularization, mse, 'g-', lw=3, label='MSE = Bias² + Variance') ax.axvline(x=optimal_reg, color='gray', linestyle='--', label=f'Optimal λ ≈ {optimal_reg:.2f}') ax.scatter([optimal_reg], [mse[optimal_idx]], color='green', s=150, zorder=5, edgecolors='darkgreen', linewidths=2) ax.set_xlabel('Regularization Strength (λ)', fontsize=12) ax.set_ylabel('Error', fontsize=12) ax.set_title('The Bias-Variance Tradeoff', fontsize=14) ax.legend(fontsize=11) ax.grid(True, alpha=0.3) ax.set_xlim([0, 3]) ax.set_ylim([0, 2.5]) # Add annotations ax.annotate('Underfitting(High Bias)', xy=(2.5, 0.7), fontsize=10, ha='center') ax.annotate('Overfitting(High Variance)', xy=(0.3, 1.8), fontsize=10, ha='center') ax.annotate('SweetSpot', xy=(optimal_reg + 0.3, mse[optimal_idx] + 0.15), fontsize=10, ha='left') plt.tight_layout() plt.show() # Run visualizationsplot_dartboard_analogy()plot_bias_variance_tradeoff()Should We Always Prefer Unbiased Estimators?
A common misconception is that unbiased estimators are always preferable. The MSE decomposition reveals why this is wrong.
The James-Stein Paradox:
In 1961, Charles Stein proved a shocking result: when estimating the mean of a multivariate Gaussian with d ≥ 3 dimensions, the sample mean (unbiased!) is inadmissible—there exist estimators with strictly lower MSE everywhere.
The James-Stein estimator shrinks the sample mean toward zero:
$$\hat{\theta}_{JS} = \left(1 - \frac{(d-2)\sigma^2}{||\bar{X}||^2}\right)\bar{X}$$
This biased estimator has lower MSE than the unbiased sample mean for all true θ when d ≥ 3!
When biased estimators win:
- High-dimensional settings: Shrinkage estimators dominate when d >> n
- Regularization: Ridge regression (biased) often beats OLS (unbiased)
- Small samples: Bias can be tolerated if variance reduction is large
- Prediction focus: We care about prediction error, not parameter accuracy
When unbiased estimators are preferred:
- Scientific inference: When the parameter itself is of interest
- Legal/medical standards: Regulations may require unbiased estimates
- Large samples: Bias becomes dominant when variance is small
- Theoretical benchmarks: Unbiasedness simplifies analysis
| Criterion | Unbiased Estimator | Biased Estimator (e.g., Ridge) |
|---|---|---|
| Expected value | E[θ̂] = θ* | E[θ̂] ≠ θ* |
| Variance | Can be high | Usually lower |
| MSE | Bias² = 0, but Var can dominate | Bias² > 0, but Var reduction can compensate |
| Small sample performance | Often poor (high variance) | Often better (controlled variance) |
| Large sample behavior | Optimal (variance → 0) | Bias doesn't vanish (may be suboptimal) |
| Predictive accuracy | Variable | Often superior |
In modern machine learning, we rarely insist on unbiasedness. What matters is predictive performance, which is better captured by MSE or cross-validation error. Regularization techniques introduce bias precisely because the variance reduction outweighs the cost. The bias-variance tradeoff is fundamental to understanding model selection.
Bias-Variance Tradeoff in Machine Learning
The bias-variance decomposition extends beyond parameter estimation to prediction error in machine learning. This is one of the most important concepts in the field.
Setup:
- True function: f(x)
- Training data: D = {(xᵢ, yᵢ)} where yᵢ = f(xᵢ) + ε, E[ε] = 0, Var(ε) = σ²
- Learned model: f̂(x; D)
For a new point x₀, the expected squared prediction error is:
$$E[(y_0 - \hat{f}(x_0))^2] = \sigma^2 + \text{Bias}(\hat{f}(x_0))^2 + \text{Var}(\hat{f}(x_0))$$
The three components:
-
Irreducible error (σ²): Noise inherent in the data. No model can eliminate this.
-
Squared bias: How well the model can approximate the true function on average.
- Simple models (few parameters) → high bias, can't capture complex patterns
- Complex models → low bias, can fit the true function
-
Variance: How much the model changes with different training samples.
- Simple models → low variance, stable across datasets
- Complex models → high variance, sensitive to training data
The tradeoff:
As model complexity increases:
- Bias ↓ (better approximation capacity)
- Variance ↑ (more sensitive to training data)
- Total error initially decreases, then increases → U-shaped curve
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115
import numpy as npimport matplotlib.pyplot as pltfrom sklearn.preprocessing import PolynomialFeaturesfrom sklearn.linear_model import LinearRegression def bias_variance_polynomial(): """ Demonstrate bias-variance tradeoff with polynomial regression. """ np.random.seed(42) # True function def true_f(x): return np.sin(2 * np.pi * x) # Generate many training datasets n_datasets = 200 n_train = 20 noise_std = 0.3 x_test = np.linspace(0, 1, 100) y_true = true_f(x_test) degrees = [1, 3, 5, 9, 15] fig, axes = plt.subplots(2, 3, figsize=(15, 10)) for idx, degree in enumerate(degrees): ax = axes.flatten()[idx] # Store predictions from each dataset all_predictions = [] for _ in range(n_datasets): # Generate training data x_train = np.random.uniform(0, 1, n_train) y_train = true_f(x_train) + np.random.normal(0, noise_std, n_train) # Fit polynomial poly = PolynomialFeatures(degree) X_train_poly = poly.fit_transform(x_train.reshape(-1, 1)) X_test_poly = poly.transform(x_test.reshape(-1, 1)) model = LinearRegression() model.fit(X_train_poly, y_train) y_pred = model.predict(X_test_poly) all_predictions.append(y_pred) all_predictions = np.array(all_predictions) # Compute bias and variance mean_prediction = np.mean(all_predictions, axis=0) bias_squared = np.mean((mean_prediction - y_true) ** 2) variance = np.mean(np.var(all_predictions, axis=0)) mse = np.mean(np.mean((all_predictions - y_true) ** 2, axis=0)) # Plot ax.plot(x_test, y_true, 'g-', lw=2, label='True f(x)') ax.plot(x_test, mean_prediction, 'r-', lw=2, label='Mean prediction') # Plot a few individual predictions for i in range(min(10, n_datasets)): ax.plot(x_test, all_predictions[i], 'b-', alpha=0.1) ax.set_title(f'Degree {degree}Bias²={bias_squared:.3f}, Var={variance:.3f}MSE≈{mse:.3f}') ax.set_xlabel('x') ax.set_ylabel('y') ax.legend(fontsize=8) ax.set_ylim([-2, 2]) # Summary plot ax_summary = axes.flatten()[5] degrees_all = range(1, 16) bias_sq_all = [] var_all = [] mse_all = [] for degree in degrees_all: all_preds = [] for _ in range(n_datasets): x_train = np.random.uniform(0, 1, n_train) y_train = true_f(x_train) + np.random.normal(0, noise_std, n_train) poly = PolynomialFeatures(degree) X_train_poly = poly.fit_transform(x_train.reshape(-1, 1)) X_test_poly = poly.transform(x_test.reshape(-1, 1)) model = LinearRegression() model.fit(X_train_poly, y_train) all_preds.append(model.predict(X_test_poly)) all_preds = np.array(all_preds) mean_pred = np.mean(all_preds, axis=0) bias_sq_all.append(np.mean((mean_pred - y_true) ** 2)) var_all.append(np.mean(np.var(all_preds, axis=0))) mse_all.append(np.mean(np.mean((all_preds - y_true) ** 2, axis=0))) ax_summary.plot(degrees_all, bias_sq_all, 'r-', lw=2, label='Bias²') ax_summary.plot(degrees_all, var_all, 'b-', lw=2, label='Variance') ax_summary.plot(degrees_all, mse_all, 'g-', lw=3, label='MSE') ax_summary.axhline(y=noise_std**2, color='gray', linestyle='--', label='Noise²') ax_summary.set_xlabel('Polynomial Degree') ax_summary.set_ylabel('Error') ax_summary.set_title('Bias-Variance Tradeoff Summary') ax_summary.legend() ax_summary.set_xlim([1, 15]) ax_summary.grid(True, alpha=0.3) plt.tight_layout() plt.show() bias_variance_polynomial()KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.