We've seen that estimators have bias and variance—they're not perfect. But here's a comforting thought: with enough data, good estimators get arbitrarily close to the truth.

Consistency formalizes this intuition. It's the guarantee that as we gather more observations, our uncertainty vanishes and we can estimate parameters with arbitrary precision. Without consistency, all the data in the world might not help us—a deeply unsatisfying property for any estimation procedure.

This page explores what consistency means, when it holds, and why it's fundamental to statistical learning theory.

Definition: Consistency

An estimator θ̂ₙ (based on n observations) is consistent for θ* if it converges to θ* in probability as n → ∞:

$$\hat{\theta}_n \xrightarrow{P} \theta^*$$

Formally, for any ε > 0:

$$\lim_{n \to \infty} P(|\hat{\theta}_n - \theta^*| > \epsilon) = 0$$

Interpretation:

As we collect more data, the probability of being "far" from the true value (by more than any fixed ε) shrinks to zero. We can make our estimate arbitrarily accurate with enough observations.

Types of convergence:

There are actually several modes of convergence, from weakest to strongest:

1. Convergence in probability (what we're discussing): $$P(|\hat{\theta}_n - \theta^*| > \epsilon) \to 0$$

2. Almost sure convergence (stronger): $$P(\lim_{n \to \infty} \hat{\theta}_n = \theta^*) = 1$$

3. Mean squared convergence (L² convergence): $$E[(\hat{\theta}_n - \theta^*)^2] \to 0$$

For practical purposes, convergence in probability (consistency) is usually sufficient. Mean squared convergence (MSE → 0) is often easier to prove and implies consistency.

Consistency and unbiasedness are often confused. They are independent properties—neither implies the other!

Four possibilities:

Example 1: Unbiased but Inconsistent

Consider estimating the population mean μ using only the first observation:

$$\hat{\mu} = X_1$$

This is unbiased: E[X₁] = μ.

But it's inconsistent! No matter how many observations we collect, we only use X₁. The variance Var(X₁) = σ² never decreases. The estimator doesn't improve with more data.

Example 2: Biased but Consistent

The MLE for Gaussian variance:

$$\hat{\sigma}^2_{MLE} = \frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^2$$

This is biased: E[σ̂²] = (n-1)/n × σ² ≠ σ².

But it's consistent:

• Bias = -σ²/n → 0 as n → ∞
• Variance → 0 as n → ∞
• Therefore MSE → 0, so consistent!

The Law of Large Numbers (LLN) is the fundamental theorem underlying consistency.

Weak Law of Large Numbers (WLLN):

For i.i.d. random variables X₁, ..., Xₙ with E[Xᵢ] = μ:

$$\bar{X}n = \frac{1}{n}\sum{i=1}^n X_i \xrightarrow{P} \mu$$

The sample mean converges in probability to the population mean.

Strong Law of Large Numbers (SLLN):

Under the same conditions:

$$P\left(\lim_{n \to \infty} \bar{X}_n = \mu\right) = 1$$

The sample mean converges almost surely (with probability 1).

Proof sketch (WLLN via Chebyshev):

By Chebyshev's inequality:

$$P(|\bar{X}_n - \mu| > \epsilon) \leq \frac{\text{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}$$

As n → ∞, the right side → 0, proving convergence in probability.

Why LLN matters for estimation:

Many estimators can be written as sample averages. For example:

• Sample mean: X̄ = (1/n)Σ Xᵢ
• Sample variance: s² = (1/n)Σ(Xᵢ - X̄)² (approximately)
• Empirical probability: P̂(A) = (1/n)Σ 1{Xᵢ ∈ A}
• MLE for many models (via first-order conditions)

The LLN immediately gives consistency for all these!

Let's rigorously prove consistency of the sample mean using the MSE approach.

Claim: The sample mean X̄ₙ = (1/n)Σᵢ₌₁ⁿ Xᵢ is a consistent estimator of μ = E[X].

Proof:

We'll show MSE(X̄ₙ) → 0, which implies consistency.

Step 1: Compute Bias

$$\text{Bias}(\bar{X}_n) = E[\bar{X}_n] - \mu = \mu - \mu = 0$$

The sample mean is unbiased for all n, so Bias = 0.

Step 2: Compute Variance

$$\text{Var}(\bar{X}n) = \text{Var}\left(\frac{1}{n}\sum{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i=1}^n \text{Var}(X_i) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}$$

Step 3: Compute MSE

$$\text{MSE}(\bar{X}_n) = \text{Bias}^2 + \text{Var} = 0 + \frac{\sigma^2}{n} = \frac{\sigma^2}{n}$$

Step 4: Take limit

$$\lim_{n \to \infty} \text{MSE}(\bar{X}n) = \lim{n \to \infty} \frac{\sigma^2}{n} = 0$$

Conclusion: MSE → 0 implies X̄ₙ →ᵖ μ, so the sample mean is consistent. ∎

One of the most important results in statistical theory is that MLEs are consistent under mild regularity conditions.

Theorem (MLE Consistency):

Under regularity conditions, the MLE θ̂ₙ is consistent for θ*:

$$\hat{\theta}_{MLE} \xrightarrow{P} \theta^*$$

Intuition:

MLE maximizes the log-likelihood:

$$\hat{\theta}{MLE} = \arg\max\theta \frac{1}{n}\sum_{i=1}^n \log P(X_i | \theta)$$

By the Law of Large Numbers:

$$\frac{1}{n}\sum_{i=1}^n \log P(X_i | \theta) \xrightarrow{P} E_{X \sim P(\cdot|\theta^*)}[\log P(X | \theta)]$$

The expectation E[log P(X|θ)] (taken over the true distribution P(X|θ*)) is maximized at θ = θ*. This is because:

$$E[\log P(X|\theta)] - E[\log P(X|\theta^)] = -D_{KL}(P(\cdot|\theta^) || P(\cdot|\theta)) \leq 0$$

KL divergence is non-negative, with equality iff θ = θ*.

Regularity conditions:

MLE consistency requires:

1. Identifiability: Different θ give different distributions. If P(X|θ₁) = P(X|θ₂) for all X, we can't distinguish θ₁ from θ₂.

2. Correct specification: The true data-generating process is in the model family. If reality is N(μ, σ²) but we fit Exponential(λ), MLE won't find the "true" parameters.

3. Compact parameter space or proper behavior at boundaries.

4. Smoothness: The log-likelihood is sufficiently smooth (differentiable, etc.).

5. Dominated convergence: Technical conditions for the LLN to apply uniformly.

Consistency tells us that estimators converge to the truth. Asymptotic normality tells us how they converge—the shape of the sampling distribution for large n.

Central Limit Theorem (CLT):

For i.i.d. X₁, ..., Xₙ with mean μ and variance σ²:

$$\sqrt{n}(\bar{X}_n - \mu) \xrightarrow{d} N(0, \sigma^2)$$

Or equivalently:

$$\bar{X}_n \approx N\left(\mu, \frac{\sigma^2}{n}\right)$$

The sample mean becomes approximately Gaussian, regardless of the original distribution!

Asymptotic normality of MLE:

Under regularity conditions:

$$\sqrt{n}(\hat{\theta}_{MLE} - \theta^) \xrightarrow{d} N(0, I(\theta^)^{-1})$$

where I(θ) is the Fisher Information.

This means MLE is approximately:

$$\hat{\theta}_{MLE} \approx N\left(\theta^, \frac{1}{nI(\theta^)}\right)$$

Why asymptotic normality matters:

1. Confidence intervals: We can construct approximate 95% CIs: $$\hat{\theta} \pm 1.96 \times \frac{\text{SE}(\hat{\theta})}{\sqrt{n}}$$

2. Hypothesis testing: Standard tests (t-tests, z-tests) are justified

3. Efficiency comparison: The Fisher Information determines the "best" variance achievable

4. Universal behavior: Regardless of the underlying distribution, MLEs behave like Gaussians for large n

The rate of convergence:

Note the √n factor. This means:

• Variance of MLE ∝ 1/n
• Standard error ∝ 1/√n
• To halve the standard error: need 4× data

Consistency has profound practical implications for machine learning and statistics:

1. Justification for MLE/MAP:

Consistency tells us that with enough data, MLE will find the truth (assuming correct model specification). This justifies using MLE even when we don't have closed-form solutions—we know we're converging to the right answer.

2. Model selection:

Consistent model selection criteria (like BIC) converge to selecting the true model with probability 1 as n → ∞. This gives theoretical backing to complexity penalties.

3. Cross-validation:

Leave-one-out and k-fold CV are consistent estimators of prediction error. With enough data, they correctly identify the best model.

4. Regularization tuning:

Optimal regularization strength typically decreases with n (we need less shrinkage with more data), consistent with the Bayesian interpretation where data overwhelms the prior.

5. Sample size planning:

Understanding that SE ∝ 1/√n guides experimental design:

• Current SE = 0.1
• Desired SE = 0.05 (half)
• Required n = 4 × current n

6. Convergence diagnostics:

When monitoring training, we expect:

• Training error → Bayes optimal error
• Validation error → 0 (for consistent procedures on realizable problems)
• Gap between training and validation → 0

7. Limitations:

• Consistency is asymptotic—no guarantees for finite n
• Misspecified models: MLE converges to best approximation, not truth
• Non-identifiable models: MLE may not converge at all

When working with a new estimator, how do we determine if it's consistent? Here's a toolkit:

Method 1: MSE Approach

Show that:

• Bias(θ̂ₙ) → 0 as n → ∞
• Var(θ̂ₙ) → 0 as n → ∞

Then MSE = Bias² + Var → 0, implying consistency.

Method 2: Direct Application of LLN

If θ̂ₙ can be written as a sample average (or continuous function of sample averages), LLN applies directly.

Method 3: Contraction Argument

Show that as n increases, the estimator stays within a shrinking ball around θ* with high probability.

Method 4: Simulation Study

Empirical verification:

1. Set true parameter θ*
2. For various sample sizes n:
   • Generate many datasets
   • Compute θ̂ₙ for each
   • Check if mean(θ̂ₙ) → θ* and std(θ̂ₙ) → 0
3. Plot convergence

Method 5: Literature Search

For standard estimators (MLE, method of moments), consistency results are well-established under regularity conditions. Check if your problem satisfies these conditions.

We've explored the asymptotic property that guarantees our estimates improve with more data. Let's consolidate:

Looking ahead:

Consistency tells us we'll eventually get it right. But among consistent estimators, some converge faster than others. The next page introduces Efficiency—the property that characterizes the best possible convergence rate and the estimators that achieve it.