Efficiency of Estimators

We've established that good estimators are consistent (converge to truth) and have low MSE (bias + variance). But among all consistent, unbiased estimators, is there a best one? Is there a fundamental limit to how precise our estimates can be?

The answer is yes, and it leads to one of the most beautiful results in statistics: the Cramér-Rao Lower Bound. This bound sets a floor on the variance of any unbiased estimator, and estimators that achieve this floor are called efficient.

Efficiency tells us when we're extracting the maximum possible information from our data—when no other unbiased estimator could do better.

The Fisher Information measures how much information an observation carries about the parameter θ. It's the key quantity that determines the precision limits of estimation.

Definition 1 (Score function):

The score function is the gradient of the log-likelihood:

$$S(\theta) = \frac{\partial}{\partial\theta} \log P(X | \theta)$$

Definition 2 (Fisher Information):

The Fisher Information is the variance of the score:

$$I(\theta) = \text{Var}\left[\frac{\partial}{\partial\theta} \log P(X | \theta)\right] = E\left[\left(\frac{\partial}{\partial\theta} \log P(X | \theta)\right)^2\right]$$

Under regularity conditions, an equivalent form is:

$$I(\theta) = -E\left[\frac{\partial^2}{\partial\theta^2} \log P(X | \theta)\right]$$

The second form is often easier to compute.

Intuition:

The score function S(θ) measures how "sensitive" the log-likelihood is to changes in θ. High sensitivity means we can distinguish between nearby parameter values—the data is informative.

• High Fisher Information: The log-likelihood is sharply curved; observations strongly distinguish between nearby θ values.
• Low Fisher Information: The log-likelihood is flat; observations barely distinguish different θ values.

Geometric interpretation:

Fisher Information is the expected curvature (negative second derivative) of the log-likelihood at the true parameter. Sharp curves mean precise estimation; flat curves mean imprecise estimation.

Additivity of Fisher Information:

For n i.i.d. observations, the total Fisher Information is:

$$I_n(\theta) = n \cdot I_1(\theta)$$

Information adds linearly! This is why more data allows more precise estimation—we accumulate information.

Example: Bernoulli Fisher Information

For X ~ Bernoulli(p):

$$\log P(X | p) = X \log p + (1-X) \log(1-p)$$

$$\frac{\partial}{\partial p} \log P = \frac{X}{p} - \frac{1-X}{1-p}$$

$$\frac{\partial^2}{\partial p^2} \log P = -\frac{X}{p^2} - \frac{1-X}{(1-p)^2}$$

$$I(p) = -E\left[-\frac{X}{p^2} - \frac{1-X}{(1-p)^2}\right] = \frac{p}{p^2} + \frac{1-p}{(1-p)^2} = \frac{1}{p(1-p)}$$

The Cramér-Rao Lower Bound (CRLB) is one of the most important results in estimation theory. It sets a fundamental limit on estimator precision.

Theorem (Cramér-Rao Lower Bound):

For any unbiased estimator θ̂ of θ:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$$

For n i.i.d. observations:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{nI_1(\theta)}$$

Interpretation:

No unbiased estimator can have variance lower than 1/(nI). The Fisher Information sets a speed limit on estimation precision. This is a fundamental lower bound—not a constraint we choose, but one that's mathematically inescapable.

Proof sketch:

The proof uses the Cauchy-Schwarz inequality. Define the score S(θ) = ∂/∂θ log P(X|θ).

Key facts:

1. E[S(θ)] = 0 (regularity condition)
2. Cov(θ̂, S) = 1 for unbiased θ̂ (can be shown)
3. Var(S) = I(θ)

By Cauchy-Schwarz: $$1 = |\text{Cov}(\hat{\theta}, S)|^2 \leq \text{Var}(\hat{\theta}) \cdot \text{Var}(S) = \text{Var}(\hat{\theta}) \cdot I(\theta)$$

Rearranging: Var(θ̂) ≥ 1/I(θ). ∎

An estimator is efficient if it achieves the Cramér-Rao Lower Bound with equality:

$$\text{Var}(\hat{\theta}) = \frac{1}{nI(\theta)}$$

When does efficiency occur?

The CRLB is achieved when the Cauchy-Schwarz inequality becomes an equality, which happens iff θ̂ and S(θ) are perfectly correlated—specifically, when:

$$\hat{\theta} - \theta = \frac{S(\theta)}{I(\theta)}$$

This is equivalent to the score being a linear function of the estimator.

The Exponential Family:

Efficient estimators exist precisely for the exponential family of distributions:

$$P(x | \theta) = h(x) \exp(\eta(\theta) T(x) - A(\theta))$$

Examples: Bernoulli, Gaussian, Exponential, Poisson, Gamma, Beta.

For these distributions, the MLE achieves the CRLB exactly.

Example: Sample Mean for Gaussian

For X₁, ..., Xₙ ~ N(μ, σ²) with known σ²:

• Fisher Information: I(μ) = 1/σ²
• CRLB: Var(μ̂) ≥ σ²/n
• Sample mean: Var(X̄) = σ²/n ✓

The sample mean achieves the bound—it's efficient!

Example: MLE for Bernoulli

For X₁, ..., Xₙ ~ Bernoulli(p):

• Fisher Information: I(p) = 1/(p(1-p))
• CRLB: Var(p̂) ≥ p(1-p)/n
• MLE: p̂ = X̄ with Var(p̂) = p(1-p)/n ✓

The sample proportion is efficient!

One of the most important results in estimation theory:

Theorem (Asymptotic Efficiency of MLE):

Under regularity conditions, the MLE is asymptotically efficient:

$$\sqrt{n}(\hat{\theta}_{MLE} - \theta^) \xrightarrow{d} N(0, I(\theta^)^{-1})$$

The asymptotic variance of √n·(θ̂ - θ*) is exactly 1/I(θ*), the Cramér-Rao bound.

What this means:

1. For large n, MLE is approximately unbiased
2. Its variance approaches the minimum possible (CRLB)
3. No other consistent estimator can have asymptotically lower variance

MLE is the best you can do asymptotically (among regular estimators).

Why MLE achieves asymptotic efficiency:

Recall MLE maximizes:

$$\frac{1}{n}\sum_{i=1}^n \log P(X_i | \theta)$$

The first-order condition is:

$$\frac{1}{n}\sum_{i=1}^n \frac{\partial}{\partial\theta}\log P(X_i | \theta) = 0$$

This is a sample average of the score—by CLT, it's asymptotically normal. A Taylor expansion around θ* connects the MLE to this average, yielding the result.

The remarkable universality:

Regardless of the specific model (Bernoulli, Gaussian, complex neural network), if you use MLE (or equivalently, minimize negative log-likelihood), you get the most precise estimates possible for large samples.

When an estimator doesn't achieve the CRLB, we can still quantify how close it comes using relative efficiency.

Definition:

The relative efficiency of estimator θ̂₁ compared to θ̂₂ is:

$$\text{RE}(\hat{\theta}_1, \hat{\theta}_2) = \frac{\text{Var}(\hat{\theta}_2)}{\text{Var}(\hat{\theta}_1)}$$

Absolute efficiency is the ratio of CRLB to actual variance:

$$e(\hat{\theta}) = \frac{1/(nI(\theta))}{\text{Var}(\hat{\theta})} = \frac{\text{CRLB}}{\text{Var}(\hat{\theta})}$$

• e(θ̂) = 1: Efficient (achieves CRLB)
• e(θ̂) < 1: Inefficient (some information wasted)
• Relative efficiency > 1: First estimator is better than second

Example: Mean vs. Median for Gaussian

For N(μ, σ²), both the sample mean and median are consistent estimators of μ.

• Sample mean: Var(X̄) = σ²/n (efficient)
• Sample median: Var(median) ≈ (π/2) × σ²/n for large n

Relative efficiency: $$\text{RE}(\bar{X}, \text{median}) = \frac{(\pi/2) \cdot \sigma^2/n}{\sigma^2/n} = \frac{\pi}{2} \approx 1.57$$

The mean is 57% more efficient than the median for Gaussian data.

Interpretation: You need 57% more data using the median to achieve the same precision as the mean.

But for heavy-tailed distributions, the median can be more efficient due to robustness!

Fisher Information has modern applications beyond classical statistics:

1. Natural Gradient Descent:

Instead of standard gradient descent: $$\theta \leftarrow \theta - \alpha abla_{\theta} L(\theta)$$

Natural gradient uses the Fisher Information Matrix (FIM): $$\theta \leftarrow \theta - \alpha F(\theta)^{-1} abla_{\theta} L(\theta)$$

The FIM accounts for the geometry of probability distributions, leading to faster, more stable convergence.

2. Elastic Weight Consolidation (EWC):

For continual learning (avoiding catastrophic forgetting), EWC uses Fisher Information to identify which weights are important for previous tasks:

$$L_{EWC} = L_{new} + \frac{\lambda}{2}\sum_i F_i (\theta_i - \theta_{old,i})^2$$

Weights with high Fisher Information (important for past tasks) are penalized more for changing.

3. Information Geometry:

The space of probability distributions can be viewed as a Riemannian manifold with the FIM as the metric tensor. This gives a geometric interpretation to learning:

• KL divergence is approximately (1/2)Δθᵀ F Δθ for nearby θ
• Natural gradient follows geodesics on this manifold
• Learning rate adaptation (Adam, etc.) implicitly approximates FIM

4. Neural Network Compression:

Fisher Information helps identify parameters whose values can be changed with minimal impact on predictions—candidates for pruning or quantization.

5. Uncertainty Quantification:

The inverse FIM provides approximate posterior uncertainty (Laplace approximation): $$P(\theta | D) \approx N(\hat{\theta}_{MLE}, F^{-1})$$

Useful for Bayesian neural networks and uncertainty estimation.

Can we ever do better than the Cramér-Rao bound? The answer is subtle.

Hodges' Superefficient Estimator:

Consider estimating μ from N(μ, 1). Define:

$$\hat{\mu}_H = \begin{cases} \bar{X} & \text{if } |\bar{X}| > n^{-1/4} \ 0 & \text{otherwise} \end{cases}$$

This estimator:

• Is consistent: μ̂_H → μ as n → ∞
• Has Var(μ̂_H) → 0 as n → ∞ (asymptotically unbiased)
• At μ = 0: Var(μ̂_H) = o(1/n)—beats the CRLB!

This seems to violate our bound. What's happening?

The resolution:

The CRLB applies to estimators that are unbiased at all θ values. Hodges' estimator:

• Is biased for μ ≠ 0 in finite samples
• Achieves superefficiency only at isolated points (μ = 0)
• Has worse performance for μ near (but not at) 0

The Convolution Theorem (Le Cam) shows that:

• Superefficiency can only occur on sets of measure zero
• No estimator can beat the CRLB uniformly
• Superefficiency at one point implies suboptimality nearby

Practical implications:

Superefficient estimators are curiosities, not tools. You can't exploit them without knowing the true parameter in advance. MLE remains the practical choice.