We've established that good estimators are consistent (converge to truth) and have low MSE (bias + variance). But among all consistent, unbiased estimators, is there a best one? Is there a fundamental limit to how precise our estimates can be?

The answer is yes, and it leads to one of the most beautiful results in statistics: the Cramér-Rao Lower Bound. This bound sets a floor on the variance of any unbiased estimator, and estimators that achieve this floor are called efficient.

Efficiency tells us when we're extracting the maximum possible information from our data—when no other unbiased estimator could do better.

The Fisher Information measures how much information an observation carries about the parameter θ. It's the key quantity that determines the precision limits of estimation.

Definition 1 (Score function):

The score function is the gradient of the log-likelihood:

$$S(\theta) = \frac{\partial}{\partial\theta} \log P(X | \theta)$$

Definition 2 (Fisher Information):

The Fisher Information is the variance of the score:

$$I(\theta) = \text{Var}\left[\frac{\partial}{\partial\theta} \log P(X | \theta)\right] = E\left[\left(\frac{\partial}{\partial\theta} \log P(X | \theta)\right)^2\right]$$

Under regularity conditions, an equivalent form is:

$$I(\theta) = -E\left[\frac{\partial^2}{\partial\theta^2} \log P(X | \theta)\right]$$

The second form is often easier to compute.

Intuition:

The score function S(θ) measures how "sensitive" the log-likelihood is to changes in θ. High sensitivity means we can distinguish between nearby parameter values—the data is informative.

• High Fisher Information: The log-likelihood is sharply curved; observations strongly distinguish between nearby θ values.
• Low Fisher Information: The log-likelihood is flat; observations barely distinguish different θ values.

Geometric interpretation:

Fisher Information is the expected curvature (negative second derivative) of the log-likelihood at the true parameter. Sharp curves mean precise estimation; flat curves mean imprecise estimation.

Additivity of Fisher Information:

For n i.i.d. observations, the total Fisher Information is:

$$I_n(\theta) = n \cdot I_1(\theta)$$

Information adds linearly! This is why more data allows more precise estimation—we accumulate information.

Example: Bernoulli Fisher Information

For X ~ Bernoulli(p):

$$\log P(X | p) = X \log p + (1-X) \log(1-p)$$

$$\frac{\partial}{\partial p} \log P = \frac{X}{p} - \frac{1-X}{1-p}$$

$$\frac{\partial^2}{\partial p^2} \log P = -\frac{X}{p^2} - \frac{1-X}{(1-p)^2}$$

$$I(p) = -E\left[-\frac{X}{p^2} - \frac{1-X}{(1-p)^2}\right] = \frac{p}{p^2} + \frac{1-p}{(1-p)^2} = \frac{1}{p(1-p)}$$

The Cramér-Rao Lower Bound (CRLB) is one of the most important results in estimation theory. It sets a fundamental limit on estimator precision.

Theorem (Cramér-Rao Lower Bound):

For any unbiased estimator θ̂ of θ:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$$

For n i.i.d. observations:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{nI_1(\theta)}$$

Interpretation:

No unbiased estimator can have variance lower than 1/(nI). The Fisher Information sets a speed limit on estimation precision. This is a fundamental lower bound—not a constraint we choose, but one that's mathematically inescapable.

Proof sketch:

The proof uses the Cauchy-Schwarz inequality. Define the score S(θ) = ∂/∂θ log P(X|θ).

Key facts:

1. E[S(θ)] = 0 (regularity condition)
2. Cov(θ̂, S) = 1 for unbiased θ̂ (can be shown)
3. Var(S) = I(θ)

By Cauchy-Schwarz: $$1 = |\text{Cov}(\hat{\theta}, S)|^2 \leq \text{Var}(\hat{\theta}) \cdot \text{Var}(S) = \text{Var}(\hat{\theta}) \cdot I(\theta)$$

Rearranging: Var(θ̂) ≥ 1/I(θ). ∎

An estimator is efficient if it achieves the Cramér-Rao Lower Bound with equality:

$$\text{Var}(\hat{\theta}) = \frac{1}{nI(\theta)}$$

When does efficiency occur?

The CRLB is achieved when the Cauchy-Schwarz inequality becomes an equality, which happens iff θ̂ and S(θ) are perfectly correlated—specifically, when:

$$\hat{\theta} - \theta = \frac{S(\theta)}{I(\theta)}$$

This is equivalent to the score being a linear function of the estimator.

The Exponential Family:

Efficient estimators exist precisely for the exponential family of distributions:

$$P(x | \theta) = h(x) \exp(\eta(\theta) T(x) - A(\theta))$$

Examples: Bernoulli, Gaussian, Exponential, Poisson, Gamma, Beta.

For these distributions, the MLE achieves the CRLB exactly.

Example: Sample Mean for Gaussian

For X₁, ..., Xₙ ~ N(μ, σ²) with known σ²:

• Fisher Information: I(μ) = 1/σ²
• CRLB: Var(μ̂) ≥ σ²/n
• Sample mean: Var(X̄) = σ²/n ✓

The sample mean achieves the bound—it's efficient!

Example: MLE for Bernoulli

For X₁, ..., Xₙ ~ Bernoulli(p):

• Fisher Information: I(p) = 1/(p(1-p))
• CRLB: Var(p̂) ≥ p(1-p)/n
• MLE: p̂ = X̄ with Var(p̂) = p(1-p)/n ✓

The sample proportion is efficient!

One of the most important results in estimation theory:

Theorem (Asymptotic Efficiency of MLE):

Under regularity conditions, the MLE is asymptotically efficient:

$$\sqrt{n}(\hat{\theta}_{MLE} - \theta^) \xrightarrow{d} N(0, I(\theta^)^{-1})$$

The asymptotic variance of √n·(θ̂ - θ*) is exactly 1/I(θ*), the Cramér-Rao bound.

What this means:

1. For large n, MLE is approximately unbiased
2. Its variance approaches the minimum possible (CRLB)
3. No other consistent estimator can have asymptotically lower variance

MLE is the best you can do asymptotically (among regular estimators).

Why MLE achieves asymptotic efficiency:

Recall MLE maximizes:

$$\frac{1}{n}\sum_{i=1}^n \log P(X_i | \theta)$$

The first-order condition is:

$$\frac{1}{n}\sum_{i=1}^n \frac{\partial}{\partial\theta}\log P(X_i | \theta) = 0$$

This is a sample average of the score—by CLT, it's asymptotically normal. A Taylor expansion around θ* connects the MLE to this average, yielding the result.

The remarkable universality:

Regardless of the specific model (Bernoulli, Gaussian, complex neural network), if you use MLE (or equivalently, minimize negative log-likelihood), you get the most precise estimates possible for large samples.

When an estimator doesn't achieve the CRLB, we can still quantify how close it comes using relative efficiency.

Definition:

The relative efficiency of estimator θ̂₁ compared to θ̂₂ is:

$$\text{RE}(\hat{\theta}_1, \hat{\theta}_2) = \frac{\text{Var}(\hat{\theta}_2)}{\text{Var}(\hat{\theta}_1)}$$

Absolute efficiency is the ratio of CRLB to actual variance:

$$e(\hat{\theta}) = \frac{1/(nI(\theta))}{\text{Var}(\hat{\theta})} = \frac{\text{CRLB}}{\text{Var}(\hat{\theta})}$$

• e(θ̂) = 1: Efficient (achieves CRLB)
• e(θ̂) < 1: Inefficient (some information wasted)
• Relative efficiency > 1: First estimator is better than second

Example: Mean vs. Median for Gaussian

For N(μ, σ²), both the sample mean and median are consistent estimators of μ.

• Sample mean: Var(X̄) = σ²/n (efficient)
• Sample median: Var(median) ≈ (π/2) × σ²/n for large n

Relative efficiency: $$\text{RE}(\bar{X}, \text{median}) = \frac{(\pi/2) \cdot \sigma^2/n}{\sigma^2/n} = \frac{\pi}{2} \approx 1.57$$

The mean is 57% more efficient than the median for Gaussian data.

Interpretation: You need 57% more data using the median to achieve the same precision as the mean.

But for heavy-tailed distributions, the median can be more efficient due to robustness!

Fisher Information has modern applications beyond classical statistics:

1. Natural Gradient Descent:

Instead of standard gradient descent: $$\theta \leftarrow \theta - \alpha \nabla_{\theta} L(\theta)$$

Natural gradient uses the Fisher Information Matrix (FIM): $$\theta \leftarrow \theta - \alpha F(\theta)^{-1} \nabla_{\theta} L(\theta)$$

The FIM accounts for the geometry of probability distributions, leading to faster, more stable convergence.

2. Elastic Weight Consolidation (EWC):

For continual learning (avoiding catastrophic forgetting), EWC uses Fisher Information to identify which weights are important for previous tasks:

$$L_{EWC} = L_{new} + \frac{\lambda}{2}\sum_i F_i (\theta_i - \theta_{old,i})^2$$

Weights with high Fisher Information (important for past tasks) are penalized more for changing.

3. Information Geometry:

The space of probability distributions can be viewed as a Riemannian manifold with the FIM as the metric tensor. This gives a geometric interpretation to learning:

• KL divergence is approximately (1/2)Δθᵀ F Δθ for nearby θ
• Natural gradient follows geodesics on this manifold
• Learning rate adaptation (Adam, etc.) implicitly approximates FIM

4. Neural Network Compression:

Fisher Information helps identify parameters whose values can be changed with minimal impact on predictions—candidates for pruning or quantization.

5. Uncertainty Quantification:

The inverse FIM provides approximate posterior uncertainty (Laplace approximation): $$P(\theta | D) \approx N(\hat{\theta}_{MLE}, F^{-1})$$

Useful for Bayesian neural networks and uncertainty estimation.

Can we ever do better than the Cramér-Rao bound? The answer is subtle.

Hodges' Superefficient Estimator:

Consider estimating μ from N(μ, 1). Define:

$$\hat{\mu}_H = \begin{cases} \bar{X} & \text{if } |\bar{X}| > n^{-1/4} \ 0 & \text{otherwise} \end{cases}$$

This estimator:

• Is consistent: μ̂_H → μ as n → ∞
• Has Var(μ̂_H) → 0 as n → ∞ (asymptotically unbiased)
• At μ = 0: Var(μ̂_H) = o(1/n)—beats the CRLB!

This seems to violate our bound. What's happening?

The resolution:

The CRLB applies to estimators that are unbiased at all θ values. Hodges' estimator:

• Is biased for μ ≠ 0 in finite samples
• Achieves superefficiency only at isolated points (μ = 0)
• Has worse performance for μ near (but not at) 0

The Convolution Theorem (Le Cam) shows that:

• Superefficiency can only occur on sets of measure zero
• No estimator can beat the CRLB uniformly
• Superefficiency at one point implies suboptimality nearby

Practical implications:

Superefficient estimators are curiosities, not tools. You can't exploit them without knowing the true parameter in advance. MLE remains the practical choice.

Understanding efficiency guides practical statistical work:

1. Sample Size Planning:

Using the CRLB, we can determine required sample sizes:

"I need standard error ≤ 0.01 for estimating proportion p ≈ 0.3"

$$\text{SE} = \sqrt{\frac{p(1-p)}{n}} \leq 0.01$$ $$n \geq \frac{0.3 \times 0.7}{0.01^2} = 2100$$

2. Experimental Design:

Fisher Information can optimize data collection:

• Which features to measure?
• Where to sample in input space?
• How to allocate resources across conditions?

Maximize information subject to constraints.

3. Model Selection:

Among models with similar predictive power, prefer those whose MLE achieves near-efficiency. This indicates the model is well-suited to the data structure.

4. Uncertainty Quantification:

For efficient estimators:

$$\text{Approximate 95% CI} = \hat{\theta} \pm 1.96 \times \frac{1}{\sqrt{nI(\hat{\theta})}}$$

The width is determined by Fisher Information—intrinsic to the problem.

5. Comparing Methods:

If method A has 80% efficiency compared to MLE, you need 25% more data to match MLE precision. This quantifies the cost of using simpler methods.

6. Regularization Tradeoffs:

Regularized estimators are biased, so CRLB doesn't directly apply. But the MSE framework applies: regularization typically increases bias, decreases variance, and can improve MSE despite losing efficiency in the classical sense.

We've explored the ultimate limits of estimation precision. Let's consolidate:

Module Complete:

You've now mastered Estimation Theory—from the mechanics of MLE and MAP to the theoretical properties that guarantee their quality. You understand:

• MLE: Maximize likelihood, the workhorse of parameter estimation
• MAP: Add priors for regularization and robustness
• Bias/Variance: Two sources of error, with tradeoffs between them
• Consistency: Asymptotic convergence to truth
• Efficiency: Optimal finite-sample precision

These concepts form the theoretical foundation for nearly all statistical and machine learning methods. Whether fitting a logistic regression or training a billion-parameter language model, you're applying these principles.