Statistical Properties of OLS

The Gauss-Markov theorem declares OLS the Best Linear Unbiased Estimator (BLUE). But what exactly makes an estimator 'best'? Why do we care about linearity and unbiasedness? And are there situations where being 'best' in the BLUE sense isn't actually what we want?

This page takes a deep dive into the BLUE property, examining:

• The precise mathematical meaning of each component
• Why minimum variance is the natural criterion for 'best'
• How BLUE compares to other optimality criteria
• When sacrificing unbiasedness might actually improve estimation

Let's examine each word in 'Best Linear Unbiased Estimator' with mathematical precision.

Estimator:

An estimator $\hat{\boldsymbol{\theta}}$ is a function of the observed data that produces an estimate of an unknown population parameter $\boldsymbol{\theta}$. In regression:

$$\hat{\boldsymbol{\beta}} = T(\mathbf{X}, \mathbf{y})$$

Where $T$ is some function (rule) mapping data to parameter estimates. Different choices of $T$ give different estimators—OLS is one particular choice among infinitely many.

Linear:

An estimator is linear if it can be expressed as:

$$\hat{\boldsymbol{\beta}} = \mathbf{A}\mathbf{y}$$

Where $\mathbf{A}$ is a $p \times n$ matrix that may depend on $\mathbf{X}$ but not on $\mathbf{y}$. This is a crucial restriction:

• ✅ OLS: $\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$ — Linear in $\mathbf{y}$
• ❌ LAD (Least Absolute Deviations): Depends on $\mathbf{y}$ nonlinearly through the $\text{sign}$ function
• ❌ Ridge (with data-dependent $\lambda$): If $\lambda$ depends on cross-validation of $\mathbf{y}$, it's nonlinear
• ❌ Maximum Likelihood under non-Gaussian errors: Generally nonlinear

Unbiased:

An estimator is unbiased if its expected value equals the true parameter:

$$\mathbb{E}[\hat{\boldsymbol{\beta}} | \mathbf{X}] = \boldsymbol{\beta}$$

This must hold for all possible values of $\boldsymbol{\beta}$. Intuitively, if we could repeat the sampling process infinitely many times, the average of our estimates would hit the true value exactly.

Bias is defined as:

$$\text{Bias}(\hat{\boldsymbol{\beta}}) = \mathbb{E}[\hat{\boldsymbol{\beta}}] - \boldsymbol{\beta}$$

An unbiased estimator has zero bias.

Best (Minimum Variance):

Among all estimators in the class being considered (linear and unbiased), 'best' means minimum variance. Specifically:

$$\text{Var}(\hat{\boldsymbol{\beta}}_{OLS}) \preceq \text{Var}(\tilde{\boldsymbol{\beta}}) \quad \text{for all LUE } \tilde{\boldsymbol{\beta}}$$

The $\preceq$ notation means the difference is positive semi-definite: $\text{Var}(\tilde{\boldsymbol{\beta}}) - \text{Var}(\hat{\boldsymbol{\beta}}_{OLS}) \succeq 0$.

Given that we've restricted to unbiased estimators (mean = truth), what's the natural way to rank them? The answer comes from considering Mean Squared Error (MSE).

MSE Decomposition:

For any estimator $\hat{\theta}$ of a scalar parameter $\theta$:

$$\text{MSE}(\hat{\theta}) = \mathbb{E}[(\hat{\theta} - \theta)^2] = \text{Var}(\hat{\theta}) + [\text{Bias}(\hat{\theta})]^2$$

This fundamental identity shows that estimation error comes from two sources:

1. Variance: Random fluctuation around the expected estimate
2. Bias²: Systematic deviation of expected estimate from truth

For unbiased estimators, bias = 0, so:

$$\text{MSE}(\hat{\theta}_{unbiased}) = \text{Var}(\hat{\theta})$$

Thus, among unbiased estimators, minimum variance = minimum MSE. The 'best' unbiased estimator is simply the one with lowest variance.

Alternative Criterion: Mean Absolute Error

One might ask: why not minimize $\mathbb{E}[|\hat{\theta} - \theta|]$ instead? This leads to the median rather than the mean, and corresponding estimators.

MSE (and thus variance for unbiased estimators) is preferred because:

• It has nicer mathematical properties (differentiable, decomposable)
• It naturally connects to the Gaussian distribution (MLE = OLS under Gaussian errors)
• For symmetric distributions, mean and median coincide
• It enables elegant analytical solutions

The Gauss-Markov theorem specifically addresses the MSE criterion for unbiased estimators.

Let's examine exactly what 'minimum variance' means in the matrix sense and prove a key result about linear combinations.

Individual Coefficients:

For each coefficient $\beta_j$, the OLS estimator $\hat{\beta}_j$ has variance:

$$\text{Var}(\hat{\beta}j | \mathbf{X}) = \sigma^2 [(\mathbf{X}^\top\mathbf{X})^{-1}]{jj}$$

For any other linear unbiased estimator $\tilde{\beta}_j$:

$$\text{Var}(\tilde{\beta}_j | \mathbf{X}) \geq \text{Var}(\hat{\beta}_j | \mathbf{X})$$

Linear Combinations:

More generally, for any linear combination $\mathbf{c}^\top\boldsymbol{\beta}$ (e.g., a contrast, a prediction):

$$\text{Var}(\mathbf{c}^\top\hat{\boldsymbol{\beta}}_{OLS} | \mathbf{X}) \leq \text{Var}(\mathbf{c}^\top\tilde{\boldsymbol{\beta}} | \mathbf{X})$$

for any linear unbiased estimator $\tilde{\boldsymbol{\beta}}$.

Matrix Positive Semi-Definiteness:

The statement $\text{Var}(\tilde{\boldsymbol{\beta}}) - \text{Var}(\hat{\boldsymbol{\beta}}_{OLS}) \succeq 0$ is a matrix inequality. It means:

1. All eigenvalues of the difference matrix are non-negative
2. For every vector $\mathbf{c}$: $\mathbf{c}^\top[\text{Var}(\tilde{\boldsymbol{\beta}}) - \text{Var}(\hat{\boldsymbol{\beta}}_{OLS})]\mathbf{c} \geq 0$
3. Equivalently: $\text{Var}(\mathbf{c}^\top\tilde{\boldsymbol{\beta}}) \geq \text{Var}(\mathbf{c}^\top\hat{\boldsymbol{\beta}}_{OLS})$ for all $\mathbf{c}$

This is the strongest possible optimality statement—OLS is best not just for each coefficient, but for every possible linear function of coefficients.

The minimum variance property has a beautiful geometric interpretation that illuminates why OLS is optimal.

The Space of Linear Unbiased Estimators:

Recall that any linear unbiased estimator has the form:

$$\tilde{\boldsymbol{\beta}} = \hat{\boldsymbol{\beta}}_{OLS} + \mathbf{D}\boldsymbol{\varepsilon}$$

Where $\mathbf{D}\mathbf{X} = \mathbf{0}$ (the unbiasedness constraint).

The rows of $\mathbf{D}$ must lie in the null space of $\mathbf{X}^\top$. This is the orthogonal complement of the column space of $\mathbf{X}$:

$$\text{col}(\mathbf{X})^\perp = {\mathbf{v} \in \mathbb{R}^n : \mathbf{X}^\top\mathbf{v} = \mathbf{0}}$$

Why Adding D Increases Variance:

The variance of the alternative estimator is:

$$\text{Var}(\tilde{\boldsymbol{\beta}}) = \text{Var}(\hat{\boldsymbol{\beta}}_{OLS}) + \sigma^2\mathbf{D}\mathbf{D}^\top$$

Geometrically, $\mathbf{D}\mathbf{D}^\top$ is a positive semi-definite matrix representing the 'extra' variance from the perturbation $\mathbf{D}\boldsymbol{\varepsilon}$.

Since $\mathbf{D}$ has rows in the null space of $\mathbf{X}^\top$ (orthogonal to the signal), $\mathbf{D}\boldsymbol{\varepsilon}$ captures only noise—it cannot contain information about $\boldsymbol{\beta}$!

Visualization in Simple Regression:

Consider simple linear regression: $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$.

The column space of $\mathbf{X} = [\mathbf{1}, \mathbf{x}]$ is a 2-dimensional plane in $\mathbb{R}^n$. The OLS fitted values $\hat{\mathbf{y}} = \mathbf{X}\hat{\boldsymbol{\beta}}$ are the orthogonal projection of $\mathbf{y}$ onto this plane.

The residual vector $\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}}$ is perpendicular to the plane—it contains no information about $\boldsymbol{\beta}$.

Alternative estimators that use components from the residual space are essentially using random noise to estimate $\boldsymbol{\beta}$—hence their increased variance.

The Fundamental Insight:

OLS is optimal because orthogonal projection extracts all signal and discards all noise. Any deviation from orthogonal projection necessarily introduces noise into the estimate.

The concept of efficiency formalizes how estimators compare in terms of variance.

Definition: Relative Efficiency

For two unbiased estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ of the same parameter:

$$\text{RE}(\hat{\theta}_1, \hat{\theta}_2) = \frac{\text{Var}(\hat{\theta}_2)}{\text{Var}(\hat{\theta}_1)}$$

If $\text{RE} > 1$, then $\hat{\theta}_1$ is more efficient (lower variance) than $\hat{\theta}_2$.

OLS Efficiency:

By Gauss-Markov, for any linear unbiased estimator $\tilde{\boldsymbol{\beta}}$:

$$\text{RE}(\hat{\boldsymbol{\beta}}_{OLS}, \tilde{\boldsymbol{\beta}}) \geq 1$$

with equality only when $\tilde{\boldsymbol{\beta}} = \hat{\boldsymbol{\beta}}_{OLS}$.

Efficiency Bound (Scalar Case):

For estimating a single coefficient $\beta_j$:

$$\text{Var}(\tilde{\beta}j) \geq \sigma^2[(\mathbf{X}^\top\mathbf{X})^{-1}]{jj}$$

This is the efficiency bound for linear unbiased estimation. OLS achieves this bound exactly.

The Gauss-Markov theorem provides a variance lower bound within the class of linear unbiased estimators. A related but more general result is the Cramér-Rao Lower Bound (CRLB), which provides a variance lower bound for all unbiased estimators (not just linear ones).

The Cramér-Rao Lower Bound:

Under regularity conditions, for any unbiased estimator $\hat{\theta}$ of $\theta$:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$$

Where $I(\theta)$ is the Fisher information:

$$I(\theta) = \mathbb{E}\left[\left(\frac{\partial \log f(\mathbf{y}; \theta)}{\partial \theta}\right)^2\right] = -\mathbb{E}\left[\frac{\partial^2 \log f(\mathbf{y}; \theta)}{\partial \theta^2}\right]$$

OLS and CRLB Under Gaussian Errors:

When $\boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$, the Fisher information for $\boldsymbol{\beta}$ is:

$$I(\boldsymbol{\beta}) = \frac{1}{\sigma^2}\mathbf{X}^\top\mathbf{X}$$

The CRLB gives:

$$\text{Var}(\hat{\boldsymbol{\beta}}) \geq [I(\boldsymbol{\beta})]^{-1} = \sigma^2(\mathbf{X}^\top\mathbf{X})^{-1}$$

This is exactly the variance of OLS! Thus, under Gaussian errors:

OLS achieves the Cramér-Rao bound—it is the minimum variance unbiased estimator among ALL unbiased estimators, not just linear ones.

The Relationship Between Gauss-Markov and Cramér-Rao:

The Gauss-Markov theorem is more general (no distributional assumption) but more limited (only linear estimators). The Cramér-Rao bound is less general (needs smooth density) but more comprehensive (all estimators).

The Gauss-Markov theorem proves OLS is optimal among unbiased estimators. But should we always demand unbiasedness?

The Bias-Variance Tradeoff:

Recall the MSE decomposition:

$$\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + [\text{Bias}(\hat{\theta})]^2$$

An estimator with small bias but much smaller variance can have lower MSE than an unbiased estimator with high variance.

Example: Ridge Regression

The Ridge estimator:

$$\hat{\boldsymbol{\beta}}_{Ridge} = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}$$

is biased:

$$\mathbb{E}[\hat{\boldsymbol{\beta}}_{Ridge}] = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta} \neq \boldsymbol{\beta}$$

But its variance is lower:

$$\text{Var}(\hat{\boldsymbol{\beta}}_{Ridge}) = \sigma^2(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{X}(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}$$

For appropriate $\lambda$, the MSE of Ridge can be lower than OLS, especially when:

• $p$ is large relative to $n$
• True coefficients are small
• Multicollinearity is present

The BLUE concept extends to more general settings beyond standard regression.

Generalized Least Squares (GLS):

When errors have non-spherical covariance $\text{Var}(\boldsymbol{\varepsilon}) = \sigma^2\boldsymbol{\Omega}$ (where $\boldsymbol{\Omega} \neq \mathbf{I}$), the Gauss-Markov theorem no longer applies to OLS.

The GLS estimator:

$$\hat{\boldsymbol{\beta}}_{GLS} = (\mathbf{X}^\top\boldsymbol{\Omega}^{-1}\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\Omega}^{-1}\mathbf{y}$$

is BLUE when $\text{Var}(\boldsymbol{\varepsilon}) = \sigma^2\boldsymbol{\Omega}$. GLS 'whitens' the error structure before applying least squares.

Special Cases of GLS:

Best Linear Unbiased Prediction (BLUP):

In mixed effects models with random effects $\mathbf{u}$:

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \mathbf{Z}\mathbf{u} + \boldsymbol{\varepsilon}$$

We want to estimate the fixed effects $\boldsymbol{\beta}$ and predict the random effects $\mathbf{u}$.

The BLUP of $\mathbf{u}$ minimizes prediction error variance among linear unbiased predictors—an extension of BLUE principles to prediction.

Aitken Theorem:

The result that GLS is BLUE under correlated errors is called the Aitken theorem (or generalized Gauss-Markov theorem):

Under the model $\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ with $\mathbb{E}[\boldsymbol{\varepsilon}] = \mathbf{0}$ and $\text{Var}(\boldsymbol{\varepsilon}) = \sigma^2\boldsymbol{\Omega}$, the GLS estimator is BLUE.

Understanding the BLUE property has direct practical consequences for regression analysis.

When to Rely on OLS Optimality:

Common Mistakes Related to BLUE:

❌ Assuming OLS is always best — It's only BLUE under specific conditions

❌ Ignoring the 'linear' restriction — Nonlinear estimators can beat OLS in MSE

❌ Confusing unbiasedness with accuracy — An unbiased estimator with huge variance can be practically useless

❌ Over-relying on OLS in high dimensions — When p ≈ n, regularization typically dominates

❌ Forgetting that BLUE is conditional on X — The comparison is among estimators for the same design matrix

We've thoroughly explored what it means for OLS to be the Best Linear Unbiased Estimator. Here are the key insights:

What's Next:

With the theoretical foundation of OLS optimality established, we now turn to practical inference. The next page examines the variance of OLS estimates in detail—how to estimate σ², construct standard errors, and understand what affects estimation precision.