Multiple Linear Regression
OLS Estimator Properties
Read aloud is unavailable in this browser.
Beyond the Formula
Having derived the OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$, we now face deeper questions: How good is this estimator?
An estimator isn't just a formula—it's a random variable because it depends on random data. Different samples yield different estimates. To evaluate an estimator, we must understand its statistical properties:
- Is it centered on the true value (unbiased)?
- How much does it vary across samples (variance)?
- Is it the best we can do under certain constraints (optimality)?
These questions lead to the Gauss-Markov theorem, one of the most celebrated results in statistical theory, which establishes OLS as the Best Linear Unbiased Estimator (BLUE).
By the end of this page, you will prove that OLS is unbiased under standard assumptions, derive its variance-covariance matrix, understand the Gauss-Markov theorem, and recognize when OLS achieves optimality.
The Standard Assumptions
All statistical properties of OLS depend on specific assumptions about the data-generating process. Let's state them precisely.
The Linear Model Assumptions:
$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$
where:
- •Linearity in parameters: The model is $\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$ (can be nonlinear in features, but linear in β).
- •Full rank: $\mathbf{X}$ has full column rank (columns are linearly independent), so $\mathbf{X}^\top\mathbf{X}$ is invertible.
- •Exogeneity: $\mathbb{E}[\boldsymbol{\epsilon} | \mathbf{X}] = \mathbf{0}$ — errors have zero mean conditional on X.
- •Spherical errors: $\text{Var}(\boldsymbol{\epsilon} | \mathbf{X}) = \sigma^2 \mathbf{I}_n$ — errors have constant variance (homoscedasticity) and are uncorrelated.
The Gauss-Markov theorem does NOT require normally distributed errors. It only requires zero mean and constant variance. Normality is needed for exact inference (t-tests, F-tests) but not for BLUE properties. We'll add normality later for hypothesis testing.
Understanding spherical errors:
The assumption $\text{Var}(\boldsymbol{\epsilon} | \mathbf{X}) = \sigma^2 \mathbf{I}_n$ packs two conditions:
- Homoscedasticity: $\text{Var}(\epsilon_i) = \sigma^2$ for all $i$ — constant variance
- No autocorrelation: $\text{Cov}(\epsilon_i, \epsilon_j) = 0$ for $i eq j$ — uncorrelated errors
The covariance matrix of $\boldsymbol{\epsilon}$ is: $$\text{Var}(\boldsymbol{\epsilon}) = \sigma^2 \begin{bmatrix} 1 & 0 & \cdots & 0 \ 0 & 1 & \cdots & 0 \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \cdots & 1 \end{bmatrix} = \sigma^2 \mathbf{I}_n$$
Unbiasedness of the OLS Estimator
Definition: An estimator $\hat{\boldsymbol{\beta}}$ is unbiased if its expected value equals the true parameter: $$\mathbb{E}[\hat{\boldsymbol{\beta}} | \mathbf{X}] = \boldsymbol{\beta}$$
Theorem: Under assumptions 1-3 (linearity, full rank, exogeneity), the OLS estimator is unbiased.
Proof:
Starting with the OLS formula: $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$$
Substitute $\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$:
$$\begin{aligned} \hat{\boldsymbol{\beta}} &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top(\mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}) &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\epsilon} &= \mathbf{I}\boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\epsilon} &= \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\epsilon} \end{aligned}$$
Taking expectations conditional on $\mathbf{X}$: $$\mathbb{E}[\hat{\boldsymbol{\beta}} | \mathbf{X}] = \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbb{E}[\boldsymbol{\epsilon} | \mathbf{X}]$$
By assumption 3, $\mathbb{E}[\boldsymbol{\epsilon} | \mathbf{X}] = \mathbf{0}$, so: $$\boxed{\mathbb{E}[\hat{\boldsymbol{\beta}} | \mathbf{X}] = \boldsymbol{\beta}}$$
On average, OLS gets the right answer. If we could repeat our experiment infinitely many times with different noise realizations, the average of our estimates would converge to the true β. This is a finite-sample property—it holds for any sample size n.
Variance-Covariance of the OLS Estimator
Knowing that OLS is unbiased tells us where it's centered, but not how much it fluctuates. The variance-covariance matrix captures this uncertainty.
Theorem: Under assumptions 1-4, the variance-covariance matrix of $\hat{\boldsymbol{\beta}}$ is: $$\text{Var}(\hat{\boldsymbol{\beta}} | \mathbf{X}) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}$$
Proof:
From the unbiasedness proof, we established: $$\hat{\boldsymbol{\beta}} - \boldsymbol{\beta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\epsilon}$$
The variance-covariance matrix is: $$\begin{aligned} \text{Var}(\hat{\boldsymbol{\beta}} | \mathbf{X}) &= \mathbb{E}[(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta})(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta})^\top | \mathbf{X}] &= \mathbb{E}[(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\boldsymbol{\epsilon}\boldsymbol{\epsilon}^\top\mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1} | \mathbf{X}] &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top \mathbb{E}[\boldsymbol{\epsilon}\boldsymbol{\epsilon}^\top | \mathbf{X}] \mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1} \end{aligned}$$
By assumption 4, $\mathbb{E}[\boldsymbol{\epsilon}\boldsymbol{\epsilon}^\top | \mathbf{X}] = \sigma^2 \mathbf{I}_n$: $$\begin{aligned} \text{Var}(\hat{\boldsymbol{\beta}} | \mathbf{X}) &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top (\sigma^2 \mathbf{I}_n) \mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1} &= \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1} &= \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \cdot \mathbf{I} &= \boxed{\sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}} \end{aligned}$$
The diagonal elements σ²[(X⊤X)⁻¹]ⱼⱼ give Var(β̂ⱼ)—the variance of each coefficient estimate. Off-diagonal elements give covariances between coefficient estimates. Standard errors are √(diagonal elements).
Factors Affecting Estimator Variance
The formula $\text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}$ reveals several important insights about what controls estimation precision:
| Factor | Effect on Variance | Intuition |
|---|---|---|
| Error variance σ² | Direct proportionality | More noise → less precise estimates |
| Sample size n | Inverse relationship (via X⊤X) | More data → more precise estimates |
| Feature variance | Inverse relationship | More spread in X → more precise estimates |
| Multicollinearity | Increases variance | Correlated features → unstable estimates |
The multicollinearity effect:
When features are highly correlated, $(\mathbf{X}^\top\mathbf{X})^{-1}$ has large elements. To see why, consider the simple case of two perfectly correlated features: then $\mathbf{X}^\top\mathbf{X}$ is singular and cannot be inverted.
Near-perfect correlation (multicollinearity) makes $(\mathbf{X}^\top\mathbf{X})^{-1}$ nearly singular with very large entries, inflating coefficient variances. Individual coefficients become unreliable, even though predictions may still be accurate.
Variance Inflation Factor (VIF): $$\text{VIF}_j = \frac{1}{1 - R_j^2}$$
where $R_j^2$ is the R-squared from regressing feature $j$ on all other features. VIF > 10 indicates problematic multicollinearity.
With multicollinearity, you can have excellent prediction accuracy (low SSE) but terrible coefficient estimates (high variance). The model 'works' but you can't interpret individual coefficients or trust extrapolations.
The Gauss-Markov Theorem
We've established that OLS is unbiased with variance $\sigma^2(\mathbf{X}^\top\mathbf{X})^{-1}$. But is there a better unbiased estimator with lower variance? The Gauss-Markov theorem answers definitively: No.
Theorem (Gauss-Markov): Under assumptions 1-4, the OLS estimator is the Best Linear Unbiased Estimator (BLUE) of $\boldsymbol{\beta}$.
- Best: Minimum variance among the class
- Linear: Estimator is a linear function of $\mathbf{y}$
- Unbiased: $\mathbb{E}[\hat{\boldsymbol{\beta}}] = \boldsymbol{\beta}$
- Estimator: It estimates the unknown parameter $\boldsymbol{\beta}$
Proof (sketch):
Consider any linear unbiased estimator $\tilde{\boldsymbol{\beta}} = \mathbf{C}\mathbf{y}$ for some matrix $\mathbf{C}$.
Unbiasedness requires: $$\mathbb{E}[\tilde{\boldsymbol{\beta}}] = \mathbb{E}[\mathbf{C}(\mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon})] = \mathbf{C}\mathbf{X}\boldsymbol{\beta} = \boldsymbol{\beta}$$
This must hold for all $\boldsymbol{\beta}$, so $\mathbf{C}\mathbf{X} = \mathbf{I}$.
Write $\mathbf{C} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D}$ where $\mathbf{D}\mathbf{X} = \mathbf{0}$.
Then: $$\text{Var}(\tilde{\boldsymbol{\beta}}) = \sigma^2\mathbf{C}\mathbf{C}^\top = \sigma^2(\mathbf{X}^\top\mathbf{X})^{-1} + \sigma^2\mathbf{D}\mathbf{D}^\top$$
Since $\mathbf{D}\mathbf{D}^\top$ is positive semi-definite: $$\text{Var}(\tilde{\boldsymbol{\beta}}) \geq \text{Var}(\hat{\boldsymbol{\beta}}_{\text{OLS}})$$
Equality holds only when $\mathbf{D} = \mathbf{0}$, i.e., when $\tilde{\boldsymbol{\beta}} = \hat{\boldsymbol{\beta}}_{\text{OLS}}$.
Among all linear unbiased estimators, OLS has the smallest variance. You cannot do better while remaining linear and unbiased. This is why OLS is the default choice for linear regression.
What Gauss-Markov Doesn't Say
The Gauss-Markov theorem is powerful, but its scope is limited. Understanding these boundaries is crucial for modern machine learning.
- •Only linear estimators: Nonlinear estimators (neural networks, trees) can outperform OLS, especially for nonlinear relationships.
- •Requires unbiasedness: Biased estimators (Ridge, Lasso) can have lower mean squared error by trading bias for reduced variance. This is the bias-variance tradeoff.
- •Assumes correct specification: If the true model is nonlinear or has omitted variables, all bets are off.
- •Spherical errors required: Heteroscedasticity or autocorrelation violates assumption 4, making OLS no longer BLUE.
- •Optimality in variance only: OLS minimizes variance, not prediction error on new data. Regularized estimators often predict better.
For linear, correctly-specified models with homoscedastic uncorrelated errors where you want unbiased estimates, OLS is provably optimal.
For high-dimensional settings (p > n), prediction tasks, or when assumptions are violated, regularization and other methods often outperform OLS.
Estimating the Error Variance
The variance formula $\text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2(\mathbf{X}^\top\mathbf{X})^{-1}$ involves the unknown $\sigma^2$. We need an estimator for the error variance.
Natural idea: Use the mean squared residual: $$\hat{\sigma}^2_{\text{naive}} = \frac{1}{n}\sum_{i=1}^n e_i^2 = \frac{1}{n}\mathbf{e}^\top\mathbf{e}$$
Unfortunately, this estimator is biased because residuals systematically underestimate true errors (the fitted line goes through the cloud of points).
Unbiased estimator: $$\hat{\sigma}^2 = \frac{\mathbf{e}^\top\mathbf{e}}{n - p - 1} = \frac{\text{SSE}}{n - p - 1}$$
where $p+1$ is the number of parameters (including intercept).
Why the denominator is n - p - 1:
The residuals $\mathbf{e} = \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}}$ satisfy $p+1$ linear constraints (the normal equations). This means the residual vector has only $n - p - 1$ degrees of freedom.
Intuitively: fitting $p+1$ parameters 'uses up' $p+1$ degrees of freedom from the data, leaving $n - p - 1$ independent pieces of information to estimate $\sigma^2$.
The estimated variance-covariance matrix: $$\widehat{\text{Var}}(\hat{\boldsymbol{\beta}}) = \hat{\sigma}^2 (\mathbf{X}^\top\mathbf{X})^{-1}$$
Standard errors are the square roots of the diagonal elements.
When n is large relative to p, the difference between n and n-p-1 is negligible. But for small samples or high-dimensional data, using the correct denominator is essential for unbiased variance estimation.
KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.