Assumptions

Everything we've derived so far—the OLS estimators, their statistical properties, confidence intervals, hypothesis tests—relies on certain assumptions being true. These aren't arbitrary mathematical conveniences; they're substantive claims about how the data was generated.

Violating these assumptions doesn't necessarily make regression useless, but it changes what we can conclude. Understanding assumptions is the difference between wielding regression as a powerful inferential tool versus as a curve-fitting exercise of unknown validity.

This page examines each assumption in depth: what it means, why it matters, how to check it, what goes wrong when it fails, and what to do about violations.

The classical linear regression model makes five key assumptions. Together, they enable the strong theoretical guarantees of OLS. We'll list them first, then examine each in depth.

The Classical Linear Model Assumptions (CLRM)

For the model $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$:

1. Linearity in Parameters: The true relationship is $y = \beta_0 + \beta_1 x + \varepsilon$

2. Random Sampling: The observations $(x_i, y_i)$ are a random sample from the population

3. Conditional Mean Zero: $E[\varepsilon | x] = 0$ (no systematic error pattern)

4. Homoscedasticity: $\text{Var}(\varepsilon | x) = \sigma^2$ (constant error variance)

5. Normality (for inference): $\varepsilon | x \sim N(0, \sigma^2)$ (errors are normally distributed)

Plus a technical condition:

6. Non-constant x: $\text{Var}(x) > 0$ (x actually varies in the sample)

Statement: The true conditional expectation function is linear:

$$E[y | x] = \beta_0 + \beta_1 x$$

This means the relationship between the expected value of $y$ and $x$ is a straight line—not a curve, not a step function, not some other shape.

Why It Matters

If the true relationship is nonlinear (e.g., quadratic), a linear model is misspecified. The fitted line will be the "best" straight line through curved data, but:

1. Predictions will be systematically wrong at different x values
2. The slope estimate won't represent the true marginal effect
3. Residuals will show a pattern (not random scatter)

Diagnosing Linearity Violations

1. Scatterplot of y vs. x: Does the data follow a straight line, or curve?

2. Residual plot (residuals vs. x or ŷ): Residuals should scatter randomly around zero. Curved patterns indicate nonlinearity.

3. Residual plot (residuals vs. ŷ): Same as above, often clearer.

4. Added variable plots: In multiple regression (advanced).

Common Patterns Indicating Nonlinearity

• U-shaped or inverted-U residuals → quadratic relationship
• Asymptotic pattern → logarithmic or saturation relationship
• S-shaped pattern → logistic-type relationship

Statement: The observations $(x_1, y_1), \ldots, (x_n, y_n)$ are a random sample from the population of interest.

This means:

• Each observation is drawn independently from the same population
• Selection into the sample doesn't depend on $y$ or $\varepsilon$
• The sample represents the population we want to generalize to

Why It Matters

Random sampling ensures that sample statistics (like $\hat{\beta}_1$) are unbiased estimates of population parameters. It also ensures that our confidence intervals have their stated coverage probability.

Diagnosing Random Sampling Violations

This is often a design issue, not detectable from the data alone:

1. Consider your sampling process: How were observations collected?
2. Compare sample to population: If you know population characteristics
3. Check for time/space patterns: If observations are not independent

Special Case: Time Series

With time series data (observations over time), observations are typically not independent—values depend on past values (autocorrelation). This violates random sampling and requires special methods (time series regression, ARIMA, etc.).

Statement: $E[\varepsilon | x] = 0$ for all values of $x$.

This is arguably the most critical assumption. It says that, conditional on knowing $x$, the expected error is zero. On average, our predictions are neither systematically too high nor too low for any value of $x$.

Equivalent Formulations:

1. $E[\varepsilon | x] = 0$ (conditional expectation)
2. $\text{Cov}(x, \varepsilon) = 0$ (implied, but weaker)
3. The linear model correctly specifies $E[y|x]$

Why It Matters

If $E[\varepsilon | x] e 0$, OLS estimates are biased. The slope $\hat{\beta}_1$ doesn't estimate the true $\beta_1$—even with infinite data. This is a fundamental failure.

Sources of Violation

1. Omitted Variable Bias: A relevant variable $z$ is left out. If $z$ is correlated with both $x$ and $y$:

$$\text{Bias} = \beta_z \cdot \frac{\text{Cov}(x, z)}{\text{Var}(x)}$$

2. Measurement Error in x: If we observe $x^* = x + \eta$ instead of true $x$:

$$\hat{\beta}1 \xrightarrow{p} \beta_1 \cdot \frac{\sigma_x^2}{\sigma_x^2 + \sigma\eta^2}$$

(Attenuation bias—slope shrinks toward zero.)

3. Simultaneity/Reverse Causation: $y$ affects $x$ as well as vice versa.

4. Model Misspecification: The true functional form is nonlinear.

Diagnosing E[ε|x] = 0 Violations

This is challenging because we don't observe $\varepsilon$, only residuals $e$:

1. Residual plots (e vs. x): Patterns suggest misspecification
2. Ramsey RESET test: Tests for omitted nonlinearity
3. Add suspected omitted variables: See if coefficients change
4. Instrumental variables: If available, can test exclusion

Key Point: We can never prove E[ε|x] = 0 from data alone. We can only check for obvious violations and think carefully about the data generating process.

Statement: $\text{Var}(\varepsilon | x) = \sigma^2$ for all $x$.

The error variance is constant across all values of $x$. "Homo" = same; "scedasticity" = spread. When this fails, we have heteroscedasticity (different spread).

Why It Matters

Homoscedasticity affects inference, not unbiasedness:

• OLS estimates are still unbiased under heteroscedasticity
• But standard errors are wrong (usually too small)
• Confidence intervals and p-values are therefore invalid
• OLS is no longer the most efficient estimator

Common Heteroscedasticity Patterns

1. Fan/cone shape: Variance increases with $x$ (e.g., income vs. spending)
2. Inverse cone: Variance decreases with $x$ (less common)
3. Discrete patterns: Different variances for different groups

Diagnosing Heteroscedasticity

1. Residual vs. fitted plot: Look for fanning patterns
2. Breusch-Pagan test: Regress squared residuals on $x$; test if slope ≠ 0
3. White test: More general test for heteroscedasticity
4. Goldfeld-Quandt test: Compare variance in subsamples

Statement: $\varepsilon | x \sim N(0, \sigma^2)$

Conditional on $x$, the errors follow a normal (Gaussian) distribution with mean 0 and variance $\sigma^2$.

Why It Matters

Normality enables exact statistical inference:

• With normality, $\hat{\beta}_1$ follows an exact normal distribution
• The t-statistics follow exact t-distributions
• F-statistics follow exact F-distributions
• Confidence intervals and p-values are exact

Diagnosing Normality Violations

1. Histogram of residuals: Should look roughly bell-shaped

2. Q-Q plot (quantile-quantile): Residuals vs. theoretical normal quantiles. Should be roughly linear.

3. Shapiro-Wilk test: Formal test of normality (H₀: data is normal)

4. Jarque-Bera test: Tests skewness and kurtosis

Common Departures

• Skewness: Residuals are asymmetric (long tail on one side)
• Heavy tails (leptokurtosis): More outliers than normal predicts
• Light tails (platykurtosis): Fewer outliers than normal predicts
• Bimodality: Two peaks (may indicate missing categorical predictor)

Remedies for Non-Normality

1. Large sample: Often no action needed—just rely on CLT
2. Transform y: log(y) often normalizes right-skewed data
3. Robust regression: Less sensitive to outliers (M-estimation, quantile regression)
4. Bootstrap inference: Doesn't assume normality
5. Non-parametric tests: When distribution is very non-normal

Not all assumption violations are equally serious. Here's a practical ranking:

The Gauss-Markov Theorem

Under assumptions 1-4 (but NOT requiring normality):

OLS is BLUE (Best Linear Unbiased Estimator)

• Best: Minimum variance among all linear unbiased estimators
• Linear: A linear function of the response values
• Unbiased: $E[\hat{\beta}] = \beta$
• Estimator: A statistic computed from data

This is a remarkable result: among all estimators that are linear and unbiased, OLS has the smallest variance. Adding normality (Assumption 5) makes OLS the best among all unbiased estimators, not just linear ones.