Least Squares Derivation

Given our model $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ and observed data ${(x_i, y_i)}_{i=1}^n$, we face a fundamental question: How do we find the "best" values for $\beta_0$ and $\beta_1$?

The answer depends on defining what "best" means. The method of least squares—first published by Legendre in 1805 and independently developed by Gauss around the same time—provides an elegant and principled answer that has stood the test of two centuries.

This page derives the least squares solution from scratch, revealing not just the formulas but the reasoning behind them.

Our goal is to find values $\hat{\beta}_0$ and $\hat{\beta}_1$ that make our predicted values $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ as close as possible to the observed values $y_i$.

The residual for observation $i$ is the prediction error:

$$e_i = y_i - \hat{y}_i = y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i)$$

We want all residuals to be small. But how do we aggregate $n$ individual residuals into a single measure of overall fit?

The least squares method minimizes the residual sum of squares (RSS), also called the sum of squared errors (SSE):

$$\text{RSS}(\beta_0, \beta_1) = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n [y_i - (\beta_0 + \beta_1 x_i)]^2$$

This is a function of two unknowns ($\beta_0$ and $\beta_1$). Our task is to find the values that minimize it:

$$\hat{\beta}0, \hat{\beta}1 = \arg\min{\beta_0, \beta_1} \sum{i=1}^n [y_i - \beta_0 - \beta_1 x_i]^2$$

We have an unconstrained optimization problem in two variables. Define the objective function:

$$S(\beta_0, \beta_1) = \sum_{i=1}^n [y_i - \beta_0 - \beta_1 x_i]^2$$

This is a convex quadratic function of $(\beta_0, \beta_1)$. Because it's quadratic with positive leading coefficients (think: bowl-shaped surface opening upward), it has a unique global minimum that we can find by setting the first-order derivatives to zero.

Visualizing the Objective

Imagine a 3D surface where the horizontal plane has coordinates $(\beta_0, \beta_1)$ and the vertical axis represents $S(\beta_0, \beta_1)$. The surface is a paraboloid—a bowl shape. The minimum is at the bottom of the bowl.

First-Order Necessary Conditions

At the minimum, the gradient must be zero:

$$ abla S = \begin{pmatrix} \frac{\partial S}{\partial \beta_0} \ \frac{\partial S}{\partial \beta_1} \end{pmatrix} = \begin{pmatrix} 0 \ 0 \end{pmatrix}$$

Let's compute each partial derivative.

Partial Derivative with Respect to $\beta_0$

Using the chain rule:

$$\frac{\partial S}{\partial \beta_0} = \sum_{i=1}^n 2[y_i - \beta_0 - \beta_1 x_i] \cdot (-1) = -2\sum_{i=1}^n [y_i - \beta_0 - \beta_1 x_i]$$

Setting equal to zero and dividing by $-2$:

$$\sum_{i=1}^n [y_i - \beta_0 - \beta_1 x_i] = 0$$

Expanding the sum:

$$\sum_{i=1}^n y_i - n\beta_0 - \beta_1 \sum_{i=1}^n x_i = 0$$

Dividing by $n$ and using the notation $\bar{y} = \frac{1}{n}\sum_{i=1}^n y_i$ (sample mean of $y$) and $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$ (sample mean of $x$):

$$\bar{y} - \beta_0 - \beta_1 \bar{x} = 0$$

This gives us the first normal equation:

$$\boxed{\beta_0 = \bar{y} - \beta_1 \bar{x}}$$

Partial Derivative with Respect to $\beta_1$

$$\frac{\partial S}{\partial \beta_1} = \sum_{i=1}^n 2[y_i - \beta_0 - \beta_1 x_i] \cdot (-x_i) = -2\sum_{i=1}^n x_i[y_i - \beta_0 - \beta_1 x_i]$$

Setting equal to zero and dividing by $-2$:

$$\sum_{i=1}^n x_i[y_i - \beta_0 - \beta_1 x_i] = 0$$

Expanding:

$$\sum_{i=1}^n x_i y_i - \beta_0 \sum_{i=1}^n x_i - \beta_1 \sum_{i=1}^n x_i^2 = 0$$

This is the second normal equation.

Solving for $\beta_1$

Substitute $\beta_0 = \bar{y} - \beta_1 \bar{x}$ into the second normal equation:

$$\sum_{i=1}^n x_i y_i - (\bar{y} - \beta_1 \bar{x}) \sum_{i=1}^n x_i - \beta_1 \sum_{i=1}^n x_i^2 = 0$$

Note that $\sum_{i=1}^n x_i = n\bar{x}$, so:

$$\sum_{i=1}^n x_i y_i - \bar{y}(n\bar{x}) + \beta_1 \bar{x}(n\bar{x}) - \beta_1 \sum_{i=1}^n x_i^2 = 0$$

Rearranging:

$$\sum_{i=1}^n x_i y_i - n\bar{x}\bar{y} = \beta_1 \left( \sum_{i=1}^n x_i^2 - n\bar{x}^2 \right)$$

Solving for $\beta_1$:

$$\boxed{\beta_1 = \frac{\sum_{i=1}^n x_i y_i - n\bar{x}\bar{y}}{\sum_{i=1}^n x_i^2 - n\bar{x}^2}}$$

Let's express the OLS estimators in their most common forms. Using centered notation, define:

$$S_{xy} = \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})$$ $$S_{xx} = \sum_{i=1}^n (x_i - \bar{x})^2$$ $$S_{yy} = \sum_{i=1}^n (y_i - \bar{y})^2$$

It can be shown (by expanding) that:

$$\sum_{i=1}^n x_i y_i - n\bar{x}\bar{y} = S_{xy}$$ $$\sum_{i=1}^n x_i^2 - n\bar{x}^2 = S_{xx}$$

The OLS Estimators

The slope estimator:

$$\boxed{\hat{\beta}1 = \frac{S{xy}}{S_{xx}} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}}$$

The intercept estimator:

$$\boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}$$

These are the ordinary least squares (OLS) estimators for simple linear regression.

The first-order conditions we derived are called the normal equations. Written together:

$$\sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}1 x_i) = 0$$ $$\sum{i=1}^n x_i(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0$$

These can be rewritten using residuals $e_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i$:

$$\sum_{i=1}^n e_i = 0$$ $$\sum_{i=1}^n x_i e_i = 0$$

These equations express fundamental properties of OLS residuals.

Matrix Form of Normal Equations

In matrix notation with $\mathbf{X} = [\mathbf{1} | \mathbf{x}]$ (design matrix with intercept column), the normal equations become:

$$\mathbf{X}^T (\mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}}) = \mathbf{0}$$

This says the residual vector $\mathbf{e} = \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}}$ is orthogonal to every column of $\mathbf{X}$. Specifically:

• Orthogonal to $\mathbf{1}$ (column of ones): $\sum_i e_i = 0$
• Orthogonal to $\mathbf{x}$: $\sum_i x_i e_i = 0$

Rearranging the normal equations:

$$\mathbf{X}^T \mathbf{y} = \mathbf{X}^T \mathbf{X} \hat{\boldsymbol{\beta}}$$

If $\mathbf{X}^T\mathbf{X}$ is invertible:

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

This is the famous closed-form solution for OLS in matrix form.

We found a critical point by setting the gradient to zero. How do we know it's a minimum and not a maximum or saddle point?

The Hessian Matrix

We need to check the second-order conditions using the Hessian matrix of second partial derivatives:

$$H = \begin{pmatrix} \frac{\partial^2 S}{\partial \beta_0^2} & \frac{\partial^2 S}{\partial \beta_0 \partial \beta_1} \ \frac{\partial^2 S}{\partial \beta_1 \partial \beta_0} & \frac{\partial^2 S}{\partial \beta_1^2} \end{pmatrix}$$

Computing each term from $S = \sum_i [y_i - \beta_0 - \beta_1 x_i]^2$:

$$\frac{\partial^2 S}{\partial \beta_0^2} = 2n$$

$$\frac{\partial^2 S}{\partial \beta_1^2} = 2\sum_{i=1}^n x_i^2$$

$$\frac{\partial^2 S}{\partial \beta_0 \partial \beta_1} = 2\sum_{i=1}^n x_i = 2n\bar{x}$$

So the Hessian is:

$$H = 2\begin{pmatrix} n & n\bar{x} \ n\bar{x} & \sum_i x_i^2 \end{pmatrix}$$

Positive Definiteness Check

For a minimum, $H$ must be positive definite. We check:

1. First principal minor: $2n > 0$ ✓ (assuming $n > 0$)

2. Second principal minor (determinant):

$$\det(H/2) = n \sum_i x_i^2 - n^2 \bar{x}^2 = n \left( \sum_i x_i^2 - n\bar{x}^2 \right) = n \cdot S_{xx}$$

This is positive as long as $S_{xx} > 0$, which holds whenever not all $x_i$ are identical.

Conclusion: The critical point is a global minimum (since the quadratic is convex). OLS gives the unique solution that minimizes RSS.

Let's implement the OLS formulas and verify our understanding. The implementation illustrates both the conceptual formula and practical considerations.