Simple Linear Regression

Linear regression stands as one of the most fundamental and enduring techniques in all of machine learning and statistics. Despite its apparent simplicity, it forms the conceptual bedrock upon which much of modern predictive modeling is built. Understanding linear regression thoroughly—not just how to use it, but why it works and when it fails—is essential for any serious practitioner of machine learning.

This page introduces the simple linear regression model, the case where we have a single input variable (feature) and seek to predict a continuous output variable (target). While "simple" in name, this model encapsulates the core principles that extend to multiple regression, regularized methods, and even neural networks.

At its heart, regression addresses a fundamental question in data analysis: Given observations of two related variables, how can we predict one from the other?

Consider concrete examples:

• Given a house's square footage, predict its sale price
• Given hours studied, predict exam score
• Given advertising spend, predict sales revenue
• Given temperature, predict energy consumption

In each case, we observe pairs of values $(x_i, y_i)$ where $x$ is the independent variable (predictor, feature, explanatory variable) and $y$ is the dependent variable (response, target, outcome). Our goal is to discover a relationship that allows prediction of $y$ for new, unseen values of $x$.

Why Linear?

The simplest possible assumption about the relationship between $x$ and $y$ is that it's linear—that $y$ changes proportionally with $x$. This means we're looking for a straight line that best describes the relationship.

This assumption might seem restrictive, but it turns out to be remarkably powerful for several reasons:

1. Interpretability: Linear relationships are easy to understand and explain
2. Mathematical Tractability: Linear models have closed-form solutions and well-understood statistical properties
3. Flexibility via Transformation: Many nonlinear relationships become linear after appropriate transformations
4. Local Approximation: Any smooth function can be approximated linearly in a small neighborhood (Taylor expansion)
5. Foundation for Extension: Linear methods generalize naturally to more complex models

The simple linear regression model posits that the true relationship between $x$ and $y$ takes the form:

$$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$$

This equation is deceptively simple, but each component carries deep meaning. Let's dissect it carefully.

Deterministic vs. Stochastic Components

The model equation can be rewritten to highlight a crucial conceptual distinction:

$$y_i = \underbrace{\beta_0 + \beta_1 x_i}{\text{systematic component}} + \underbrace{\varepsilon_i}{\text{random component}}$$

The systematic component $\beta_0 + \beta_1 x_i$ represents the deterministic part—the expected or average value of $y$ given $x$. If we knew $\beta_0$ and $\beta_1$ exactly, this would be our best prediction.

The random component $\varepsilon_i$ represents the stochastic part—the unpredictable deviation of any individual observation from the expected value. This is inherently uncertain and differs for each data point.

This decomposition is fundamental: we can model and estimate the systematic component, but we can never eliminate the random component—we can only characterize its statistical properties.

A critical distinction in statistical modeling—one that sometimes confuses newcomers—is between population parameters and sample estimates. Understanding this distinction is essential for interpreting regression results correctly.

Population Parameters (True but Unknown)

The quantities $\beta_0$, $\beta_1$, and the distribution of $\varepsilon$ describe the true underlying relationship in the entire population. If we could somehow observe every possible $(x, y)$ pair in the universe (every house ever sold, every student ever tested), we would know these parameters exactly.

But we can't. We have only a finite sample of $n$ observations: ${(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)}$.

Sample Estimates (Computed from Data)

From our sample, we compute estimates of the population parameters, denoted with hats:

$$\hat{\beta}_0, \quad \hat{\beta}_1$$

These estimates are our best guesses based on available data. They will differ from the true parameters, and they would differ if we collected a different sample. This variability is the subject of statistical inference.

The Fitted Model

Once we estimate $\hat{\beta}_0$ and $\hat{\beta}_1$ from our sample, we obtain the fitted regression line:

$$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$$

This is our predictive model. For any value $x$, we can compute a predicted value $\hat{y}$. The residuals are the differences between observed and predicted values:

$$e_i = y_i - \hat{y}_i = y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i)$$

Residuals are observable (we can compute them), while errors are not (they involve unknown true parameters). Much of regression diagnostics involves analyzing residuals as proxies for the unobservable errors.

A powerful way to understand the regression model is through the lens of conditional expectation. This perspective connects regression to fundamental concepts in probability theory and provides deeper insight into what the model actually represents.

What Does the Regression Function Represent?

The systematic component $\beta_0 + \beta_1 x$ equals the conditional expectation of $y$ given $x$:

$$E[y \mid x] = \beta_0 + \beta_1 x$$

This tells us: for any fixed value of $x$, the expected (average) value of $y$ is $\beta_0 + \beta_1 x$. The regression line traces out these conditional means as $x$ varies.

Deriving the Error Properties

The conditional expectation view implies key properties of the error term. Starting from:

$$y = \beta_0 + \beta_1 x + \varepsilon$$

Take the conditional expectation given $x$:

$$E[y \mid x] = E[\beta_0 + \beta_1 x + \varepsilon \mid x]$$ $$\beta_0 + \beta_1 x = \beta_0 + \beta_1 x + E[\varepsilon \mid x]$$

This requires:

$$E[\varepsilon \mid x] = 0$$

This is the zero conditional mean assumption: the expected value of the error, given $x$, is zero. This is one of the most important assumptions in regression analysis.

What Does Zero Conditional Mean Imply?

The condition $E[\varepsilon \mid x] = 0$ means:

1. On average, predictions are neither systematically too high nor too low for any value of $x$
2. The error is uncorrelated with the predictor: $\text{Cov}(x, \varepsilon) = 0$
3. No information about $\varepsilon$ can be extracted from knowing $x$
4. The relationship between mean $y$ and $x$ is correctly specified as linear

While simple linear regression involves only one predictor, it's instructive to see how the model can be written in matrix form—the notation that scales naturally to multiple regression.

Scalar Form (What We've Seen)

For each observation $i = 1, \ldots, n$: $$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$$

System of Equations

Writing out all $n$ observations:

$$\begin{aligned} y_1 &= \beta_0 + \beta_1 x_1 + \varepsilon_1 y_2 &= \beta_0 + \beta_1 x_2 + \varepsilon_2 &\vdots y_n &= \beta_0 + \beta_1 x_n + \varepsilon_n \end{aligned}$$

Matrix Form

Define:

$$\mathbf{y} = \begin{pmatrix} y_1 \ y_2 \ \vdots \ y_n \end{pmatrix}, \quad \mathbf{X} = \begin{pmatrix} 1 & x_1 \ 1 & x_2 \ \vdots & \vdots \ 1 & x_n \end{pmatrix}, \quad \boldsymbol{\beta} = \begin{pmatrix} \beta_0 \ \beta_1 \end{pmatrix}, \quad \boldsymbol{\varepsilon} = \begin{pmatrix} \varepsilon_1 \ \varepsilon_2 \ \vdots \ \varepsilon_n \end{pmatrix}$$

The $\mathbf{X}$ matrix is called the design matrix. Notice the column of 1's—this corresponds to the intercept term $\beta_0$.

Now the entire system becomes:

$$\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$$

This compact notation is remarkably powerful. It expresses all $n$ equations in a single matrix equation, and as we'll see, it leads to elegant solutions and insights.

A useful mental model is to think of the regression equation as capturing how data is generated. This data generating process (DGP) perspective helps clarify assumptions and their consequences.

Thought Experiment: Simulating Data

Imagine you're Nature (or a simulator), and you want to generate regression data:

1. Choose true parameters: Fix $\beta_0 = 5$, $\beta_1 = 2$, $\sigma^2 = 1$
2. For each observation $i$:
   • Observe or choose $x_i$ (e.g., $x_i = 3$)
   • Compute deterministic component: $5 + 2(3) = 11$
   • Draw random error: $\varepsilon_i \sim N(0, 1)$ (say $\varepsilon_i = 0.4$)
   • Generate observed response: $y_i = 11 + 0.4 = 11.4$
3. Repeat for $n$ observations

The observer (statistician) sees only $(x_i, y_i)$ pairs. They don't know $\beta_0$, $\beta_1$, $\sigma^2$, or the realized $\varepsilon_i$ values. Their task is to recover these from the observed data.

Linear regression predates modern machine learning by over two centuries—Legendre published the first account of least squares in 1805. Yet it remains central to contemporary ML. Understanding this connection helps contextualize what we're learning.

Linear Regression as Supervised Learning

In the ML framework:

• Hypothesis class $\mathcal{H}$: All functions of the form $h(x) = \beta_0 + \beta_1 x$
• Loss function: Squared error $L(y, h(x)) = (y - h(x))^2$
• Risk: Expected loss $R(h) = E[(y - h(x))^2]$
• Empirical risk: Average loss on training data $\hat{R}(h) = \frac{1}{n}\sum_{i=1}^n (y_i - h(x_i))^2$
• Learning: Find $h \in \mathcal{H}$ that minimizes empirical risk (empirical risk minimization)

Ordinary least squares is precisely ERM with the linear hypothesis class and squared loss.

Why Both Views Matter

The statistical and ML perspectives are complementary, not contradictory:

• Understanding OLS statistical properties (unbiasedness, variance) helps diagnose when predictions will be reliable
• ML's focus on generalization guards against overfitting, even in simple regression
• Statistical hypothesis tests tell us which variables actually matter
• ML validation methods tell us how well we'll predict on new data

A complete understanding of linear regression draws from both traditions. This module emphasizes the statistical foundations; subsequent modules on regularization and model selection emphasize the ML perspective.

We've established the mathematical foundation for simple linear regression. Let's consolidate the key concepts:

What's Next

With the model formulated, the natural question is: how do we estimate $\beta_0$ and $\beta_1$ from data? The next page develops the least squares principle—the most fundamental estimation method in regression. We'll derive the OLS estimators by minimizing the sum of squared residuals and explore why this criterion makes sense both geometrically and statistically.