Quantile Regression

Every regression technique you've encountered thus far—from simple linear regression to sophisticated kernel methods—shares a common objective: estimating the conditional mean. When you predict house prices or stock returns, you're implicitly asking, "What is the average value of y given x?"

But this singular focus on the mean tells only part of the story. In many real-world scenarios, knowing the average is insufficient or even misleading:

• Risk Management: A financial analyst needs to understand not just the expected portfolio return, but the worst-case scenarios—the 5th percentile of potential losses.
• Healthcare: A clinician prescribing medication cares about the typical response, but also about patients who respond poorly (lower quantiles) or exceptionally well (upper quantiles).
• Supply Chain: An operations manager must ensure adequate inventory not just for average demand, but for peak demand scenarios.
• Climate Science: Modeling extreme weather events requires understanding the tails of the distribution, not the center.

Quantile regression provides the mathematical framework to address these questions directly, allowing us to model any quantile of the conditional distribution—not just the mean.

To appreciate quantile regression, we must first understand what we sacrifice when focusing exclusively on conditional means.

The Standard Regression Setup:

Consider the familiar regression model:

$$y_i = f(x_i) + \varepsilon_i$$

where $y_i$ is the response, $x_i$ are features, $f(x_i)$ is the conditional mean function, and $\varepsilon_i$ is zero-mean noise. Ordinary Least Squares (OLS) estimates $f$ by minimizing:

$$\hat{f}{OLS} = \arg\min_f \sum{i=1}^{n} (y_i - f(x_i))^2$$

This estimator has a well-known property: it provides the conditional mean $\mathbb{E}[Y \mid X = x]$.

Why the mean can be inadequate:

A Motivating Example:

Consider predicting medical costs based on patient characteristics. The distribution of healthcare expenditures is notoriously right-skewed: most patients incur modest costs, but a small fraction generates extremely high bills. The conditional mean—influenced heavily by these expensive cases—may not represent what most patients will actually pay.

If an insurer sets premiums based only on the mean, they might:

• Overcharge typical patients (mean > median)
• Underestimate capital reserves needed for high-cost cases
• Miss crucial risk factors that affect the tails differently than the center

Quantile regression provides a richer picture by estimating the entire conditional distribution.

Before constructing the quantile loss function, let's establish rigorous definitions and properties of quantiles themselves.

Definition (Quantile):

For a random variable $Y$ with cumulative distribution function (CDF) $F_Y(y) = P(Y \leq y)$, the $\tau$-th quantile $q_\tau$ is defined as:

$$q_\tau = F_Y^{-1}(\tau) = \inf{y : F_Y(y) \geq \tau}$$

for $\tau \in (0, 1)$.

Key Properties of Quantiles:

Common Quantiles and Their Names:

Conditional Quantiles:

In regression, we're interested in conditional quantiles—the quantiles of $Y$ given $X = x$:

$$Q_\tau(Y \mid X = x) = \inf{y : P(Y \leq y \mid X = x) \geq \tau}$$

For the conditional mean, we have $\mathbb{E}[Y \mid X = x]$. Quantile regression generalizes this to:

$$Q_\tau(Y \mid X = x) = x^\top \beta(\tau)$$

where $\beta(\tau)$ is the coefficient vector for quantile $\tau$. Note that the coefficients themselves depend on $\tau$—different quantiles have different regression coefficients!

The choice of loss function determines what aspect of the conditional distribution we estimate. OLS uses squared loss, which minimizes to the mean. What loss function yields quantiles?

The Key Insight:

To estimate the $\tau$-th quantile, we need a loss function that:

1. Penalizes underestimation and overestimation differently
2. Achieves its minimum at the $\tau$-th population quantile
3. Weights errors asymmetrically based on $\tau$

The Check Function (Quantile Loss):

The quantile loss function (also called the check function or tilted absolute value function) for quantile level $\tau \in (0, 1)$ is defined as:

$$\rho_\tau(u) = u \cdot (\tau - \mathbb{1}{u < 0})$$

where $\mathbb{1}{u < 0}$ is the indicator function that equals 1 if $u < 0$ and 0 otherwise.

Equivalent piecewise formulation:

$$\rho_\tau(u) = \begin{cases} \tau \cdot u & \text{if } u \geq 0 \ (\tau - 1) \cdot u & \text{if } u < 0 \end{cases}$$

Since $\tau - 1$ is negative and $u < 0$ in the second case, this makes $(\tau - 1) \cdot u$ positive, as required for a loss function.

Understanding the Asymmetry:

Let's analyze how the loss penalizes positive errors (underestimation: $y > \hat{y}$, so $u = y - \hat{y} > 0$) versus negative errors (overestimation: $y < \hat{y}$, so $u < 0$):

• Positive residuals $u > 0$: penalty = $\tau \cdot |u|$
• Negative residuals $u < 0$: penalty = $(1 - \tau) \cdot |u|$

For $\tau = 0.9$ (estimating the 90th percentile):

• Underestimation ($u > 0$): penalty = $0.9 |u|$
• Overestimation ($u < 0$): penalty = $0.1 |u|$

The optimizer is pushed to make underestimation errors rare because they are penalized 9× more heavily than overestimation. This drives the estimate toward the 90th percentile—a value exceeded by only 10% of observations.

The quantile loss function possesses several mathematical properties that make it suitable for optimization and statistical inference.

Property 1: Convexity

The quantile loss $\rho_\tau(u)$ is a convex function of $u$. This is easily verified:

• It is piecewise linear with slopes $\tau$ (for $u \geq 0$) and $\tau - 1$ (for $u < 0$)
• Since $\tau - 1 < 0 < \tau$ for $\tau \in (0,1)$, the function has a 'V' shape with the vertex at $u = 0$
• A piecewise linear function with increasing slopes is convex

Implication: The quantile regression objective function is convex in the parameters, guaranteeing a unique global minimum (or a convex set of minimizers).

Property 2: Population Minimizer

The $\tau$-th population quantile minimizes the expected quantile loss. Formally:

$$q_\tau = \arg\min_q \mathbb{E}[\rho_\tau(Y - q)]$$

Proof Sketch:

Take the derivative (in the distributional sense) of $\mathbb{E}[\rho_\tau(Y - q)]$ with respect to $q$:

$$\frac{d}{dq} \mathbb{E}[\rho_\tau(Y - q)] = -\tau P(Y > q) + (1 - \tau) P(Y < q)$$

$$= -\tau(1 - F(q)) + (1 - \tau)F(q) = F(q) - \tau$$

Setting this to zero gives $F(q) = \tau$, hence $q = F^{-1}(\tau) = q_\tau$. □

Property 3: Subgradient Structure

At $u = 0$, the quantile loss is not differentiable. However, it has a well-defined subgradient:

$$\partial \rho_\tau(u) = \begin{cases} {\tau} & \text{if } u > 0 \ [\tau - 1, \tau] & \text{if } u = 0 \ {\tau - 1} & \text{if } u < 0 \end{cases}$$

The subgradient at zero is the interval $[\tau - 1, \tau]$, which always contains zero when $\tau \in (0, 1)$. This structure is exploited by optimization algorithms.

Property 4: Robustness

Unlike squared loss, which grows quadratically with $|u|$, the quantile loss grows only linearly. This bounded influence function provides:

• Natural robustness to outliers
• Bounded impact from any single observation
• Similar behavior to robust M-estimators

Understanding the differences between quantile loss and squared loss illuminates when each is appropriate.

The Two Loss Functions Side by Side:

$$L_{squared}(u) = u^2 \qquad \text{vs.} \qquad \rho_\tau(u) = u(\tau - \mathbb{1}{u < 0})$$

Geometric Interpretation:

The squared loss creates a symmetric bowl that pushes the estimate toward the center of the residual distribution. The quantile loss creates an asymmetric 'V' that tilts toward one side, pushing the estimate toward a specific quantile.

When to Choose Each:

Use Squared Loss (OLS) when:

• Errors are roughly symmetric and Gaussian
• The conditional mean is the quantity of interest
• Computational efficiency is paramount (closed-form solution)
• Homoscedasticity is a reasonable assumption

Use Quantile Loss when:

• Modeling different parts of the distribution
• Data contains outliers or heavy-tailed noise
• Heteroscedasticity is present (variance changes with x)
• Risk or tail behavior is the primary concern
• Distribution-free inference is desired

Visual understanding of the quantile loss function solidifies these concepts.

Key Observations from the Visualization:

1. Asymmetry increases as τ departs from 0.5: For τ = 0.9, the left branch is nearly flat while the right branch is steep.

2. All curves pass through the origin: Zero residual incurs zero loss, regardless of τ.

3. Median loss equals half the absolute error: The τ = 0.5 curve matches |u|/2.

4. Mirror symmetry: ρ_τ(u) and ρ_{1-τ}(-u) are identical—estimating the τ-quantile with one orientation is equivalent to estimating the (1-τ)-quantile with reversed sign.

With the quantile loss function established, we can now formulate the quantile regression optimization problem.

Problem Statement:

Given data ${(x_i, y_i)}_{i=1}^{n}$, where $x_i \in \mathbb{R}^p$ and $y_i \in \mathbb{R}$, the $\tau$-quantile regression estimator is:

$$\hat{\beta}(\tau) = \arg\min_{\beta \in \mathbb{R}^p} \sum_{i=1}^{n} \rho_\tau(y_i - x_i^\top \beta)$$

Expanding the loss function:

$$\hat{\beta}(\tau) = \arg\min_{\beta} \left[ \tau \sum_{i: y_i \geq x_i^\top \beta} |y_i - x_i^\top \beta| + (1 - \tau) \sum_{i: y_i < x_i^\top \beta} |y_i - x_i^\top \beta| \right]$$

Properties of the Estimator:

1. Existence: A minimizer always exists (the objective is continuous and coercive).

2. Uniqueness: The minimizer is generally unique when $X^\top X$ is full rank, but the objective is not strictly convex, so uniqueness isn't guaranteed in degenerate cases.

3. Robustness: The estimator has bounded influence function, providing resistance to outliers.

4. Equivariance: The estimator is:

   • Location equivariant: Shifting $y$ by $a$ shifts the fitted values by $a$
   • Scale equivariant: Scaling $y$ by $c > 0$ scales the coefficients and fitted values by $c$

Interpretation of Coefficients:

For the linear quantile regression model:

$$Q_\tau(Y \mid X = x) = x^\top \beta(\tau)$$

The coefficient $\beta_j(\tau)$ represents:

The change in the τ-th conditional quantile of Y for a one-unit increase in $x_j$, holding other variables constant.

This contrasts with OLS, where $\beta_j$ represents the change in the conditional mean.

We have established the mathematical foundation of quantile regression through the quantile loss function.

What's Next:

In the next page, we'll explore the pinball loss function—an alternative formulation of quantile loss that provides additional insight and computational advantages. We'll see how pinball loss connects to probabilistic forecasting, scoring rules, and modern machine learning frameworks.