Log-Odds Interpretation

Logistic regression appears nonlinear—its predictions follow an S-shaped curve, and the relationship between features and predicted probabilities is decidedly non-linear. Yet beneath this curved surface lies a perfectly linear model.

The key insight is that logistic regression is linear on the log-odds scale. While probabilities range from 0 to 1 and odds range from 0 to infinity, log-odds span all real numbers—exactly the domain where linear models are natural.

This log-odds perspective transforms logistic regression from a 'magic squashing function' into an intuitive, interpretable model. Coefficients gain clear meaning: each unit increase in a feature multiplies the odds by a specific factor. Decision boundaries become linear hyperplanes. The entire framework suddenly makes sense.

Before diving into log-odds, we need a solid understanding of odds themselves—a concept that predates probability theory and remains central to gambling, epidemiology, and statistical inference.

Definition of Odds

Given a probability $p$ of an event occurring (where $0 < p < 1$), the odds of that event are defined as:

$$\text{odds} = \frac{p}{1-p}$$

This is the ratio of the probability of success to the probability of failure. If an event has probability 0.75, its odds are:

$$\text{odds} = \frac{0.75}{0.25} = 3$$

We often express this as "3 to 1 odds" or "odds of 3:1 in favor."

Intuitive Interpretation

Odds express how many times more likely success is compared to failure:

• Odds = 1: Success and failure equally likely ($p = 0.5$)
• Odds = 2: Success is twice as likely as failure ($p = 0.67$)
• Odds = 9: Success is nine times as likely as failure ($p = 0.90$)
• Odds = 0.5: Failure is twice as likely as success ($p = 0.33$)

The odds take values in $(0, \infty)$:

• As $p \to 0$: odds $\to 0$
• As $p \to 1$: odds $\to \infty$
• At $p = 0.5$: odds $= 1$

Converting Back from Odds to Probability

Given odds $\omega$, we can recover the probability:

$$p = \frac{\omega}{1 + \omega}$$

This is exactly the sigmoid function applied to $\log(\omega)$! The relationships form a complete system:

$$p \to \frac{p}{1-p} = \omega \to \log(\omega) = z \to \sigma(z) = p$$

Why Odds Matter

Odds have several advantages over probabilities in certain contexts:

1. Multiplicative effects are natural: Doubling the odds has a clear meaning; doubling a probability often doesn't make sense (what if $p = 0.75$?).

2. Symmetric treatment of outcomes: Odds of $k$ and odds of $1/k$ are equally 'extreme' in opposite directions.

3. Range matches linear predictors better: Probabilities are bounded; odds can take any positive value, making them easier to model linearly after a log transform.

While odds improve upon probabilities for modeling purposes, their range $(0, \infty)$ is still asymmetric around a 'neutral' value. Taking the logarithm of odds—the log-odds or logit—creates a symmetric scale spanning all real numbers.

Definition of Log-Odds (Logit)

$$\text{logit}(p) = \log\left(\frac{p}{1-p}\right) = \log(p) - \log(1-p)$$

The logit function is the inverse of the sigmoid:

$$\sigma(z) = p \iff z = \text{logit}(p)$$

Properties of the Log-Odds Scale

1. Domain and Range: logit maps $(0, 1) \to (-\infty, +\infty)$

2. Symmetry around zero: $\text{logit}(1-p) = -\text{logit}(p)$

   • $\text{logit}(0.9) = 2.20$, $\text{logit}(0.1) = -2.20$

3. Zero at neutrality: $\text{logit}(0.5) = \log(1) = 0$

4. Additive structure: Multiplicative changes in odds become additive changes in log-odds

   • If odds double, log-odds increase by $\log(2) \approx 0.693$
   • If odds triple, log-odds increase by $\log(3) \approx 1.099$

We can now state the logistic regression model equation in its most illuminating form.

The Core Equation: Linear in Log-Odds

$$\log\left(\frac{P(Y=1|X)}{1-P(Y=1|X)}\right) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p = \mathbf{w}^T \mathbf{x} + b$$

Or equivalently:

$$\text{logit}(P(Y=1|X)) = \mathbf{w}^T \mathbf{x} + b$$

This reveals the essential structure: logistic regression assumes log-odds are a linear function of the features. The nonlinearity comes entirely from inverting this relationship to obtain probabilities.

Three Equivalent Forms

The same model can be written in three equivalent ways, each revealing different aspects:

Form 1: Log-Odds (Logit) Form $$\log\left(\frac{p}{1-p}\right) = \mathbf{w}^T \mathbf{x} + b$$

Shows the linear structure directly. Coefficients are log-odds changes.

Form 2: Odds Form $$\frac{p}{1-p} = e^{\mathbf{w}^T \mathbf{x} + b} = e^{b} \cdot e^{w_1 x_1} \cdot e^{w_2 x_2} \cdots$$

Shows multiplicative structure. Coefficients are exponentiated to get odds multipliers.

Form 3: Probability Form $$p = P(Y=1|X) = \frac{e^{\mathbf{w}^T \mathbf{x} + b}}{1 + e^{\mathbf{w}^T \mathbf{x} + b}} = \sigma(\mathbf{w}^T \mathbf{x} + b)$$

The form we use for predictions. Shows the sigmoid transformation.

Why This Matters

The log-odds formulation explains several key properties of logistic regression:

1. Bounded predictions: No matter how extreme the linear predictor, probabilities stay in (0, 1).

2. Interpretable coefficients: Each coefficient has a clear meaning in terms of odds.

3. Proper uncertainty: Near the decision boundary ($z \approx 0$), small changes in features produce noticeable probability changes. Far from the boundary (saturated regions), even large feature changes barely affect the probability—encoding appropriate confidence.

4. Multiplicative odds model: Effects multiply rather than add. If drug A doubles survival odds and drug B triples them (independently), together they multiply to 6× the odds.

The log-odds formulation provides the foundation for interpreting logistic regression coefficients. Each coefficient $\beta_j$ tells us how the log-odds change per unit increase in $X_j$, holding other features constant.

The Coefficient as Log-Odds Change

Consider a single feature model: $\text{logit}(p) = \beta_0 + \beta_1 X$

When $X$ increases by 1 unit:

$$\text{logit}(p_{new}) = \beta_0 + \beta_1(X+1) = \text{logit}(p_{old}) + \beta_1$$

The log-odds increase by exactly $\beta_1$.

The Odds Ratio Interpretation

Exponentiating both sides:

$$\text{odds}{new} = e^{\beta_1} \times \text{odds}{old}$$

The quantity $e^{\beta_1}$ is called the odds ratio (OR). A one-unit increase in $X$ multiplies the odds by $e^{\beta_1}$.

Examples:

• If $\beta_1 = 0$: OR = $e^0 = 1$ — no effect on odds
• If $\beta_1 = 0.693$: OR = $e^{0.693} = 2$ — odds double
• If $\beta_1 = -0.693$: OR = $e^{-0.693} = 0.5$ — odds halve
• If $\beta_1 = 1.0$: OR = $e^1 \approx 2.72$ — odds nearly triple

While log-odds effects are constant (linear), probability effects are not. The marginal effect of a feature on probability depends on where we are in the probability space.

Mathematical Derivation

Starting from $p = \sigma(\mathbf{w}^T \mathbf{x} + b)$, the marginal effect of $X_j$ on $p$ is:

$$\frac{\partial p}{\partial X_j} = \frac{\partial \sigma(z)}{\partial z} \cdot \frac{\partial z}{\partial X_j} = \sigma(z)(1-\sigma(z)) \cdot w_j = p(1-p) \cdot w_j$$

This reveals the critical insight: the marginal effect of feature $j$ on probability is $w_j \cdot p(1-p)$.

The factor $p(1-p)$ is maximized at $p = 0.5$ (where it equals 0.25) and approaches zero as $p \to 0$ or $p \to 1$.

Implications

1. Near certainty, effects are small: If $p = 0.99$, then $p(1-p) = 0.0099$. Even a large coefficient $w_j = 2$ produces only a marginal effect of $\approx 0.02$.

2. At maximum uncertainty, effects are largest: If $p = 0.5$, then $p(1-p) = 0.25$. The same $w_j = 2$ produces a marginal effect of $0.5$.

3. Symmetry around 0.5: The marginal effect at $p = 0.1$ equals that at $p = 0.9$ (both have $p(1-p) = 0.09$).

Let's work through a complete example to solidify our understanding of log-odds interpretation.

Scenario: Predicting Heart Disease

A logistic regression model predicts heart disease probability based on:

• Age: in years
• Cholesterol: in mg/dL
• Smoker: binary (0 = no, 1 = yes)

Fitted model (hypothetical): $$\text{logit}(p) = -6.0 + 0.05 \cdot \text{Age} + 0.01 \cdot \text{Cholesterol} + 0.8 \cdot \text{Smoker}$$

Interpreting Each Coefficient

1. Intercept ($\beta_0 = -6.0$):

   • Log-odds when Age = 0, Cholesterol = 0, Smoker = 0
   • Not directly meaningful (no one has age 0), but affects baseline

2. Age ($\beta_{Age} = 0.05$):

   • Each additional year increases log-odds by 0.05
   • Odds ratio: $e^{0.05} = 1.051$
   • Interpretation: Each year of age increases heart disease odds by ~5%
   • 10-year increase: $e^{0.5} = 1.65$ — odds increase by 65%

3. Cholesterol ($\beta_{Chol} = 0.01$):

   • Each mg/dL increases log-odds by 0.01
   • Odds ratio: $e^{0.01} = 1.01$
   • Interpretation: Each 1 mg/dL increases odds by ~1%
   • 50 mg/dL increase: $e^{0.5} = 1.65$ — odds increase by 65%

4. Smoker ($\beta_{Smoker} = 0.8$):

   • Being a smoker increases log-odds by 0.8
   • Odds ratio: $e^{0.8} = 2.23$
   • Interpretation: Smokers have 2.23× the odds of heart disease compared to non-smokers

In practice, we need uncertainty estimates for our odds ratios. Since maximum likelihood estimation gives us standard errors on the log-odds scale, we construct confidence intervals there and then exponentiate.

The Procedure

1. Estimate coefficient: $\hat{\beta}$ with standard error $SE(\hat{\beta})$

2. Construct CI for log-odds coefficient: $$\hat{\beta} \pm z_{\alpha/2} \cdot SE(\hat{\beta})$$

   For 95% CI with $z_{0.025} = 1.96$: $$[\hat{\beta} - 1.96 \cdot SE(\hat{\beta}), \hat{\beta} + 1.96 \cdot SE(\hat{\beta})]$$

3. Exponentiate to get CI for odds ratio: $$[e^{\hat{\beta} - 1.96 \cdot SE(\hat{\beta})}, e^{\hat{\beta} + 1.96 \cdot SE(\hat{\beta})}]$$

Why Exponentiate the Entire Interval?

Because the exponential function is monotonic, the confidence interval transforms correctly: $$P(L \leq \beta \leq U) = P(e^L \leq e^\beta \leq e^U)$$

Note that the odds ratio CI is asymmetric around the point estimate (because exponentiation is nonlinear), but the coverage probability is preserved.