The Sigmoid Function

In the transition from regression to classification, we face a fundamental challenge: how do we convert unbounded real-valued outputs into probabilities?

Linear regression produces predictions that span all real numbers—from negative infinity to positive infinity. Yet probabilities must live in the interval [0, 1]. We need a mathematical bridge between these two worlds, and the sigmoid function (also called the logistic function) provides exactly this transformation.

This isn't merely a convenient mathematical trick. The sigmoid function emerges naturally from probabilistic reasoning about binary outcomes, making it the theoretically justified choice for classification—not just an arbitrary squashing function. Understanding why requires diving deep into its origins, properties, and connections to probability theory.

The logistic function has a fascinating history that predates machine learning by over a century. Understanding this history illuminates why the function is so natural for modeling growth processes and binary outcomes.

Origins in Population Dynamics (1838)

The Belgian mathematician Pierre François Verhulst introduced the logistic function in 1838 to model population growth. Unlike the exponential growth model proposed by Malthus (which predicts unbounded growth), Verhulst recognized that populations face resource constraints. His logistic growth model describes how populations grow quickly when small, slow as they approach carrying capacity, and eventually stabilize:

$$P(t) = \frac{K}{1 + e^{-r(t-t_0)}}$$

where $K$ is the carrying capacity, $r$ is the growth rate, and $t_0$ is the time of maximum growth rate.

Transition to Statistics (1944)

Joseph Berkson coined the term logit in 1944, introducing the logistic function to biostatistics. He recognized that the logistic function could model the probability of binary outcomes—like survival vs. death, disease vs. health—as a function of continuous predictors. This was revolutionary because it provided a principled way to perform regression on binary outcomes.

Why Not Simply Use a Linear Model for Classification?

Before examining the sigmoid function's definition, let's understand why we can't just use linear regression for classification. Consider predicting whether an email is spam (1) or not spam (0) based on word counts.

If we fit a linear regression model $\hat{y} = w^T x + b$, several problems arise:

1. Predictions outside [0, 1]: Linear models can predict values like -0.3 or 1.7, which are meaningless as probabilities.

2. Sensitivity to outliers: Extreme feature values can wildly swing predictions, making the decision boundary unstable.

3. No probabilistic interpretation: Linear regression minimizes squared error, but this doesn't correspond to any sensible probabilistic model for binary outcomes.

4. Homoscedasticity violation: Binary outcomes have variance $p(1-p)$, which depends on the prediction itself, violating linear regression assumptions.

The sigmoid function addresses all these issues by providing a principled mapping from real-valued linear predictors to valid probabilities.

The sigmoid function (or standard logistic function) is defined as:

$$\sigma(z) = \frac{1}{1 + e^{-z}} = \frac{e^z}{1 + e^z}$$

where $z \in \mathbb{R}$ is any real number, and $\sigma(z) \in (0, 1)$ is the output.

These two forms are mathematically equivalent—the second form is obtained by multiplying numerator and denominator by $e^z$. Each form has computational advantages in different contexts, as we'll see.

Deriving from a Simple Constraint

We can derive the sigmoid function axiomatically by asking: what is the simplest function $f(z)$ that satisfies these requirements?

1. $f: \mathbb{R} \to (0, 1)$ (maps reals to valid probabilities)
2. $f$ is monotonically increasing
3. $f(0) = 0.5$ (neutral input yields neutral probability)
4. $f(-z) = 1 - f(z)$ (symmetry around the origin)

Starting from the exponential family of distributions and requiring maximum entropy subject to these constraints, we arrive uniquely at the sigmoid function. This isn't coincidence—it's a deep mathematical necessity.

The sigmoid function possesses several remarkable properties that make it ideal for probabilistic classification. Understanding these properties deeply is essential for understanding logistic regression.

Property 1: Output Range (0, 1)

$$\lim_{z \to -\infty} \sigma(z) = 0, \quad \lim_{z \to +\infty} \sigma(z) = 1$$

The sigmoid function maps any real number to a value strictly between 0 and 1. This is exactly what we need to interpret outputs as probabilities. Note that the limits 0 and 1 are approached but never reached—we can never be completely certain of either class based on a logistic model.

Property 2: Symmetry

$$\sigma(-z) = 1 - \sigma(z)$$

This beautiful symmetry means that if the probability of class 1 given input $z$ is $\sigma(z)$, then the probability of class 0 is simply $\sigma(-z)$. The function is symmetric around the point $(0, 0.5)$.

Property 3: Derivative (Self-Referential Form)

$$\frac{d\sigma}{dz} = \sigma(z)(1 - \sigma(z))$$

This is perhaps the most elegant property. The derivative of the sigmoid is expressible entirely in terms of the sigmoid itself! This has profound implications:

• The derivative is always positive, confirming monotonicity
• Maximum at $z = 0$ where $\sigma'(0) = 0.25$
• Approaches 0 as $|z| \to \infty$ (saturation regions)
• Simplifies gradient computations dramatically during optimization

Property 4: Log-Odds Representation

The inverse of the sigmoid function is the logit function:

$$\text{logit}(p) = \sigma^{-1}(p) = \log\left(\frac{p}{1-p}\right)$$

The quantity $\frac{p}{1-p}$ is called the odds of success. If $p = 0.75$, the odds are $0.75/0.25 = 3$, meaning success is 3 times more likely than failure.

The logit function converts probabilities to log-odds (also called the logit scale). This reveals that logistic regression is simply linear regression on the log-odds scale:

$$\log\left(\frac{p}{1-p}\right) = w^T x + b$$

This connection is profound and will be explored in depth in the next page.

Property 5: Belongs to the Exponential Family

The sigmoid function is the canonical link function for Bernoulli-distributed outcomes. This means logistic regression can be understood as a special case of Generalized Linear Models (GLMs), providing a unified theoretical framework. This will be covered extensively in a later module.

Visualizing the sigmoid function reveals intuitions that mathematical formulas alone cannot convey. Let's examine its shape, its derivative, and how these translate to classification behavior.

The S-Curve Shape

The sigmoid produces the characteristic S-curve (hence 'sigmoid' from the Greek sigma). This shape reflects three distinct behavioral regions:

1. Saturation Region (z ≪ 0): Output close to 0, near-flat gradient. Strong evidence for class 0.

2. Linear Region (z ≈ 0): Approximately linear with slope ~0.25. Maximum uncertainty, where small input changes significantly affect output.

3. Saturation Region (z ≫ 0): Output close to 1, near-flat gradient. Strong evidence for class 1.

This behavior is exactly what we want: confidence should saturate. Once we have overwhelming evidence for a class, additional evidence shouldn't dramatically change our probability estimate.

Comparison with Step Function

The sigmoid can be viewed as a 'soft' version of the unit step function:

$$\text{step}(z) = \begin{cases} 0 & \text{if } z < 0 \ 1 & \text{if } z \geq 0 \end{cases}$$

The step function gives a hard decision but has zero gradient everywhere (except at zero, where it's undefined). This makes it completely unusable for gradient-based optimization. The sigmoid provides a smooth, differentiable approximation that retains the 'decision' character while enabling optimization.

The sigmoid function isn't an arbitrary choice—it emerges inevitably from foundational probability theory. Understanding these connections reveals logistic regression as a deeply principled model, not merely a convenient hack.

Derivation from Maximum Entropy Principle

The maximum entropy principle states that among all probability distributions consistent with known constraints, we should choose the one with maximum entropy (maximum uncertainty). For binary outcomes with a linear constraint on the expected value of features, the maximum entropy distribution is precisely the logistic model.

Given:

• Binary outcome $y \in {0, 1}$
• Feature vector $x$
• Constraint: $\mathbb{E}[y \cdot x] = \text{observed feature correlations}$

The maximum entropy solution is:

$$P(y = 1 | x) = \frac{e^{w^T x}}{1 + e^{w^T x}} = \sigma(w^T x)$$

This derivation shows that logistic regression makes the minimum assumptions necessary to respect the observed correlations between features and outcomes.

Derivation from Bernoulli-Exponential Family

The Bernoulli distribution belongs to the exponential family:

$$P(y | p) = p^y (1-p)^{1-y} = \exp\left(y \log\frac{p}{1-p} + \log(1-p)\right)$$

The natural parameter of this distribution is $\eta = \log\frac{p}{1-p}$ (the log-odds). The canonical link function maps the linear predictor to the natural parameter:

$$\eta = w^T x + b$$

Solving for $p$:

$$p = \frac{e^\eta}{1 + e^\eta} = \sigma(w^T x + b)$$

The sigmoid is thus the unique link function that makes the Bernoulli distribution a proper exponential family GLM.

The Binary Cross-Entropy Connection

When we use the sigmoid to produce probabilities, the natural loss function is the binary cross-entropy (also called log-loss):

$$\mathcal{L}(y, \hat{p}) = -[y \log(\hat{p}) + (1-y) \log(1-\hat{p})]$$

This loss is not arbitrary—it's the negative log-likelihood of the Bernoulli distribution. Minimizing cross-entropy is equivalent to maximum likelihood estimation. The sigmoid and cross-entropy are paired constructs: using one naturally implies the other.

This pairing has computational benefits too. The gradient of cross-entropy with sigmoid output simplifies beautifully:

$$\frac{\partial \mathcal{L}}{\partial z} = \hat{p} - y = \sigma(z) - y$$

This simple form (predicted probability minus true label) makes implementation clean and numerically stable.

While the sigmoid function has elegant mathematical properties, it suffers from a critical practical issue: vanishing gradients. This problem is particularly severe in deep neural networks but also affects logistic regression optimization.

The Problem Explained

Recall that the sigmoid derivative is:

$$\sigma'(z) = \sigma(z)(1 - \sigma(z))$$

This derivative has maximum value 0.25 (at z = 0) and approaches 0 exponentially as |z| increases. When the input z is far from zero in either direction:

• For z = 5: $\sigma'(5) \approx 0.0067$
• For z = 10: $\sigma'(10) \approx 0.0000454$
• For z = 20: $\sigma'(20) \approx 2 \times 10^{-9}$

These tiny gradients cause slow learning when the model is confident (whether correctly or incorrectly).

Implications for Optimization

1. Slow Convergence from Poor Initialization: If initial weights produce saturated sigmoids (outputs near 0 or 1), gradients are tiny and learning crawls.

2. Difficulty Updating Early Layers: In multi-layer networks, gradients are multiplied together. If each sigmoid contributes a factor < 0.25, gradients shrink exponentially with depth.

3. Confident Mistakes Are Slow to Fix: If the model is 99% confident and wrong, the gradient is small despite the large error. This seems counterintuitive—shouldn't we update more when we're very wrong?

While the sigmoid function is the standard for binary classification, several related functions serve different purposes or offer different properties.

The Hyperbolic Tangent (tanh)

$$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2\sigma(2z) - 1$$

The tanh function is a scaled and shifted sigmoid mapping to (-1, 1) instead of (0, 1). It's zero-centered, which can improve gradient flow in neural networks. The relationship between tanh and sigmoid is:

$$\tanh(z) = 2\sigma(2z) - 1$$

The Softmax Function (Multi-class Generalization)

For K-class classification, the sigmoid generalizes to the softmax function:

$$\text{softmax}(z)k = \frac{e^{z_k}}{\sum{j=1}^K e^{z_j}}$$

When K = 2, softmax reduces to the sigmoid:

$$\text{softmax}([z, 0])_1 = \frac{e^z}{e^z + e^0} = \frac{e^z}{e^z + 1} = \sigma(z)$$

The Probit Function

An alternative to the logistic link is the probit function, which uses the cumulative distribution function (CDF) of the standard normal distribution:

$$\Phi^{-1}(p) = \text{linear predictor}$$

Probit regression assumes errors follow a normal distribution, while logistic regression assumes a logistic distribution. In practice, results are nearly identical for most problems, but logistic regression is computationally simpler and has more interpretable coefficients (odds ratios).