Maximum Likelihood Estimation for Logistic Regression

In the previous module, we explored the logistic regression model—how the sigmoid function transforms linear combinations of features into probabilities, how the log-odds provide a natural interpretation, and how decision boundaries emerge from the model structure.

But we left a crucial question unanswered: How do we find the optimal model parameters?

In linear regression, we derived elegant closed-form solutions: the ordinary least squares estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$ minimizes squared error and equals the maximum likelihood estimator under Gaussian noise. The mathematics aligned beautifully.

Logistic regression presents a fundamentally different challenge. The response is binary—not continuous. The errors follow a Bernoulli distribution—not Gaussian. The optimization landscape is curved in ways that preclude simple matrix algebra solutions. Yet the principle remains the same: we seek parameters that make the observed data most probable.

This principle—Maximum Likelihood Estimation—forms the theoretical backbone of modern statistical learning. Understanding it deeply is essential not just for logistic regression, but for neural networks, probabilistic graphical models, and virtually every parametric model in machine learning.

Let us formally establish the probabilistic foundations of logistic regression. Recall that we model the conditional probability of the positive class:

$$P(Y = 1 | \mathbf{x}; \boldsymbol{\theta}) = \sigma(\boldsymbol{\theta}^T \mathbf{x}) = \frac{1}{1 + e^{-\boldsymbol{\theta}^T \mathbf{x}}}$$

where:

• $Y \in {0, 1}$ is the binary response variable
• $\mathbf{x} \in \mathbb{R}^{p+1}$ is the feature vector (including a 1 for the intercept)
• $\boldsymbol{\theta} \in \mathbb{R}^{p+1}$ is the parameter vector we seek to estimate
• $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid (logistic) function

The Bernoulli Distribution Connection:

Given the features $\mathbf{x}$, the response $Y$ follows a Bernoulli distribution with success probability $\pi(\mathbf{x}) = \sigma(\boldsymbol{\theta}^T \mathbf{x})$:

$$Y | \mathbf{x} \sim \text{Bernoulli}(\pi(\mathbf{x}))$$

The probability mass function is:

$$P(Y = y | \mathbf{x}; \boldsymbol{\theta}) = \pi(\mathbf{x})^y (1 - \pi(\mathbf{x}))^{1-y}$$

This compact expression unifies both cases:

• When $y = 1$: $P(Y = 1) = \pi(\mathbf{x})^1 (1-\pi(\mathbf{x}))^0 = \pi(\mathbf{x})$
• When $y = 0$: $P(Y = 0) = \pi(\mathbf{x})^0 (1-\pi(\mathbf{x}))^1 = 1 - \pi(\mathbf{x})$

Key Properties of the Bernoulli Model:

Before constructing the likelihood function for logistic regression, let us deeply understand what "likelihood" means in statistics.

Likelihood vs. Probability — The Critical Distinction:

Both likelihood and probability involve the same mathematical formula, but they answer fundamentally different questions:

$$\mathcal{L}(\boldsymbol{\theta}; \text{data}) = P(\text{data}; \boldsymbol{\theta})$$

Intuitive Example:

Suppose you flip a coin 10 times and observe 7 heads.

• Probability question: "If the coin is fair ($\theta = 0.5$), what's the probability of exactly 7 heads?"
• Likelihood question: "Given we observed 7 heads, how plausible is it that $\theta = 0.5$ vs. $\theta = 0.7$ vs. $\theta = 0.9$?"

The probability calculation is: $$P(7 \text{ heads} | \theta) = \binom{10}{7} \theta^7 (1-\theta)^3$$

The same formula, viewed as a function of $\theta$ with data fixed, gives the likelihood: $$\mathcal{L}(\theta; 7 \text{ heads}) = \binom{10}{7} \theta^7 (1-\theta)^3$$

The Maximum Likelihood Principle:

Given observed data, we estimate parameters by choosing values that maximize the likelihood—the values under which the observed data would be most probable:

$$\hat{\boldsymbol{\theta}}_{\text{MLE}} = \underset{\boldsymbol{\theta}}{\arg\max} ; \mathcal{L}(\boldsymbol{\theta}; \text{data})$$

This is intuitive: we prefer parameter values that render our observations unsurprising. If parameters $\boldsymbol{\theta}_A$ would make the data extremely unlikely, while $\boldsymbol{\theta}_B$ would make it probable, we favor $\boldsymbol{\theta}_B$.

Why Maximum Likelihood?

Now we construct the likelihood function for logistic regression. Suppose we have $n$ independent observations:

$${(\mathbf{x}_1, y_1), (\mathbf{x}_2, y_2), \ldots, (\mathbf{x}_n, y_n)}$$

where each $\mathbf{x}_i \in \mathbb{R}^{p+1}$ (including intercept) and each $y_i \in {0, 1}$.

Independence Assumption:

We assume observations are conditionally independent given the features. This means the probability of observing the entire dataset is the product of individual observation probabilities:

$$P(y_1, y_2, \ldots, y_n | \mathbf{x}_1, \ldots, \mathbf{x}n; \boldsymbol{\theta}) = \prod{i=1}^{n} P(y_i | \mathbf{x}_i; \boldsymbol{\theta})$$

The Likelihood Function:

Substituting the Bernoulli probability mass function:

$$\mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^{n} \pi_i^{y_i} (1 - \pi_i)^{1-y_i}$$

where we use shorthand $\pi_i = \pi(\mathbf{x}_i) = \sigma(\boldsymbol{\theta}^T \mathbf{x}_i)$.

Expanding this expression:

For a concrete dataset, consider what this product looks like:

The total likelihood is: $$\mathcal{L}(\boldsymbol{\theta}) = \pi_1 \cdot (1-\pi_2) \cdot \pi_3 \cdot \pi_4 \cdot (1-\pi_5) \cdot \ldots$$

Interpretation:

The likelihood measures how well our model "explains" the observed labels:

• For each positive example ($y_i = 1$), we want $\pi_i$ to be high
• For each negative example ($y_i = 0$), we want $1 - \pi_i$ to be high (equivalently, $\pi_i$ to be low)

Good parameters $\boldsymbol{\theta}$ assign high probability to positive examples and low probability to negative examples—exactly what a good classifier should do.

The Likelihood as a Function of Parameters:

It's crucial to view the likelihood as a function of $\boldsymbol{\theta}$. Substituting $\pi_i = \sigma(\boldsymbol{\theta}^T \mathbf{x}_i)$:

$$\mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^{n} \left[\sigma(\boldsymbol{\theta}^T \mathbf{x}_i)\right]^{y_i} \left[1 - \sigma(\boldsymbol{\theta}^T \mathbf{x}_i)\right]^{1-y_i}$$

This is a nonlinear function of $\boldsymbol{\theta}$ due to the sigmoid. The product structure and the nonlinearity are what make logistic regression fundamentally different from linear regression.

Using the Sigmoid Property:

Recall that the sigmoid has a beautiful symmetry: $1 - \sigma(z) = \sigma(-z)$. This allows us to write:

$$\mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^{n} \sigma(\boldsymbol{\theta}^T \mathbf{x}_i)^{y_i} \cdot \sigma(-\boldsymbol{\theta}^T \mathbf{x}_i)^{1-y_i}$$

This symmetric form hints at deeper structure we'll exploit in optimization.

To build intuition, let's visualize the likelihood surface for a simple one-dimensional example (one feature plus intercept, so $\boldsymbol{\theta} = (\theta_0, \theta_1)^T$).

Visualization Setup:

Consider a small dataset where we vary the parameters and compute the likelihood at each point. The likelihood surface reveals the optimization landscape.

What the Likelihood Surface Reveals:

Understanding how individual observations contribute to the likelihood provides deep insight into the learning process.

Decomposing the Likelihood:

Each observation $(\mathbf{x}_i, y_i)$ contributes a factor to the likelihood:

$$\mathcal{L}_i(\boldsymbol{\theta}) = \pi_i^{y_i} (1-\pi_i)^{1-y_i}$$

The total likelihood is the product: $\mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^n \mathcal{L}_i(\boldsymbol{\theta})$

How Different Observations Affect the Likelihood:

The Multiplicative Effect:

Because likelihood uses multiplication, a single misclassified example can dramatically reduce the total likelihood. If one observation has $\mathcal{L}_i = 0.001$, the entire dataset's likelihood is reduced by a factor of 1000.

Consider two parameter settings:

• Setting A: 99 observations with $\mathcal{L}_i = 0.9$, 1 observation with $\mathcal{L}_i = 0.01$
• Setting B: All 100 observations with $\mathcal{L}_i = 0.7$

Setting A: $\mathcal{L} = 0.9^{99} \times 0.01 \approx 2.5 \times 10^{-7}$ Setting B: $\mathcal{L} = 0.7^{100} \approx 3.2 \times 10^{-16}$

Wait—Setting B is worse! But the numbers are hard to parse. This is why we use log-likelihood:

• Setting A: $\ell = 99 \cdot \log(0.9) + \log(0.01) \approx -10.43 - 4.61 = -15.04$
• Setting B: $\ell = 100 \cdot \log(0.7) \approx -35.67$

Setting A is actually better despite the poor fit on one observation.

Gradient Intuition:

When optimizing, the gradient of the likelihood (or log-likelihood) tells us how to adjust parameters. Observations that are confidently correct provide less gradient signal—they're already well-handled. Observations that are misclassified or uncertain provide strong gradient signal—they're where the model needs to improve.

This is fundamentally different from squared error, where all observations contribute proportionally to their error magnitude. The logistic likelihood naturally focuses optimization effort on the observations that matter most for classification.

The likelihood function has deep connections to information theory, revealing why Maximum Likelihood Estimation produces good classifiers.

Likelihood and Coding Theory:

From an information-theoretic perspective, the likelihood measures how "surprised" we would be to see the data under a given model. High likelihood means low surprise—the model "expected" this data.

The negative log-likelihood is directly related to the codelength required to transmit the data using a code based on the model's probability distribution:

$$\text{Codelength} = -\log_2 \mathcal{L}(\boldsymbol{\theta}) = -\sum_{i=1}^n \log_2 P(y_i | \mathbf{x}_i; \boldsymbol{\theta})$$

Minimizing negative log-likelihood (equivalently, maximizing likelihood) corresponds to finding the model that most efficiently "compresses" the observed labels.

Cross-Entropy Connection:

The cross-entropy between the true label distribution and model predictions is:

$$H(p, q) = -\sum_y p(y) \log q(y)$$

For our binary case with empirical distribution:

$$H(y, \pi) = -\frac{1}{n}\sum_{i=1}^n \left[ y_i \log \pi_i + (1-y_i) \log(1-\pi_i) \right]$$

This is exactly the negative log-likelihood divided by $n$! The cross-entropy loss used in neural network training is directly derived from the likelihood principle.

Logistic regression fits within the broader framework of Generalized Linear Models (GLMs), which are based on the exponential family of distributions. This perspective illuminates deep structural properties.

Exponential Family Form:

A distribution belongs to the exponential family if its probability density/mass function can be written as:

$$p(y; \eta) = h(y) \exp\left( \eta \cdot T(y) - A(\eta) \right)$$

where:

• $\eta$ is the natural (canonical) parameter
• $T(y)$ is the sufficient statistic
• $A(\eta)$ is the log-partition function (normalizing constant)
• $h(y)$ is the base measure

Bernoulli as Exponential Family:

For Bernoulli$(\pi)$:

$$P(Y = y) = \pi^y (1-\pi)^{1-y}$$

Taking the logarithm and rearranging:

$$\log P(Y = y) = y \log\pi + (1-y)\log(1-\pi) = y\log\frac{\pi}{1-\pi} + \log(1-\pi)$$

Therefore:

• Natural parameter: $\eta = \log\frac{\pi}{1-\pi}$ (the log-odds!)
• Sufficient statistic: $T(y) = y$
• Log-partition function: $A(\eta) = \log(1 + e^\eta)$
• Base measure: $h(y) = 1$

Why the Exponential Family Matters:

The exponential family structure has profound implications:

We have established the theoretical foundation for Maximum Likelihood Estimation in logistic regression. Let's consolidate the key concepts before moving to the log-likelihood derivation.

What's Next:

The likelihood as written involves a product of many terms—computationally problematic and difficult to differentiate. In the next page, we'll transform to log-likelihood, which converts products to sums, eliminates numerical underflow, and reveals the beautiful concave structure that makes optimization tractable. We'll derive the log-likelihood explicitly and prove its concavity—the property that guarantees global optimality of gradient-based solutions.