Regularized Logistic Regression

Logistic regression, despite its conceptual elegance and interpretability, faces a fundamental challenge that becomes critical as model complexity increases: overfitting. When we fit a logistic regression model to data—particularly in high-dimensional settings where the number of features approaches or exceeds the number of observations—the maximum likelihood estimator can exhibit pathological behavior that renders the model useless for prediction.

L2 regularization (also known as Ridge regularization or Tikhonov regularization in the broader numerical analysis context) provides a principled solution to this problem by introducing a penalty term that constrains the magnitude of the model coefficients. This page develops the complete theory of L2-regularized logistic regression, from mathematical foundations to practical implementation.

Before diving into the solution, we must thoroughly understand the problem. Overfitting in logistic regression manifests differently than in linear regression, and understanding these nuances is essential for appreciating why regularization is necessary.

1.1 Maximum Likelihood Estimation Pathologies

In standard logistic regression, we maximize the log-likelihood:

$$\mathcal{L}(\boldsymbol{\beta}) = \sum_{i=1}^{n} \left[ y_i \log p_i + (1-y_i) \log(1-p_i) \right]$$

where $p_i = \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})$ and $\sigma(z) = 1/(1+e^{-z})$ is the sigmoid function.

The Perfect Separation Problem: When data is linearly separable, the MLE does not exist in the finite parameter space. The optimization algorithm will drive coefficients toward infinity, attempting to create a decision boundary that perfectly separates the classes.

1.2 High-Dimensional Settings

When the number of features $p$ is large relative to the sample size $n$, several problems emerge:

Variance Explosion: Even when the MLE exists, the estimator variance becomes extremely large. Small changes in training data cause dramatic changes in the estimated coefficients, making the model unreliable.

Multicollinearity: Correlated features create near-singular design matrices, amplifying coefficient instability. A small amount of noise can completely change which correlated feature gets the "credit" for predictive power.

Spurious Correlations: With many features, some will be spuriously correlated with the outcome by chance alone. The MLE will exploit these spurious correlations, fitting noise rather than signal.

1.3 The Bias-Variance Tradeoff Revisited

For classification, we care about the expected prediction error on new data. This error can be decomposed:

$$\text{EPE} = \text{Irreducible Error} + \text{Bias}^2 + \text{Variance}$$

Unregularized MLE has zero bias (it's asymptotically unbiased under correct model specification) but potentially unbounded variance in problematic settings. Regularization introduces controlled bias in exchange for substantial variance reduction, often improving overall prediction accuracy.

This is not merely a theoretical concern—it's the fundamental justification for regularization. We deliberately move away from the "best" unbiased estimator because bias can be a worthwhile price for stability.

2.1 The Regularized Objective Function

L2 regularization modifies the maximum likelihood objective by adding a penalty term proportional to the squared L2 norm of the coefficient vector. The regularized objective becomes:

$$\mathcal{L}{\lambda}(\boldsymbol{\beta}) = \sum{i=1}^{n} \left[ y_i \log p_i + (1-y_i) \log(1-p_i) \right] - \frac{\lambda}{2} |\boldsymbol{\beta}|_2^2$$

or equivalently, we minimize the penalized negative log-likelihood:

$$J(\boldsymbol{\beta}) = -\sum_{i=1}^{n} \left[ y_i \log p_i + (1-y_i) \log(1-p_i) \right] + \frac{\lambda}{2} \sum_{j=1}^{p} \beta_j^2$$

where:

• $\lambda \geq 0$ is the regularization strength (hyperparameter)
• $|\boldsymbol{\beta}|2^2 = \sum{j=1}^{p} \beta_j^2$ is the squared L2 norm
• The factor of $1/2$ is a convention that simplifies derivatives

2.2 Constrained Optimization Perspective

The penalized formulation is equivalent to a constrained optimization problem. For every value of $\lambda$, there exists a corresponding constraint $t$ such that:

$$\begin{aligned} \text{maximize} \quad & \mathcal{L}(\boldsymbol{\beta}) \ \text{subject to} \quad & |\boldsymbol{\beta}|_2^2 \leq t \end{aligned}$$

This constrained formulation is often more intuitive: we're maximizing the likelihood subject to a "budget" constraint on how large coefficients can be. The relationship between $\lambda$ and $t$ is monotonic—larger $\lambda$ corresponds to smaller $t$ (tighter constraint).

Geometric Interpretation: The constraint defines an ellipsoid in parameter space (specifically, a hypersphere when all features are on the same scale). We seek the point on this ellipsoid that maximizes the log-likelihood.

2.3 Alternative Parameterizations

Different software packages use different parameterizations of the regularization strength:

scikit-learn convention: Uses $C$ where larger values mean less regularization (confusing but historical). The objective becomes:

$$J(\boldsymbol{\beta}) = \frac{1}{C} \cdot \frac{1}{2}|\boldsymbol{\beta}|2^2 + \frac{1}{n}\sum{i=1}^{n} \ell(y_i, \hat{y}_i)$$

Always verify the parameterization when using new software to ensure correct hyperparameter specification.

3.1 The Shrinkage Effect

L2 regularization implements shrinkage: it systematically pulls coefficient estimates toward zero. This isn't merely a computational trick but has profound statistical consequences.

Consider the gradient of the L2 penalty:

$$\frac{\partial}{\partial \beta_j} \left( \frac{\lambda}{2} \sum_k \beta_k^2 \right) = \lambda \beta_j$$

This gradient is proportional to the coefficient value itself. Large coefficients incur larger gradients pushing them toward zero, while small coefficients experience less penalty. The result is proportional shrinkage:

$$\hat{\beta}_j^{\text{ridge}} \approx \frac{1}{1 + \lambda/\text{curvature}} \cdot \hat{\beta}_j^{\text{MLE}}$$

where "curvature" refers to the second derivative of the log-likelihood. Coefficients are shrunk toward zero by a factor that depends on both $\lambda$ and how well-determined they are by the data.

3.2 Geometric Visualization

The geometry of L2 regularization illuminates why it works:

Likelihood Contours: In the two-coefficient case, the log-likelihood defines contour lines (curves of constant likelihood) in the $(\beta_1, \beta_2)$ plane. The MLE sits at the center of these concentric ellipses.

Constraint Region: The L2 constraint $\beta_1^2 + \beta_2^2 \leq t$ defines a circular disk centered at the origin.

Solution Location: The regularized solution occurs where the highest-valued likelihood contour touches the constraint disk. Due to the circular shape, this tangent point typically has both coefficients non-zero but smaller in magnitude than the MLE.

3.3 Effect on Correlated Features

L2 regularization handles correlated features gracefully through coefficient sharing:

When two features $x_1$ and $x_2$ are highly correlated and both predictive of $y$:

• Unregularized MLE: Arbitrarily assigns most predictive power to one feature, with the other potentially receiving an opposite-signed coefficient for "correction"
• L2 Regularized: Distributes weight more evenly between correlated predictors, producing stable, interpretable coefficients

This behavior arises because the L2 penalty is minimized when equal total weight is spread across correlated features rather than concentrated in one. If $\beta_1 + \beta_2 = c$ is required for good fit, then $\beta_1 = \beta_2 = c/2$ minimizes $\beta_1^2 + \beta_2^2$.

4.1 Convexity Guarantee

The L2-regularized logistic regression objective is strictly convex when $\lambda > 0$. This has profound computational implications:

Original Log-Likelihood: The negative log-likelihood is convex (the sigmoid function's log-likelihood is concave), but not strictly convex—the Hessian can be singular when features are collinear.

Adding L2 Penalty: The L2 penalty $\frac{\lambda}{2}|\boldsymbol{\beta}|_2^2$ is strictly convex with Hessian $\lambda \mathbf{I}$. Adding it to any convex function yields a strictly convex function.

Consequence: The regularized objective has a unique global minimum. Any local minimum is the global minimum, and gradient-based methods are guaranteed to converge to it.

4.2 Gradient and Hessian

For efficient optimization, we need the gradient and Hessian of the regularized objective.

Gradient (for minimization):

$$\nabla J(\boldsymbol{\beta}) = \sum_{i=1}^{n} (p_i - y_i) \mathbf{x}_i + \lambda \boldsymbol{\beta}$$

Or in matrix form:

$$\nabla J(\boldsymbol{\beta}) = \mathbf{X}^\top (\mathbf{p} - \mathbf{y}) + \lambda \boldsymbol{\beta}$$

where $\mathbf{p} = [p_1, \ldots, p_n]^\top$ is the vector of predicted probabilities.

Hessian:

$$\mathbf{H} = \nabla^2 J(\boldsymbol{\beta}) = \mathbf{X}^\top \mathbf{W} \mathbf{X} + \lambda \mathbf{I}$$

where $\mathbf{W} = \text{diag}(p_1(1-p_1), \ldots, p_n(1-p_n))$ is a diagonal matrix of variance weights.

The regularization term $\lambda \mathbf{I}$ ensures the Hessian is always positive definite, guaranteeing well-conditioned optimization.

4.3 Optimization Algorithms

Several algorithms efficiently solve the L2-regularized problem:

Newton-Raphson / IRLS: The Iteratively Reweighted Least Squares algorithm is Newton's method applied to logistic regression. With L2 regularization, the update becomes:

$$\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - (\mathbf{H} + \lambda \mathbf{I})^{-1} \nabla J(\boldsymbol{\beta}^{(t)})$$

This converges quadratically near the optimum.

L-BFGS: A quasi-Newton method that approximates the Hessian using gradient history. Efficient for medium-scale problems where the Hessian is expensive to compute or store.

Stochastic Gradient Descent (SGD): For massive datasets, SGD updates parameters using gradient estimates from mini-batches. The L2 penalty gradient is trivially incorporated as $\lambda \boldsymbol{\beta}$.

5.1 Ridge as Gaussian Prior

L2 regularization admits a compelling Bayesian interpretation: it corresponds to Maximum A Posteriori (MAP) estimation with a Gaussian prior on the coefficients.

Setup:

• Likelihood: $p(\mathbf{y} | \mathbf{X}, \boldsymbol{\beta})$ is the binomial likelihood from logistic regression
• Prior: $p(\boldsymbol{\beta}) = \mathcal{N}(\mathbf{0}, \tau^2 \mathbf{I})$, a multivariate Gaussian centered at zero

MAP Objective: We maximize the posterior probability:

$$\hat{\boldsymbol{\beta}}{\text{MAP}} = \arg\max{\boldsymbol{\beta}} , p(\boldsymbol{\beta} | \mathbf{y}, \mathbf{X}) = \arg\max_{\boldsymbol{\beta}} , p(\mathbf{y} | \mathbf{X}, \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta})$$

Taking the log:

$$\log p(\boldsymbol{\beta} | \mathbf{y}, \mathbf{X}) \propto \underbrace{\sum_i [y_i \log p_i + (1-y_i)\log(1-p_i)]}_{\text{Log-likelihood}} - \underbrace{\frac{1}{2\tau^2} |\boldsymbol{\beta}|2^2}{\text{Log-prior}}$$

Setting $\lambda = 1/\tau^2$ recovers the L2-penalized objective exactly.

5.2 Implications of the Gaussian Prior

The Gaussian prior has several important properties that explain L2 regularization behavior:

Continuous Density at Zero: Unlike the Laplace prior (L1), the Gaussian prior has a smooth, continuous density at zero with zero derivative. This explains why L2 regularization shrinks coefficients smoothly toward zero without ever reaching exactly zero.

Quadratic Penalization: The log of a Gaussian is a quadratic function. This ensures the penalty is differentiable everywhere, enabling efficient gradient-based optimization.

Independence Assumption: The isotropic prior $\mathcal{N}(\mathbf{0}, \tau^2 \mathbf{I})$ assumes coefficients are independent a priori. This is a simplifying assumption—in practice, domain knowledge might suggest correlated priors, leading to more sophisticated regularizers.

5.3 Posterior Approximation

While we typically compute only the MAP estimate, the full Bayesian approach would compute the posterior distribution:

$$p(\boldsymbol{\beta} | \mathbf{y}, \mathbf{X}) \propto p(\mathbf{y} | \mathbf{X}, \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta})$$

This posterior is not analytically tractable for logistic regression (unlike linear regression with conjugate priors). However, we can approximate it using:

• Laplace approximation: Gaussian approximation centered at the MAP estimate
• Variational inference: Optimize a tractable approximating distribution
• MCMC: Sample from the posterior using Markov Chain Monte Carlo

These approaches provide uncertainty quantification—not just point estimates but credible intervals for coefficients and predictions.

6.1 Preprocessing Requirements

Feature Standardization: As emphasized earlier, L2 regularization requires standardized features. The standard approach:

$$\tilde{x}{ij} = \frac{x{ij} - \bar{x}_j}{s_j}$$

where $\bar{x}_j$ and $s_j$ are the mean and standard deviation of feature $j$.

Binary and Categorical Features: For binary features (0/1), standardization still applies. For categorical features, use appropriate encoding (one-hot or effect coding) and standardize the resulting columns.

Target Encoding: The target variable $y$ should be 0/1 (not -1/+1 or other encodings) for standard log-loss formulation.

6.2 Handling Class Imbalance

L2 regularization interacts with class imbalance in important ways:

Intercept Bias: With unbalanced classes (e.g., 1% positives), the fitted intercept captures the prior class probability. The intercept should not be regularized to preserve this calibration.

Weighted Samples: For highly imbalanced problems, weight the log-likelihood contributions:

$$J(\boldsymbol{\beta}) = -\sum_{i=1}^{n} w_i \left[ y_i \log p_i + (1-y_i) \log(1-p_i) \right] + \frac{\lambda}{2} |\boldsymbol{\beta}|_2^2$$

Common weighting schemes:

• Inverse class frequency: $w_i = 1/n_c$ where $n_c$ is the count of class $c$
• Balanced weights: class_weight='balanced' in scikit-learn

6.3 Coefficient Interpretation

L2-regularized coefficients require careful interpretation:

Shrinkage Adjustment: Coefficients are systematically shrunk toward zero. Their magnitudes are biased estimates of the true effect sizes. Use them for prediction and ranking feature importance, but not for unbiased effect size estimation.

Relative Importance: Larger |β_j| indicates greater predictive importance, but the absolute scale depends on λ. Compare coefficients within a model, not across models with different λ.

Odds Ratio Interpretation: For standardized features, $e^{\beta_j}$ gives the multiplicative change in odds for a one-standard-deviation increase in feature $j$. This interpretation is approximate due to regularization bias.

L2 regularization transforms logistic regression from a potentially unstable estimator into a robust, reliable classifier. We've covered the complete theory from multiple perspectives.