A model is only as good as the signal guiding its learning. For classification tasks, that signal comes from the cross-entropy loss—a function so fundamental that it appears in virtually every neural network classifier, from simple logistic regression to state-of-the-art transformers.

Cross-entropy isn't an arbitrary choice. It emerges naturally from maximum likelihood estimation, has deep roots in information theory, and possesses mathematical properties that make gradient-based optimization particularly effective. Understanding cross-entropy deeply is essential for anyone serious about machine learning.

The cross-entropy loss emerges naturally when we apply maximum likelihood estimation to the multinomial logistic regression model. Let's trace this derivation carefully.

The Setup

We have $n$ training examples ${(\mathbf{x}i, y_i)}{i=1}^n$ where:

• $\mathbf{x}_i \in \mathbb{R}^d$ is the feature vector
• $y_i \in {1, 2, \ldots, K}$ is the true class label

Our model predicts: $$P(y = k | \mathbf{x}; \boldsymbol{\theta}) = \hat{p}_k = \text{softmax}(\mathbf{z})k = \frac{e^{z_k}}{\sum{j=1}^{K} e^{z_j}}$$

where $z_k = \mathbf{w}_k^T \mathbf{x} + b_k$ and $\boldsymbol{\theta}$ comprises all parameters.

Writing the Likelihood

Assuming training examples are i.i.d., the likelihood of observing the data given parameters is:

$$L(\boldsymbol{\theta}) = \prod_{i=1}^{n} P(y_i | \mathbf{x}i; \boldsymbol{\theta}) = \prod{i=1}^{n} \hat{p}_{i, y_i}$$

where $\hat{p}_{i,k}$ denotes the predicted probability of class $k$ for sample $i$.

Taking the Log-Likelihood

Maximizing likelihood is equivalent to maximizing log-likelihood (log is monotonic):

$$\ell(\boldsymbol{\theta}) = \log L(\boldsymbol{\theta}) = \sum_{i=1}^{n} \log \hat{p}_{i, y_i}$$

To use one-hot encoding, let $\mathbf{y}i \in {0,1}^K$ be the one-hot representation of label $y_i$ (i.e., $y{ik} = 1$ if $y_i = k$, else 0). Then:

$$\log \hat{p}{i, y_i} = \sum{k=1}^{K} y_{ik} \log \hat{p}_{ik}$$

(Only the true class contributes since $y_{ik} = 0$ for incorrect classes.)

Thus: $$\ell(\boldsymbol{\theta}) = \sum_{i=1}^{n} \sum_{k=1}^{K} y_{ik} \log \hat{p}_{ik}$$

Converting to Loss (Negative Log-Likelihood)

Optimization frameworks minimize losses, so we negate:

$$\mathcal{L}{\text{NLL}}(\boldsymbol{\theta}) = -\ell(\boldsymbol{\theta}) = -\sum{i=1}^{n} \sum_{k=1}^{K} y_{ik} \log \hat{p}_{ik}$$

This is exactly the categorical cross-entropy loss (also called softmax loss or log loss).

Why Not Squared Error?

One might ask: why not simply minimize $(y_k - \hat{p}_k)^2$? Several reasons:

1. Probabilistic foundation: Cross-entropy is the MLE objective; squared error assumes Gaussian noise, inappropriate for categorical outcomes.

2. Gradient behavior: For softmax + squared error, gradients can saturate (become tiny) when predictions are very wrong. Cross-entropy gradients remain informative.

3. Convexity: Cross-entropy with softmax is convex in the logits. Squared error with softmax is not.

4. Information-theoretic meaning: Cross-entropy has a principled interpretation (next section); squared error for probabilities does not.

Cross-entropy has deep roots in information theory, providing a principled interpretation beyond its role as a likelihood-derived loss.

Entropy Review

For a discrete probability distribution $\mathbf{p} = (p_1, \ldots, p_K)$, the entropy is:

$$H(\mathbf{p}) = -\sum_{k=1}^{K} p_k \log p_k$$

Entropy measures the average 'surprise' or uncertainty about outcomes sampled from $\mathbf{p}$. It's measured in nats (natural log) or bits (log base 2).

Key properties:

• $H(\mathbf{p}) \geq 0$ always
• $H(\mathbf{p}) = 0$ iff $\mathbf{p}$ is deterministic (one $p_k = 1$)
• $H(\mathbf{p})$ is maximized when $\mathbf{p}$ is uniform

Cross-Entropy Definition

The cross-entropy between distributions $\mathbf{p}$ (true) and $\mathbf{q}$ (predicted) is:

$$H(\mathbf{p}, \mathbf{q}) = -\sum_{k=1}^{K} p_k \log q_k$$

Interpretation: The average number of nats/bits needed to encode samples from $\mathbf{p}$ using a code optimized for $\mathbf{q}$.

KL Divergence Connection

The Kullback-Leibler divergence (relative entropy) from $\mathbf{q}$ to $\mathbf{p}$ is:

$$D_{KL}(\mathbf{p} | \mathbf{q}) = \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k} = -H(\mathbf{p}) + H(\mathbf{p}, \mathbf{q})$$

Rearranging: $$H(\mathbf{p}, \mathbf{q}) = H(\mathbf{p}) + D_{KL}(\mathbf{p} | \mathbf{q})$$

Key insight: Cross-entropy = Entropy + KL Divergence.

Since $H(\mathbf{p})$ is fixed (depends only on true labels), minimizing cross-entropy is equivalent to minimizing KL divergence between true and predicted distributions.

$$\text{Minimizing } H(\mathbf{p}, \hat{\mathbf{p}}) \iff \text{Minimizing } D_{KL}(\mathbf{p} | \hat{\mathbf{p}})$$

Properties of KL Divergence:

• $D_{KL}(\mathbf{p} | \mathbf{q}) \geq 0$ (Gibbs' inequality)
• $D_{KL}(\mathbf{p} | \mathbf{q}) = 0$ iff $\mathbf{p} = \mathbf{q}$
• Not symmetric: $D_{KL}(\mathbf{p} | \mathbf{q}) eq D_{KL}(\mathbf{q} | \mathbf{p})$ in general

Cross-Entropy for One-Hot Labels

In classification, true labels are one-hot: $\mathbf{y} = (0, \ldots, 0, 1, 0, \ldots, 0)$ with a single 1 at the true class $c$.

The entropy of a one-hot distribution is zero: $$H(\mathbf{y}) = -1 \cdot \log(1) - 0 \cdot \log(0) = 0$$

(Using convention $0 \cdot \log(0) = 0$.)

So: $$H(\mathbf{y}, \hat{\mathbf{p}}) = D_{KL}(\mathbf{y} | \hat{\mathbf{p}}) = -\log \hat{p}_c$$

Cross-entropy reduces to the negative log-probability of the true class.

Intuition:

• If $\hat{p}_c \to 1$: loss $\to 0$ (correct prediction, confident)
• If $\hat{p}_c \to 0$: loss $\to \infty$ (wrong prediction, very penalized)

The logarithm creates an asymmetric, sharp penalty for confident wrong predictions.

Cross-entropy's mathematical properties make it ideal for optimization. Understanding these properties explains why it's the universal choice for classification.

Property 1: Non-Negativity

$$\mathcal{L}_{CE} = -\log \hat{p}_y \geq 0$$

since $\hat{p}_y \in (0, 1)$ implies $\log \hat{p}_y \leq 0$.

The minimum occurs when $\hat{p}y = 1$, giving $\mathcal{L}{CE} = 0$. However, softmax never outputs exactly 1 (only approaches it as logits $\to \pm\infty$), so the practical minimum is approached asymptotically.

Property 2: Convexity in Logits

For multinomial logistic regression, the cross-entropy loss is convex as a function of the logits $\mathbf{z}$ (and hence the parameters $\boldsymbol{\theta}$).

Proof sketch: The negative log-likelihood can be written as: $$\mathcal{L} = -z_y + \log\left(\sum_{j=1}^{K} e^{z_j}\right)$$

• $-z_y$ is linear (hence convex) in $z_y$
• $\log\sum e^{z_j}$ (log-sum-exp) is convex

Sum of convex functions is convex. Since $z_k = \mathbf{w}_k^T \mathbf{x} + b_k$ is linear in parameters, composition with convex loss gives a convex optimization problem.

Implication: Gradient descent finds the global optimum (if one exists). No local minima trap the optimizer.

Property 3: Elegant Gradient Structure

The gradient of cross-entropy with respect to logits has a remarkably simple form:

$$\frac{\partial \mathcal{L}}{\partial z_k} = \hat{p}_k - y_k$$

where $y_k = 1$ if $k$ is the true class, else $0$.

Derivation:

For true class $c$: $$\mathcal{L} = -z_c + \log\left(\sum_j e^{z_j}\right)$$

$$\frac{\partial \mathcal{L}}{\partial z_k} = -\mathbf{1}_{k=c} + \frac{e^{z_k}}{\sum_j e^{z_j}} = -y_k + \hat{p}_k = \hat{p}_k - y_k$$

Interpretation:

• For the true class $c$: gradient = $\hat{p}_c - 1 < 0$, pushing to increase $z_c$
• For other classes $k eq c$: gradient = $\hat{p}_k > 0$, pushing to decrease $z_k$
• Gradient magnitude proportional to error: More wrong → larger gradient

This $(\text{prediction} - \text{target})$ structure is the hallmark of exponential family likelihoods with canonical link functions.

Property 4: Calibration Under Correct Specification

When the model is correctly specified and trained to convergence, cross-entropy training produces well-calibrated probability estimates: predicted probabilities match observed frequencies.

Formally, among all test samples where the model predicts $P(y=k) \approx p$, approximately a fraction $p$ actually belong to class $k$.

This occurs because:

1. MLE is consistent under correct specification
2. Cross-entropy is the proper scoring rule for categorical distributions
3. No systematic bias in the loss encourages over/under-confidence

However, deep neural networks often exhibit miscalibration (overconfidence). Post-hoc calibration methods (temperature scaling, Platt scaling) can address this.

Property 5: Information-Theoretic Optimality

Cross-entropy is a proper scoring rule: the expected score is uniquely minimized when the predicted distribution equals the true distribution. This prevents 'gaming' the loss with non-informative predictions.

Computing cross-entropy naively leads to numerical disasters. Understanding and preventing these issues is essential for robust implementations.

Issue 1: Log of Softmax Underflow

Softmax outputs can be very small. For logits $\mathbf{z} = (10, 0, 0)$:

$$\hat{p}_2 = \frac{e^0}{e^{10} + e^0 + e^0} \approx \frac{1}{22048} \approx 4.5 \times 10^{-5}$$

For more extreme logits $\mathbf{z} = (100, 0, 0)$:

$$\hat{p}_2 \approx e^{-100} \approx 3.7 \times 10^{-44}$$

If $\hat{p}_2$ underflows to 0 and the true class is 2, then $\log(0) = -\infty$.

Issue 2: Overflow in Softmax

As discussed in the softmax page, large logits cause $e^z$ to overflow before the ratio is computed.

The Solution: Log-Softmax

Compute log-probabilities directly without computing probabilities first:

$$\log \hat{p}_k = \log \frac{e^{z_k}}{\sum_j e^{z_j}} = z_k - \log\left(\sum_j e^{z_j}\right) = z_k - \text{LSE}(\mathbf{z})$$

where $\text{LSE}(\mathbf{z}) = \log\sum_j e^{z_j}$ is computed stably:

$$\text{LSE}(\mathbf{z}) = m + \log\left(\sum_j e^{z_j - m}\right) \quad \text{where } m = \max(\mathbf{z})$$

Now cross-entropy is: $$\mathcal{L} = -\log \hat{p}_y = -z_y + \text{LSE}(\mathbf{z})$$

No intermediate probability computation—no underflow!

The cross-entropy family includes several related loss functions. Understanding their relationships prevents confusion and errors.

Binary Cross-Entropy (BCE)

For binary classification with $y \in {0, 1}$ and predicted probability $\hat{p} = P(y=1)$:

$$\mathcal{L}_{BCE} = -[y \log \hat{p} + (1-y) \log(1 - \hat{p})]$$

Expanded:

• If $y = 1$: loss = $-\log \hat{p}$ (penalize if $\hat{p}$ is low)
• If $y = 0$: loss = $-\log(1 - \hat{p})$ (penalize if $\hat{p}$ is high)

Categorical Cross-Entropy (CCE)

For multi-class with one-hot $\mathbf{y}$ and predicted distribution $\hat{\mathbf{p}}$:

$$\mathcal{L}{CCE} = -\sum{k=1}^{K} y_k \log \hat{p}_k$$

Special case: For $K=2$ with one-hot encoding:

• $\mathbf{y} = (1, 0)$ or $(0, 1)$
• CCE = $-y_1 \log \hat{p}_1 - y_2 \log \hat{p}_2 = -y_1 \log \hat{p}_1 - (1-y_1) \log(1-\hat{p}_1)$

This is exactly BCE! The two are equivalent for $K=2$.

Sparse Categorical Cross-Entropy

For large $K$, storing one-hot vectors is memory-wasteful. Sparse categorical CE takes integer labels directly:

$$\mathcal{L}_{\text{sparse}} = -\log \hat{p}_y$$

where $y \in {0, 1, \ldots, K-1}$ is the integer label.

Mathematically identical to categorical CE, but computationally efficient:

• No one-hot encoding needed
• $O(1)$ indexing instead of $O(K)$ dot product

Modern frameworks prefer sparse format:

• PyTorch F.cross_entropy expects integer labels
• TensorFlow offers both CategoricalCrossentropy (one-hot) and SparseCategoricalCrossentropy (integers)

Multi-Label BCE

For multi-label (non-mutually exclusive) classification, apply BCE independently to each label:

$$\mathcal{L}{\text{multi-label}} = -\frac{1}{K}\sum{k=1}^{K} [y_k \log \hat{p}_k + (1-y_k) \log(1-\hat{p}_k)]$$

Each $\hat{p}_k$ comes from an independent sigmoid, not softmax: $$\hat{p}_k = \sigma(z_k)$$

Probabilities don't sum to 1—each class is a separate binary decision.

Standard cross-entropy treats all samples and classes equally. In many real-world scenarios, this is suboptimal. Several variants address specific challenges.

Weighted Cross-Entropy

Assign different weights to different classes:

$$\mathcal{L}{\text{weighted}} = -\sum{k=1}^{K} w_k \cdot y_k \log \hat{p}_k$$

For integer labels: $$\mathcal{L}_{\text{weighted}} = -w_y \log \hat{p}_y$$

Use cases:

1. Class imbalance: Up-weight rare classes to balance effective sample size
2. Asymmetric costs: Higher weight for costly misclassifications (e.g., missing a disease)

Weight selection:

• Inverse frequency: $w_k \propto 1/n_k$ where $n_k$ is class count
• Effective number: $w_k \propto (1 - \beta^{n_k})/(1-\beta)$ for some $\beta$
• Manual tuning based on domain costs

Focal Loss

Introduced for object detection (RetinaNet), focal loss down-weights easy examples to focus learning on hard ones:

$$\mathcal{L}_{\text{focal}} = -\alpha_y (1 - \hat{p}_y)^\gamma \log \hat{p}_y$$

where:

• $\gamma \geq 0$ is the focusing parameter (typically 2)
• $\alpha_y$ is the class weight (optional)

How it works:

• Easy examples: $\hat{p}_y \to 1 \Rightarrow (1-\hat{p}_y)^\gamma \to 0$, loss heavily reduced
• Hard examples: $\hat{p}_y \to 0 \Rightarrow (1-\hat{p}_y)^\gamma \to 1$, loss unchanged

Effect: The model doesn't drown in easy examples that flood gradients; it focuses on informative hard cases.

Gradient analysis:

For cross-entropy: $| abla| \propto |\hat{p}_y - 1|$ For focal loss: $| abla| \propto |\hat{p}_y - 1|^{1+\gamma}$

The extra $(1-\hat{p}_y)^\gamma$ factor suppresses gradients from easy examples (where $1 - \hat{p}_y$ is small).

Label Smoothing

Instead of one-hot targets, use 'soft' targets:

$$\tilde{y}_k = \begin{cases} 1 - \epsilon + \frac{\epsilon}{K} & k = y \ \frac{\epsilon}{K} & k eq y \end{cases}$$

Benefits:

• Prevents overconfidence (probabilities don't approach 0/1)
• Acts as regularization
• Improves calibration and generalization in deep networks

Loss formula: $$\mathcal{L}_{\text{smooth}} = (1-\epsilon) \cdot (-\log \hat{p}_y) + \epsilon \cdot H(\text{uniform}, \hat{\mathbf{p}})$$

The second term encourages output entropy, preventing collapse to extreme predictions.

Understanding how gradients flow through softmax + cross-entropy is essential for debugging, implementing custom layers, and theoretical understanding.

Setup

For a single sample with:

• Logits: $\mathbf{z} = (z_1, \ldots, z_K)$
• True class: $c$ (integer label)
• Predictions: $\hat{p}_k = \text{softmax}(\mathbf{z})_k$
• Loss: $\mathcal{L} = -\log \hat{p}_c$

Goal: Compute $\frac{\partial \mathcal{L}}{\partial z_k}$ for all $k$.

Step 1: Expand the loss

$$\mathcal{L} = -\log \hat{p}_c = -\log \frac{e^{z_c}}{\sum_j e^{z_j}} = -z_c + \log\left(\sum_j e^{z_j}\right)$$

Step 2: Differentiate w.r.t. $z_k$

$$\frac{\partial \mathcal{L}}{\partial z_k} = -\mathbf{1}_{k=c} + \frac{\partial}{\partial z_k}\left[\log\left(\sum_j e^{z_j}\right)\right]$$

The second term: $$\frac{\partial}{\partial z_k}\left[\log\left(\sum_j e^{z_j}\right)\right] = \frac{e^{z_k}}{\sum_j e^{z_j}} = \hat{p}_k$$

Step 3: Combine

$$\frac{\partial \mathcal{L}}{\partial z_k} = \hat{p}k - \mathbf{1}{k=c}$$

Using one-hot notation where $y_k = \mathbf{1}_{k=c}$:

$$\boxed{\frac{\partial \mathcal{L}}{\partial z_k} = \hat{p}_k - y_k}$$

Vector form: $$ abla_{\mathbf{z}} \mathcal{L} = \hat{\mathbf{p}} - \mathbf{y}$$

This is the predicted distribution minus the true distribution.

Properties of this gradient:

1. Sums to zero: $\sum_k (\hat{p}_k - y_k) = 1 - 1 = 0$

   Interpretation: The gradient has no component in the 'all equal' direction—it's purely about rebalancing probability mass.

2. Bounded: Each component $|\hat{p}_k - y_k| \leq 1$

3. Interpretable: Positive gradient pushes logit down; negative pushes it up. The true class always gets pushed up (since $\hat{p}_c - 1 < 0$ for $\hat{p}_c < 1$).

Backpropagating to Parameters

For logits $z_k = \mathbf{w}_k^T \mathbf{x} + b_k$:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{w}_k} = \frac{\partial \mathcal{L}}{\partial z_k} \cdot \frac{\partial z_k}{\partial \mathbf{w}_k} = (\hat{p}_k - y_k) \mathbf{x}$$

$$\frac{\partial \mathcal{L}}{\partial b_k} = \hat{p}_k - y_k$$

For a batch of $n$ samples:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{w}k} = \frac{1}{n}\sum{i=1}^{n} (\hat{p}{ik} - y{ik}) \mathbf{x}_i$$

In matrix form with $X \in \mathbb{R}^{n \times d}$ (rows are samples), $P \in \mathbb{R}^{n \times K}$ (predicted), $Y \in \mathbb{R}^{n \times K}$ (one-hot):

$$\frac{\partial \mathcal{L}}{\partial W} = \frac{1}{n}(P - Y)^T X$$

where $W \in \mathbb{R}^{K \times d}$ and the gradient has the same shape.

We have thoroughly explored the cross-entropy loss—the engine that drives classification learning. Let's consolidate the key insights:

What's Next:

With the softmax model and cross-entropy loss established, we now explore an alternative approach to multi-class classification: one-vs-all (OvA) strategies. This decomposition method trains $K$ binary classifiers and offers different tradeoffs compared to multinomial logistic regression.