In many real-world classification problems, class imbalance isn't just a minor annoyance—it's a fundamental obstacle that standard loss functions fail to overcome. Consider object detection: in a single image, there might be 3 objects of interest and 100,000 background regions. Traditional cross-entropy treats each sample equally, causing the overwhelming majority of easy negatives to dominate the loss and gradient.

The result? Models that predict "background" everywhere, achieving low loss by ignoring the rare but important foreground objects.

Focal Loss, introduced by Lin et al. in 2017 for the RetinaNet detector, elegantly solves this problem. Rather than just reweighting classes (which helps but doesn't fully solve the problem), focal loss fundamentally changes how cross-entropy behaves: it down-weights easy examples so that the model focuses its learning capacity on hard examples.

This page will take you through focal loss from motivation to implementation, exploring why it works, when to use it, and how to tune its hyperparameters.

The Scale of the Problem

Object Detection Example:

• A typical image: ~100,000 anchor boxes evaluated
• Foreground objects: 3-10
• Background anchors: 99,990+
• Imbalance ratio: 10,000:1 or worse

Medical Diagnosis:

• Disease prevalence: 0.1-1%
• Healthy samples: 99-99.9%
• Imbalance ratio: 100:1 to 1000:1

Fraud Detection:

• Fraudulent transactions: 0.01%
• Normal transactions: 99.99%
• Imbalance ratio: 10,000:1

Why Standard Cross-Entropy Fails

Let's analyze what happens with standard cross-entropy under extreme imbalance.

The Loss Breakdown: For binary classification with cross-entropy: $$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N}[y_i \log p_i + (1-y_i) \log(1-p_i)]$$

With 99% negatives and 1% positives:

• 99% of samples contribute to the "predict negative correctly" signal
• 1% contribute to the "predict positive correctly" signal
• Model learns: predicting negative is almost always right!

The Gradient Problem: Even worse, the easy negatives contribute non-trivial gradients. A model predicting $p=0.1$ for an easy negative has:

• Cross-entropy: $-\log(0.9) \approx 0.105$
• Gradient: $0.1$ (pushing toward lower prediction)

With 99,000 such samples, these small per-sample gradients sum to a massive signal that overwhelms the 100 hard positives.

Traditional Solutions and Their Limits

1. Class Reweighting: $$\mathcal{L} = -\frac{1}{N}\sum[w_1 \cdot y_i \log p_i + w_0 \cdot (1-y_i) \log(1-p_i)]$$

with weights inversely proportional to class frequency.

Problem: Helps with the frequency imbalance but doesn't address that most samples are easy. Easy negatives (p=0.01 for background) still contribute, just with lower weight.

2. Oversampling / Undersampling:

• Oversample minority class or undersample majority class

Problem: Oversampling can cause overfitting to minority class; undersampling discards potentially useful data.

3. Hard Example Mining:

• Only use the K hardest examples for each batch
• Common in object detection (OHEM)

Problem: Introduces additional complexity, hyperparameters, and can miss useful information from medium-difficulty examples.

The key insight missed: The problem isn't just class frequency—it's that easy examples dominate the gradient, regardless of their frequency.

Core Idea: Down-Weight Easy Examples

Focal loss modifies cross-entropy to reduce the contribution from easy, well-classified examples:

$$\mathcal{L}_{FL} = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$

where:

• $p_t$ is the model's predicted probability for the true class:
  • $p_t = p$ if $y = 1$
  • $p_t = 1 - p$ if $y = 0$
• $\gamma \geq 0$ is the focusing parameter
• $\alpha_t$ is the class balancing weight (optional)

Unpacking the Formula

The new term: $(1 - p_t)^\gamma$

This is the modulating factor. Let's see how it behaves:

• When $p_t$ is high (correct and confident): $(1 - p_t)^\gamma \to 0$

  • Example: $p_t = 0.9$, $\gamma = 2$ → $(0.1)^2 = 0.01$ (loss reduced by 100×!)

• When $p_t$ is low (wrong or uncertain): $(1 - p_t)^\gamma \to 1$

  • Example: $p_t = 0.1$, $\gamma = 2$ → $(0.9)^2 = 0.81$ (loss nearly unchanged)

Interpretation: The modulating factor acts as a soft attention mechanism that focuses learning on hard examples.

The Focusing Parameter $\gamma$

$\gamma$ controls how much to down-weight easy examples:

• $\gamma = 0$: Focal loss = cross-entropy (no modulation)
• $\gamma = 1$: Moderate focus on hard examples
• $\gamma = 2$: Strong focus (standard recommendation)
• $\gamma = 5$: Very aggressive focus (may ignore some useful signal)

Deriving the Focal Loss Gradient

For thorough understanding, let's derive the gradient of focal loss with respect to the logit $z$ (where $p = \sigma(z)$).

Setup: $\mathcal{L}_{FL} = -(1-p_t)^\gamma \log(p_t)$

For $y = 1$ (positive class): $p_t = p = \sigma(z)$

$$\frac{\partial \mathcal{L}_{FL}}{\partial z} = -(1-p)^\gamma \cdot \frac{1}{p} \cdot p(1-p) + \gamma(1-p)^{\gamma-1} \cdot (-1) \cdot \log(p) \cdot p(1-p)$$

Simplifying: $$\frac{\partial \mathcal{L}_{FL}}{\partial z} = (1-p)^\gamma (p - 1) + \gamma(1-p)^{\gamma} p \log(p)$$

Further simplification (after algebra): $$\frac{\partial \mathcal{L}_{FL}}{\partial z} = (1-p)^\gamma \left[\gamma p \log(p) + p - 1\right]$$

Gradient Comparison: CE vs Focal

Cross-Entropy Gradient (for y=1): $p - 1$

Focal Loss Gradient (for y=1): $(1-p)^\gamma \cdot [\gamma p \log(p) + p - 1]$

The modulating factor $(1-p)^\gamma$ appears in the gradient, directly suppressing gradients from easy examples.

Numerical Analysis

Aggregate Effect on Training

Consider a batch with:

• 1 hard positive (p=0.1)
• 99 easy negatives (p=0.05, i.e., correctly predicting low probability)

Cross-Entropy Gradients:

• Hard positive: $|p - 1| = 0.9$
• Easy negatives: $|p - 0| = 0.05$ each, sum = 4.95
• Total from negatives: 5.5× harder positives!

Focal Loss Gradients (γ=2):

• Hard positive: ~0.9 × modulating factor ≈ 0.73
• Easy negatives: ~0.05 × 0.05² ≈ 0.000125 each, sum ≈ 0.012
• Total from negatives: 60× smaller than hard positive!

The reversal is dramatic: Cross-entropy lets easy negatives dominate; focal loss makes hard positives dominate.

Effective Sample Weighting

Focal loss can be viewed as an adaptive weighting scheme:

$$w_i = (1 - p_{t,i})^\gamma$$

Easy examples get weight near 0; hard examples get weight near 1. Unlike fixed class weights, this is dynamic—it adapts based on current model confidence, effectively doing implicit curriculum learning where easy examples are removed from training as they become easy.

Combining γ and α

Focal loss can include an optional class balancing weight $\alpha$:

$$\mathcal{L}_{FL} = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$

where:

• $\alpha_t = \alpha$ for positive class (foreground)
• $\alpha_t = 1 - \alpha$ for negative class (background)

The Role of α

While $\gamma$ addresses the easy/hard imbalance, $\alpha$ addresses the frequency imbalance:

• Even with $\gamma > 0$, if true positives are 1% of data, they contribute less aggregate gradient
• $\alpha > 0.5$ up-weights positives to compensate for frequency

Common settings:

• RetinaNet paper: $\alpha = 0.25$, $\gamma = 2$
• Wait—why $\alpha < 0.5$ for rare positives?

The Counter-Intuitive Alpha

With $\gamma = 2$, easy negatives are already heavily down-weighted. Setting $\alpha > 0.5$ would further reduce their weight, potentially making positives too dominant.

The optimal $\alpha$ depends on $\gamma$:

• $\gamma = 0$ (standard CE): use $\alpha \propto$ inverse frequency
• $\gamma = 2$: use $\alpha = 0.25$ (original paper)
• Higher $\gamma$: may need even lower $\alpha$

The paper's finding: $\alpha = 0.25$ works well with $\gamma = 2$. This slightly down-weights positives because the $(1-p_t)^2$ term so dramatically up-weights hard examples that additional raw weighting would over-correct.

Extending to K Classes

The focal loss naturally extends to multi-class classification:

$$\mathcal{L}{FL-multi} = -\sum{k=1}^{K} \alpha_k (1 - p_k)^\gamma y_k \log(p_k)$$

where:

• $y_k \in {0, 1}$ is the one-hot label for class $k$
• $p_k$ is the softmax probability for class $k$
• $\alpha_k$ is the weight for class $k$

Simplified form (for one-hot labels where only true class $c$ has $y_c = 1$):

$$\mathcal{L}_{FL-multi} = -\alpha_c (1 - p_c)^\gamma \log(p_c)$$

Per-Class Alpha Weights

For multi-class, $\alpha$ becomes a vector $\boldsymbol{\alpha} = [\alpha_1, \ldots, \alpha_K]$:

Inversely proportional to frequency: $$\alpha_k = \frac{N}{K \cdot n_k}$$ where $n_k$ is the number of samples in class $k$.

Effective number of samples (Cui et al., 2019): $$\alpha_k \propto \frac{1 - \beta^{n_k}}{1 - \beta}$$ where $\beta \in [0, 1)$ is a hyperparameter. This provides smoother weights than inverse frequency.

Learned weights: Treat $\boldsymbol{\alpha}$ as trainable parameters or tune on validation set.

Object Detection: The Origin Story

Focal loss was invented to save the one-stage detector paradigm.

Problem: Two-stage detectors (R-CNN family) use a region proposal network to filter easy negatives before classification. One-stage detectors (YOLO, SSD) classify all anchors directly, suffering from extreme foreground-background imbalance.

Solution: RetinaNet with focal loss achieved accuracy comparable to two-stage detectors with one-stage speed.

Key insight: The problem wasn't the one-stage architecture; it was using standard cross-entropy.

Medical Image Analysis

Medical imaging is rife with class imbalance:

• Lesion detection: tiny lesions in large volumes
• Disease screening: rare positive cases
• Cell segmentation: few abnormal cells among millions

Focal loss benefits:

• Focuses on hard-to-detect lesions
• Reduces false negative rate (critical in medicine)
• Requires less data augmentation for rare classes

NLP Applications

Named Entity Recognition (NER):

• Most tokens are 'O' (no entity): 90%+
• Rare entity types: possibly <1%
• Focal loss improves recall on rare entities

Sentiment Analysis with Nuanced Classes:

• Strong negative/positive: common
• Neutral or mixed: rarer, harder
• Focal loss focuses on ambiguous cases

Intent Classification:

• Common intents: well-represented
• Long-tail intents: few training examples
• Focal loss combined with class-balanced sampling

Instance Segmentation

In Mask R-CNN and similar models:

• Foreground/background pixel imbalance within each mask
• Small objects have fewer foreground pixels
• Focal loss for mask branch improves small object segmentation

Practical Recommendations

1. Start with γ=2, α=0.25 (the original settings)
2. Tune γ first: If still dominated by easy examples, increase γ
3. Tune α second: Adjust based on validation performance per class
4. Monitor per-class metrics: Not just overall accuracy
5. Consider label smoothing: Can combine with focal loss for regularization

The Log-Sum-Exp Challenge

Focal loss involves $\log(p_t)$ where $p_t$ can be very close to 0 or 1. Combined with the modulating factor, numerical issues can arise.

Stable Binary Focal Loss Implementation:

Using the identity for binary cross-entropy from logits: $$-\log(\sigma(z)) = \log(1 + e^{-z}) = \text{softplus}(-z)$$ $$-\log(1 - \sigma(z)) = \log(1 + e^z) = \text{softplus}(z)$$

We can write stable focal loss without explicitly computing probabilities.

Framework Implementations

PyTorch (manual implementation—not built-in as of 2024):

import torch
import torch.nn.functional as F

def focal_loss(inputs, targets, gamma=2.0, alpha=0.25):
    bce = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')
    p = torch.sigmoid(inputs)
    p_t = p * targets + (1 - p) * (1 - targets)
    modulating = (1 - p_t) ** gamma
    alpha_t = alpha * targets + (1 - alpha) * (1 - targets)
    return (alpha_t * modulating * bce).mean()

TensorFlow:

import tensorflow as tf

loss_fn = tf.keras.losses.BinaryFocalCrossentropy(
    gamma=2.0, alpha=0.25, from_logits=True
)

Detectron2 (Facebook's detection library):

from fvcore.nn import sigmoid_focal_loss_jit

Focal Loss with Label Smoothing

Label smoothing can be combined with focal loss: $$y_{smooth} = (1 - \epsilon) \cdot y + \epsilon / K$$

The focal loss then uses smoothed targets, which:

• Reduces overconfidence
• Provides gradients even for "easy" examples (since targets are never 0 or 1)
• Requires careful tuning as it interacts with the focusing mechanism

We've explored focal loss from its motivation in object detection to practical implementation. Here are the essential takeaways:

Looking Forward

We've now covered the major loss functions: cross-entropy for general classification, MSE for regression, hinge for margin-based learning, and focal for imbalanced data. The final topic brings these together:

• Custom Losses: Combining loss functions, designing domain-specific objectives, and encoding prior knowledge directly into the optimization target.