Classification Metrics

If you ask a non-technical stakeholder how to measure whether a classifier is performing well, their answer will almost certainly involve some form of 'percentage correct.' This intuition leads directly to accuracy, the most natural and universally understood classification metric.

Yet accuracy, despite its simplicity and intuitive appeal, is perhaps the most dangerously misleading metric in the machine learning practitioner's toolkit. Its apparent straightforwardness conceals fundamental limitations that have led countless projects astray, produced models that fail spectacularly in production, and fostered a false sense of confidence in systems that don't actually work.

This page will thoroughly examine accuracy—its definition, its proper use cases, and its critical limitations—so that you can wield it appropriately and recognize when it will mislead you.

Accuracy is defined as the proportion of predictions that are correct:

$$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} = \frac{TP + TN}{TP + TN + FP + FN}$$

In terms of the confusion matrix:

• The numerator is the sum of diagonal entries (correct predictions)
• The denominator is the sum of all entries (total predictions)

For multi-class classification with $K$ classes:

$$\text{Accuracy} = \frac{\sum_{i=1}^{K} C_{ii}}{\sum_{i=1}^{K} \sum_{j=1}^{K} C_{ij}} = \frac{\text{Trace}(C)}{\sum C}$$

where $C_{ij}$ is the confusion matrix entry for true class $i$, predicted class $j$.

Accuracy's widespread use isn't accidental—it possesses several properties that make it cognitively appealing:

1. Interpretability: An accuracy of 95% is immediately understandable: 'The model is correct 95% of the time.' No technical explanation is needed. Stakeholders, executives, and non-technical team members can immediately grasp what this means.

2. Bounded range: Accuracy falls in $[0, 1]$ (or 0-100%), providing a clear sense of scale. Values near 1 are good; values near 0 are bad. The endpoints are meaningful and interpretable.

3. Global view: Accuracy considers all predictions equally, providing a single summary of overall performance without favoring any particular class or outcome.

4. Comparison simplicity: Comparing models by accuracy is trivial: the model with higher accuracy is better (under the assumption that all errors are equally costly).

5. Historical precedent: Accuracy has been used for decades across statistics, machine learning, and related fields, making it a lingua franca for performance reporting.

The accuracy paradox is the phenomenon where a model with higher accuracy can be objectively worse than a model with lower accuracy. This occurs when class distributions are imbalanced.

The Setup:

Consider a fraud detection dataset with 10,000 transactions:

• 9,900 legitimate transactions (99%)
• 100 fraudulent transactions (1%)

The Trivial Baseline:

A 'model' that simply predicts 'legitimate' for every transaction achieves:

$$\text{Accuracy}_{trivial} = \frac{9900}{10000} = 99%$$

This model:

• Catches zero frauds (100% miss rate)
• Is completely useless for its intended purpose
• Yet has 99% accuracy!

The Paradox:

A real fraud detection model that catches 80% of frauds while having a 2% false alarm rate:

• Correctly identifies 80 frauds (TP = 80)
• Misses 20 frauds (FN = 20)
• Correctly clears 9,702 legitimate transactions (TN = 9702)
• Falsely flags 198 legitimate transactions (FP = 198)

$$\text{Accuracy}_{real} = \frac{80 + 9702}{10000} = 97.82%$$

The real model has lower accuracy (97.82% vs 99%) but is infinitely more valuable for its intended purpose.

Let's formalize why accuracy fails under class imbalance. Let $\pi$ be the proportion of positive examples in the dataset ($\pi = P/n$), and $(1-\pi)$ be the proportion of negatives.

Baseline Accuracy:

A trivial classifier that always predicts the majority class achieves:

$$\text{Accuracy}_{baseline} = \max(\pi, 1-\pi)$$

This means:

• If $\pi = 0.5$: Baseline accuracy = 50%
• If $\pi = 0.01$: Baseline accuracy = 99%
• If $\pi = 0.001$: Baseline accuracy = 99.9%

The more imbalanced the dataset, the higher the 'free' baseline accuracy.

Why This Matters:

Two components contribute to accuracy:

$$\text{Accuracy} = \pi \cdot TPR + (1-\pi) \cdot TNR$$

where TPR is the True Positive Rate and TNR is the True Negative Rate.

When $\pi \ll 0.5$ (highly imbalanced toward negatives):

• The weight on TNR is much larger than on TPR
• High accuracy can be achieved with TNR ≈ 1 and TPR ≈ 0
• The metric effectively ignores minority class performance

Beyond class imbalance, accuracy fails when error costs are asymmetric—which is almost always in real-world applications.

The Implicit Assumption:

Accuracy treats all errors equally:

• A False Positive costs the same as a False Negative
• Misclassifying class 1 as class 2 is as bad as misclassifying class 2 as class 1

This assumption is rarely valid in practice.

Example: Medical Screening

Consider a cancer screening test:

• False Negative (missed cancer): Patient doesn't receive treatment, disease progresses, potentially fatal
• False Positive (false alarm): Patient undergoes additional tests, experiences anxiety, incurs costs

While both errors have costs, a False Negative is dramatically more serious. A metric that weights them equally cannot guide model selection appropriately.

Several metrics address accuracy's limitations while retaining some of its simplicity:

1. Balanced Accuracy:

Balanced accuracy averages the recall across classes, giving equal weight to each class regardless of size:

$$\text{Balanced Accuracy} = \frac{1}{K} \sum_{i=1}^{K} \text{Recall}_i = \frac{TPR + TNR}{2} \text{ (for binary)}$$

For binary classification: $\text{Balanced Accuracy} = \frac{1}{2}\left(\frac{TP}{TP+FN} + \frac{TN}{TN+FP}\right)$

This metric:

• Ranges from 0 to 1
• Equals 0.5 for a random classifier (regardless of class balance)
• Gives equal importance to identifying each class correctly

2. Weighted Accuracy:

Assigns explicit weights $w_i$ to each class:

$$\text{Weighted Accuracy} = \frac{\sum_{i=1}^{K} w_i \cdot C_{ii}}{\sum_{i=1}^{K} w_i \cdot \sum_{j=1}^{K} C_{ij}}$$

This allows incorporating domain knowledge about class importance.

3. Geometric Mean of Class Accuracies:

$$G\text{-Mean} = \sqrt{TPR \cdot TNR} = \sqrt{\text{Sensitivity} \cdot \text{Specificity}}$$

This metric is zero if either TPR or TNR is zero, penalizing models that completely fail on any class.

Despite its limitations, accuracy remains the right metric in specific circumstances. Understanding when to use it—rather than blanket avoidance—is the mark of a thoughtful practitioner.

Conditions for Appropriate Use:

Decision Framework:

Should I use accuracy?
├── Are classes balanced (within 40-60%)?
│   ├── No → Use balanced accuracy, F1, or class-specific metrics
│   └── Yes → Continue to next check
├── Are error costs roughly symmetric?
│   ├── No → Use cost-sensitive metrics or threshold optimization
│   └── Yes → Continue to next check
├── Do I care equally about all classes?
│   ├── No → Use class-weighted metrics or macro/micro averages
│   └── Yes → Accuracy is appropriate

For completeness, let's examine accuracy's statistical properties:

1. Accuracy as a Random Variable:

When evaluated on a finite test set of size $n$, accuracy is a point estimate of the true probability of correct classification $p$. If each prediction is independent (often approximately true), the number of correct predictions follows a Binomial distribution:

$$\text{Correct predictions} \sim \text{Binomial}(n, p)$$

2. Confidence Intervals:

A $(1-\alpha)$ confidence interval for accuracy can be constructed using:

$$\hat{p} \pm z_{1-\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

where $\hat{p}$ is the observed accuracy. For small $n$ or extreme $\hat{p}$, use Wilson score intervals or exact binomial intervals.

3. Sample Size Requirements:

To estimate accuracy within margin of error $\epsilon$ with confidence $(1-\alpha)$:

$$n \geq \frac{z_{1-\alpha/2}^2 \cdot p(1-p)}{\epsilon^2}$$

For the worst case ($p = 0.5$) with 95% confidence and $\epsilon = 0.01$:

$$n \geq \frac{1.96^2 \cdot 0.25}{0.01^2} = 9604$$

Accuracy is the most intuitive and widely reported classification metric—and also the most commonly misused. Understanding its proper scope is essential for any ML practitioner.

What's Next:

Recognizing accuracy's limitations motivates the search for better metrics. The next page introduces precision and recall—two complementary metrics that separately address different types of classifier errors and provide a more nuanced view of performance.