Why Interpretability

In 2018, Amazon discovered that its experimental hiring algorithm had taught itself to penalize resumes containing the word 'women's'—as in 'women's chess club captain.' The system, trained on a decade of hiring decisions, had learned the historical bias embedded in Amazon's own data. The company couldn't explain exactly why the algorithm made specific decisions, only that its outputs systematically discriminated against women.

This wasn't a bug in the traditional sense. The algorithm was doing exactly what it was designed to do: learn patterns from data. The problem was that no one understood what it had learned until discriminatory outcomes became undeniable.

Welcome to the interpretability problem—one of the most critical challenges in modern machine learning.

Machine learning has undergone a fundamental transformation. Early ML systems—linear regression, decision trees, rule-based systems—were inherently transparent. A data scientist could inspect coefficients, trace decision paths, and explain predictions in human terms.

Modern ML is different. The models that achieve state-of-the-art performance are often the least interpretable:

• Deep neural networks with millions or billions of parameters operating through complex, non-linear transformations
• Ensemble methods combining hundreds of base learners in ways that obscure any single explanation
• Transformer architectures with attention mechanisms that distribute reasoning across thousands of dimensions

This opacity isn't accidental—it's often a feature. Complex models capture subtle patterns that simpler models miss. The power comes from their ability to represent intricate relationships in data. But this power creates a profound tension.

The opacity spectrum:

Not all models are equally opaque. Understanding where different models fall on the interpretability spectrum is crucial:

The critical insight: interpretability and accuracy are often in tension, but this is not an absolute tradeoff. Understanding this relationship—and the techniques to navigate it—is what this module is about.

Interpretability isn't a single property with a single purpose. Different stakeholders need different types of explanations for different reasons. Understanding these diverse needs is essential for choosing appropriate interpretability methods.

Interpretability is no longer optional in many domains. Regulatory frameworks worldwide now mandate explainability for automated decision-making, transforming interpretability from a nice-to-have into a legal requirement.

The regulatory trend is clear:

Regulations are becoming more stringent, not less. The EU AI Act establishes the most comprehensive AI regulation framework to date, and similar legislation is emerging globally. Organizations building ML systems must treat interpretability as a first-class engineering requirement, not an afterthought.

What 'meaningful explanation' means legally:

Regulatory bodies have not precisely defined what constitutes adequate explanation, creating uncertainty. However, emerging consensus suggests explanations must be:

1. Comprehensible to the affected individual, not just technical experts
2. Accurate reflections of the actual decision process
3. Actionable where possible, indicating how outcomes might change
4. Timely provided within reasonable timeframes
5. Substantive addressing the specific factors relevant to that decision

Beyond regulatory compliance and user trust, interpretability serves fundamental scientific purposes. Understanding what a model has learned is essential for advancing machine learning as a discipline.

Discovering spurious correlations:

A landmark study by Ribeiro et al. (2016) found that a highly accurate wolf vs. husky classifier had learned to detect snow rather than wolves. Images of wolves in snow were classified correctly, but only because snow correlated with wolf photos in the training data. The model would fail catastrophically on wolves without snow.

Without interpretability analysis, this model would have been deployed with apparent high accuracy but terrible real-world performance. The interpretation revealed the model had learned a shortcut, not the actual concept.

Similar examples abound:

• Medical imaging models learning hospital equipment artifacts instead of disease markers
• Sentiment classifiers learning author style instead of sentiment
• Object detectors learning photo metadata instead of visual features
• Language models learning dataset annotation patterns instead of language understanding

Enabling scientific discovery:

Interpretability isn't just about catching errors—it's about enabling discovery. When we understand what patterns a model has learned, we sometimes discover genuinely new knowledge:

• Drug discovery: Interpretable models have revealed unexpected molecular structures that serve as leads for new medications
• Materials science: Neural networks analyzing crystal structures have identified novel material properties beyond known physics
• Genomics: Deep learning models have revealed regulatory elements in DNA that were previously unknown to biologists
• Climate science: Interpretable climate models have highlighted interaction effects between variables that refined climate theory

In these cases, the model isn't just a prediction tool—it's a hypothesis generator. But hypothesis generation requires interpretability. A black box that predicts outcomes helps no one understand why those outcomes occur.

For ML engineers, interpretability is a practical necessity. Building, debugging, and maintaining ML systems requires understanding model behavior at a level that aggregate metrics cannot provide.

The most accurate model in the world is useless if no one trusts it enough to act on its predictions. Interpretability is the foundation for trust, and trust is the prerequisite for adoption.

The physician adoption study:

A revealing study of AI-assisted medical diagnosis found that physicians were more likely to follow AI recommendations when explanations were provided—even when the explanations were simplified or imperfect. The mere presence of reasoning, not its completeness, significantly increased trust and adoption.

However, this creates a concerning dynamic: explanations can increase trust beyond what accuracy warrants. Humans are susceptible to plausible-sounding explanations even when those explanations are post-hoc rationalizations rather than true causal accounts. This places ethical responsibility on ML practitioners to provide honest, accurate explanations rather than persuasive but misleading ones.

Appropriate trust calibration:

The goal isn't maximum trust—it's calibrated trust. Users should trust models exactly as much as models deserve:

• Trust should be higher when models are well-validated and operating within their training distribution
• Trust should be lower when inputs are unusual, uncertainty is high, or the domain has known model limitations
• Trust should be absent when models lack validation evidence or are deployed outside their intended scope

Interpretability enables calibrated trust by making model behavior transparent enough for users to adjust their reliance appropriately.

Interpretability requirements vary dramatically across domains. What's sufficient for ad targeting is wholly inadequate for medical diagnosis. Understanding domain context shapes interpretability strategy.