Bias Sources

Machine learning systems are not neutral arbiters of truth. They inherit, amplify, and sometimes create biases that can perpetuate discrimination, cause harm, and undermine the very goals they were designed to achieve. Understanding where bias originates is the first—and arguably most critical—step toward building fair and equitable ML systems.

Bias in ML is not a bug; it's an inherent characteristic that emerges from how we represent the world in data. Every dataset is a snapshot of a particular time, place, and perspective. Every feature engineering decision encodes assumptions. Every optimization objective privileges certain outcomes. Recognizing these sources of bias transforms machine learning from a black box into a system we can interrogate, critique, and improve.

Before we proceed, let's establish a fundamental principle: bias is not always harmful, and not all harmful effects stem from bias in the technical sense. In statistics, 'bias' refers to systematic deviation from a true value. In machine learning fairness, we're concerned with biases that lead to unjust disparities across protected groups. Throughout this page, we'll navigate both technical and socio-ethical dimensions with precision.

Bias can enter an ML system at virtually any point in its lifecycle. To systematically identify and address it, we need a comprehensive taxonomy. Researchers have proposed various frameworks; here we synthesize the most influential work from Friedman & Nissenbaum (1996), Barocas & Selbst (2016), Suresh & Guttag (2019), and Mehrabi et al. (2021) into a unified hierarchy.

Let's examine each category in depth, with formal definitions, mathematical characterizations where applicable, and illustrative case studies.

Historical bias emerges when the data accurately reflects the world, but the world itself contains inequities we shouldn't perpetuate. This is perhaps the most philosophically challenging form of bias because the data isn't 'wrong' in any technical sense—it accurately captures reality. The problem is that reality itself is biased.

Formal Definition: Let $P^(Y|X)$ denote the ideal, fair relationship between features $X$ and outcomes $Y$ that we would observe in a just world. Historical bias exists when the observed distribution $P(Y|X)$ deviates from $P^(Y|X)$ due to historical discrimination or systemic inequity:

$$\text{Historical Bias} = D_{KL}(P(Y|X) | P^*(Y|X)) > 0$$

where $D_{KL}$ is the Kullback-Leibler divergence.

Addressing Historical Bias:

Historical bias requires more than technical fixes—it demands engagement with normative questions about what a 'fair' distribution would look like. Approaches include:

1. Counterfactual Data Augmentation: Generate synthetic examples representing how the data might look without historical inequity
2. Re-weighting: Adjust sample weights to counteract historical imbalances
3. Causal Modeling: Explicitly model and remove effects of protected attributes on outcomes
4. Policy Integration: Combine ML predictions with affirmative policies that address historical disadvantage

No purely technical solution resolves historical bias because it's fundamentally a socio-political problem encoded in data.

Representation bias occurs when the training data doesn't adequately represent the population where the model will be deployed. Unlike historical bias, where the data accurately reflects a biased world, representation bias involves systematic sampling errors that create blind spots.

Formal Definition: Let $P_{train}(X, Y)$ denote the training distribution and $P_{deploy}(X, Y)$ the deployment population. Representation bias exists when:

$$P_{train}(X, Y) eq P_{deploy}(X, Y)$$

More specifically, for protected groups $G = {g_1, g_2, ..., g_k}$, representation bias manifests when:

$$P_{train}(G = g_i) eq P_{deploy}(G = g_i) \text{ for some } i$$

This is a form of selection bias or sampling bias that leads to covariate shift during deployment.

Quantifying Representation Bias:

We can measure representation bias using various metrics:

1. Demographic Parity in Data: Compare group proportions in training data to target population $$\text{DPD} = \sum_{g} |P_{train}(G = g) - P_{target}(G = g)|$$

2. Effective Sample Size: For each group, compute the effective sample size after accounting for weighting: $$n_{eff,g} = \frac{(\sum_i w_i \mathbb{1}[G_i = g])^2}{\sum_i w_i^2 \mathbb{1}[G_i = g]}$$

3. Coverage Metrics: What fraction of the input space is 'covered' by training examples?

Measurement bias arises when the features or labels we can measure systematically differ from the constructs we actually want to capture. Every ML problem involves mapping abstract concepts (creditworthiness, job performance, health risk) to concrete, measurable proxies (credit scores, performance reviews, diagnostic codes). When this mapping is imperfect—and it always is—bias can enter.

Formal Definition: Let $Y^*$ be the true outcome of interest and $Y$ be the measured proxy. Measurement bias exists when:

$$\mathbb{E}[Y | Y^, G = g_1] eq \mathbb{E}[Y | Y^, G = g_2]$$

That is, the relationship between the proxy and the true outcome differs across groups.

Types of Measurement Bias:

1. Label Bias (Outcome Measurement Error) The target variable is measured with systematic error. For example:

• Arrests as a proxy for criminality (over-policing affects minority communities)
• Health costs as a proxy for health needs (access barriers affect who can spend)
• Teacher evaluations as a proxy for instructional quality (student prejudice affects ratings)

2. Feature Measurement Bias (Input Measurement Error) Input features are measured differently across groups:

• Self-reported income may be less accurate for gig workers
• Credit history may be incomplete for recent immigrants
• Medical records may miss conditions in populations with healthcare access barriers

3. Proxy Discrimination Seemingly neutral features encode protected information:

• Zip code encodes race due to residential segregation
• Name encodes gender and ethnicity
• School attended encodes socioeconomic status

Mathematical Example: Differential Measurement Error

Consider predicting job success using interview scores. Let $Y^*$ be true job performance and $Y$ be the interview score:

For Group A (majority): $Y = Y^* + \epsilon_A$, where $\epsilon_A \sim N(0, \sigma^2)$

For Group B (minority): $Y = Y^* + \epsilon_B - \delta$, where $\epsilon_B \sim N(0, \sigma^2)$ and $\delta > 0$

The bias term $\delta$ represents systematic underrating of Group B. A model trained on these labels will learn to undervalue Group B candidates, even if the features are unbiased.

Aggregation bias occurs when a single model is used across groups that have fundamentally different data generating processes. The assumption that one set of features and one model architecture can serve everyone equally is often false, especially when subpopulations have distinct causal relationships between inputs and outcomes.

Formal Definition: Let $f^*_g(X)$ be the optimal predictor for group $g$. Aggregation bias exists when:

$$f^_{g_1}(X) eq f^_{g_2}(X)$$

but we train a single model $\hat{f}(X)$ that cannot adapt to group-specific patterns.

The error introduced by aggregation is: $$\text{Aggregation Error} = \sum_g P(G=g) \cdot \mathbb{E}[(\hat{f}(X) - f^*_g(X))^2 | G=g]$$

Case Study: Medical Diagnosis

Consider diagnosing diabetes risk. The same features (blood glucose levels, BMI, age) have different predictive relationships across:

• Ethnic groups: Type 2 diabetes thresholds vary by ethnicity; a universal threshold over-diagnoses some groups and under-diagnoses others
• Age groups: Diabetes presentation differs in young vs. elderly patients
• Geographic regions: Environmental factors (diet, activity levels) modify feature-outcome relationships

A single global model makes systematic errors for groups whose patterns differ from the population average.

Illustrative Example with Linear Models:

Suppose the true model for Group A is: $Y_A = 2X_1 + X_2 + \epsilon_A$

And for Group B: $Y_B = X_1 + 3X_2 + \epsilon_B$

If we fit a single model with equal group representation, we might get: $\hat{Y} = 1.5X_1 + 2X_2$

This compromises on both groups, performing optimally for neither.

Learning bias refers to how machine learning algorithms can amplify, distort, or create disparities beyond what exists in the training data. Even with unbiased data, the learning process itself can introduce or exacerbate unfairness through its inductive biases, optimization dynamics, and architectural choices.

Key Insight: ML algorithms are not passive learners; they actively construct representations and decision boundaries that may not align with fairness objectives.

Mathematical Analysis: Bias Amplification

Consider a binary classification task where Group A has $n_A$ samples and Group B has $n_B$ samples, with $n_A >> n_B$. Using empirical risk minimization:

$$\hat{f} = \arg\min_f \frac{1}{n} \sum_{i=1}^{n} L(f(x_i), y_i)$$

This is equivalent to: $$\hat{f} = \arg\min_f \left[ \frac{n_A}{n} \cdot \text{EmpRisk}_A(f) + \frac{n_B}{n} \cdot \text{EmpRisk}_B(f) \right]$$

Since $\frac{n_A}{n} >> \frac{n_B}{n}$, the optimizer prioritizes Group A performance. This isn't just about statistical power—it fundamentally shapes the solution.

Empirical Observation: Research has shown that models can exhibit bias amplification—disparities in model predictions that exceed disparities in training labels. For example, a model trained to predict 'cooking' in images may associate cooking with women more strongly than the training data itself does.

Deep Learning Specific Issues:

1. Embedding Space Geometry: Word embeddings trained on web data encode stereotypes (e.g., 'man is to computer programmer as woman is to homemaker'). These biases propagate to downstream tasks.

2. Attention Mechanism Bias: Transformers may systematically attend differently to content associated with different groups, even when instructed to be neutral.

3. Gradient Starvation: In multi-class settings, gradients from rare classes may be overwhelmed by frequent classes, preventing learning for minority categories.

4. Memorization vs. Generalization: Models may memorize patterns for majority groups while poorly generalizing for minorities due to different effective learning dynamics.

The final stages of the ML pipeline—evaluation and deployment—introduce their own bias sources that are often overlooked during development but critically impact real-world performance.

Evaluation Bias occurs when the benchmark used to assess model performance doesn't represent the deployment population. This creates a false sense of confidence; a model that excels on evaluation metrics may fail for real users whose characteristics differ from the test set.

Deployment Bias (sometimes called application bias) arises when systems are used in contexts or populations beyond their intended scope, or when the population changes after deployment (concept drift, population shift).

Detection and Mitigation:

For Evaluation Bias:

• Disaggregated Evaluation: Report metrics separately for demographic subgroups, not just overall
• Worst-Group Performance: Optimize for minimum subgroup performance, not average
• Diverse Benchmark Construction: Ensure test sets represent deployment populations through active curation
• Community-Based Evaluation: Involve affected communities in identifying failure modes

For Deployment Bias:

• Monitoring and Alerting: Track performance metrics continuously after deployment
• Population Drift Detection: Statistical tests for distribution shift (e.g., MMD, KL divergence)
• Scope Constraints: Technical and organizational controls limiting deployment to validated populations
• Continuous Retraining: Update models as population characteristics change
• Human-in-the-Loop: Maintain human oversight for high-stakes decisions