What Algorithmic Bias Is (and Isn't)
Algorithmic bias occurs when an AI system produces systematically different outcomes for different groups — by race, gender, age, disability status, or other protected characteristics — in ways that are harmful or unfair. It's not a software bug (most bias arises from statistically correct learning on biased data) and it's not always intentional. But it can have devastating real-world consequences: loans not approved, jobs not offered, medical conditions not detected, people misidentified by law enforcement.
Sources of Bias
Bias enters AI systems through multiple channels:
- Historical bias: Training data reflects past discrimination. A hiring model trained on historical hiring decisions will perpetuate the historical preference for certain demographics.
- Representation bias: Some groups are underrepresented in training data. Facial recognition trained mostly on lighter-skinned faces performs worse on darker-skinned faces.
- Measurement bias: The labels themselves are biased. "Creditworthy" labels assigned by human loan officers with their own biases produce a biased dataset.
- Feedback loops: Predictions affect the data collected for future training. Predictive policing sends more police to over-policed neighborhoods, generating more arrests there, training the model to continue over-policing them.
Fairness Metrics
Multiple statistical definitions of fairness exist, and they measure different things:
- Demographic parity: Equal positive prediction rates across groups. Controversial because it requires ignoring actual differences in the outcome variable.
- Equalized odds: Equal true positive rates AND equal false positive rates across groups. Ensures that both errors are distributed equally.
- Equal opportunity: Equal true positive rates only. Focus on ensuring that qualified individuals get equal consideration.
- Calibration: Equal accuracy within each group. Ensures risk scores mean the same thing for different groups.
The impossibility result (Chouldechova, 2017; Kleinberg et al., 2017): you cannot simultaneously satisfy demographic parity, equalized odds, and calibration when base rates differ across groups. Any fairness intervention involves choosing which definition to prioritize.
Mitigation Approaches
Pre-processing: modify the training data to reduce bias (resampling, reweighting, fairness-aware representation learning). In-processing: add fairness constraints to the training objective. Post-processing: adjust decision thresholds by group after training. Each approach has tradeoffs between fairness metric satisfaction, overall accuracy, and computational cost.
The Limits
Technical bias mitigation is necessary but not sufficient. No amount of debiasing can compensate for fundamentally flawed problem framing. Predicting who will be a good employee from historical data assumes the historical selection process was legitimate. Using arrest records as a proxy for criminality builds in the biases of policing. Sometimes the answer isn't "debias this model better" but "don't build this model at all." ETH-410 at Meridian AI examines these limits through detailed case studies.