What Causes AI Bias? The Real Sources Explained

Every biased AI system has an origin story. A hiring tool that rejected women did not wake up sexist — someone fed it ten years of hiring decisions made by sexist managers. A facial recognition system that failed on dark skin was not designed with malice — it was trained on photos of light-skinned people because those were the photos available. Bias in AI is rarely a villain twirling a mustache. It is usually a series of small, reasonable-looking decisions that compound into something ugly.

For the full picture of what bias looks like once it is in a system, see my guide to AI bias.

The Short Answer

AI bias is caused by the gap between the world as it is and the world as the training process assumes it to be. Machine learning does one thing extremely well: it finds patterns in data and reproduces them. If the data contains prejudice — underrepresentation, stereotyped labels, historical discrimination — the model learns it faithfully. If the design choices encode narrow assumptions — who the user is, what “good” looks like, which errors matter — the model enforces them at scale.

The rest of this article is the long answer: eight distinct causes, each with its own mechanism and its own famous failure.

Cause 1: Unrepresentative Training Data

The most common cause, and the simplest to understand: the model never saw enough of some group to learn them properly.

The textbook case is facial recognition. In 2018, researchers showed that commercial gender-classification systems from major tech companies had error rates under 1% for light-skinned men and over 30% for dark-skinned women. The cause was not complicated: the training datasets were overwhelmingly composed of light-skinned faces. The models were not prejudiced; they were ignorant. They had simply never learned what dark-skinned women look like, because nobody showed them.

This pattern repeats everywhere. Speech recognition trained mostly on male voices struggles with female voices. Medical AI trained on data from wealthy hospitals underperforms on patients from poor ones. Language models trained overwhelmingly on English text from the internet reflect the demographics of who writes online — young, male, Western, educated.

Cause 2: Biased Labels

Sometimes the data is representative but the labels — the “correct answers” the model learns from — carry human prejudice.

Amazon’s famous recruiting tool is the canonical example. The company trained a hiring model on ten years of resumes submitted to Amazon, labeled by whether the candidate was hired. The historical hiring data reflected a male-dominated tech culture, so the model learned to penalize resumes containing the word “women’s” — as in “women’s chess club captain” — and to downgrade graduates of women’s colleges. Amazon killed the project. The model had done exactly what it was asked: replicate past hiring decisions. Past hiring decisions were the problem.

Label bias is insidious because it is invisible in the data pipeline. The dataset looks clean — real resumes, real outcomes. The prejudice is baked into the outcomes themselves. Any model trained to predict human decisions inherits the biases of the humans who made them: loan officers, judges, doctors, hiring managers. The more “objective” the training process looks, the easier it is to miss that the ground truth was never ground truth at all.

Cause 3: Historical Bias Baked Into Records

Related to label bias but broader: the historical record itself is distorted. Even when individual decisions were made fairly, the data reflects centuries of unequal treatment.

Consider a model predicting creditworthiness from financial history. On paper, the features are neutral: payment history, debt levels, account age. But financial history is downstream of historical discrimination — redlining denied mortgages to Black neighborhoods, which suppressed homeownership, which suppressed wealth accumulation, which shows up in every financial variable a model might use. The model is not biased against anyone; it is accurately measuring the financial consequences of bias that happened decades ago, and then perpetuating them.

This is the hardest cause to address because there is no clean dataset to switch to. The distortion is in reality, not just in the records. Fixing it requires a judgment call about what the data should look like — a counterfactual — and reasonable people disagree about which counterfactual is right. Should a lending model treat applicants as if historical discrimination never happened? Most fairness advocates say yes. But implementing that “as if” requires explicit intervention, which is itself controversial.

Cause 4: Proxy Variables

Remove race from the dataset and the model finds it anyway. This is the proxy problem, and it defeats the most common naive fix for bias.

Proxies are features that correlate with protected attributes: zip code predicts race (thanks to residential segregation), first name predicts gender and ethnicity, purchase history predicts both, gaps in employment history predict motherhood, the college someone attended predicts class and race. In high-dimensional data, there are always proxies — dozens of weak signals that combine into a strong one.

The COMPAS recidivism tool controversy illustrates the dynamic. The tool’s makers said it did not use race. Investigative analysis showed it flagged Black defendants as high-risk at far higher rates than white defendants. Whether or not race was an explicit input, the model’s 137 questionnaire items — criminal history, neighborhood, family background, employment — carried racial signal through correlated variables. The debate about COMPAS raged for years precisely because both sides could be right: no explicit racial input, deeply racially disparate output.

Cause 5: Design and Objective Choices

Bias is not always in the data. Sometimes it is in the decisions engineers make before training starts.

The objective function. A model optimizes exactly what you tell it to optimize, and nothing else. Optimize a hiring model for “predicted tenure” and it will favor candidates who resemble long-tenured employees — who, in many companies, are disproportionately male and white. The bias was not in the data; it was in the choice of target. Every objective function is a value judgment wearing a math costume.

The evaluation metrics. Teams optimize what they measure. If the success metric is overall accuracy, the model will sacrifice performance on small groups to gain accuracy on large ones — because that is what maximizes the number. Group-level performance is not measured, so it is not optimized, so it degrades. The metric caused the bias as surely as any dataset.

Default assumptions about users. Voice assistants that struggle with non-American accents, health apps designed around male physiology, facial recognition that fails on darker skin — many of these trace back to a development team that implicitly designed for people like themselves. Homogeneous teams build systems that work for homogeneous users. This is not malice; it is the most predictable failure mode in product development.

Cause 6: Measurement and Sampling Bias

Even well-intentioned data collection introduces bias through how things are measured and who gets sampled.

Measurement bias. The instruments themselves can be biased. The infamous case: pulse oximeters, which overestimate blood oxygen in patients with dark skin — a hardware measurement bias that, when fed into clinical AI systems, produced systematically wrong assessments for Black patients. The AI was faithfully processing biased measurements. No amount of algorithmic fairness fixes a sensor that lies.

Cause 7: Feedback Loops

Some of the worst bias is not in the initial model at all — it emerges after deployment, as the model’s outputs reshape the world it learns from.

Predictive policing is the clearest example. The model predicts crime hotspots from historical arrest data. Police are deployed to those hotspots. More policing means more arrests in those areas — which become new training data confirming the original prediction. The loop is self-reinforcing: the model does not discover crime, it discovers policing, and then it manufactures the evidence for more of it. Neighborhoods that were over-policed get more over-policed, forever, with mathematical justification.

Hiring tools do the same thing more quietly. A model that slightly favors one group leads to more hires from that group, which becomes the new training data, which strengthens the preference. Each retraining cycle amplifies the bias a little more. Without intervention, feedback loops converge toward monoculture: the model eventually only “likes” one kind of candidate.

Breaking feedback loops requires deliberate design: random exploration (showing the model things outside its comfort zone), human review of edge cases, and periodic retraining on data collected outside the model’s influence. Most deployed systems do none of this.

Cause 8: Deployment and Interaction Bias

The final cause lives outside the model entirely — in how humans use it.

Automation bias. Humans trust machine outputs too much, especially under time pressure. A hiring manager who would carefully weigh a human recruiter’s recommendation will rubber-stamp an algorithm’s score. The model’s biases — small, measurable, manageable — get amplified into decisions by humans who stop thinking. The bias that matters is not in the model; it is in the deference.

Interface bias. How the model’s output is presented shapes how it is used. A risk score shown as a stark red “HIGH RISK” triggers different behavior than the same score shown as “62nd percentile.” Ranked lists, default options, the prominence of caveats — all of these are design choices that modulate bias in practice.

How the Causes Combine

In real systems, these causes never appear alone. They stack.

Take the Amazon recruiting tool: historical bias in tech hiring (Cause 3) produced biased labels (Cause 2) in an unrepresentative dataset (Cause 1), the objective function rewarded replicating past decisions (Cause 5), and had it deployed, feedback loops (Cause 7) would have compounded the skew with every hiring cycle. Five causes, one system.

Or facial recognition bias: unrepresentative training data (Cause 1) met measurement realities of cameras optimized for lighter skin (Cause 6), deployed by agencies with their own patterns of whom they surveil (Cause 8). The technology, the data, and the deployment each contributed.

Frequently Asked Questions