Data Bias

Biased Data → Biased Model → Unfair or Unreliable Predictions

Bias can affect real people in hiring, healthcare, lending, and many other high-stakes areas.

• Hiring and recruitment
• Medical diagnosis
• Loan approval
• Criminal justice
• Education
• Facial recognition

Group A — 50,000 images

Group B — 500 images

Problem — The model will perform much better for Group A than Group B

If trained mostly on one accent, the model may struggle with other accents that are underrepresented in training data.

Historical records show Group A received loans more often than Group B.

Result — A model trained on this data may repeat the same unfair pattern

A hiring model trained on past decisions may learn to reject candidates similar to those historically rejected — even if those rejections were originally unfair.

A hospital uses different diagnostic tests for different patient groups.

Problem — Data quality is inconsistent across groups, making the model less reliable for some patients

An essay labeling task where one evaluator consistently gives lower scores to essays written by certain groups.

Problem — The labels reflect the evaluator's bias rather than objective quality

App recommends popular music → Users listen → More popular music data collected → App recommends popular music even more → Less popular genres get less exposure

Platform shows certain jobs to certain demographics → Those demographics apply → System learns those demographics prefer those jobs → Bias reinforces over time

• Who is represented in the dataset?
• Are some groups underrepresented?
• Does the model perform differently across groups?
• Were labels assigned consistently and fairly?
• What historical context produced this data?

Group A: 80%

Group B: 15%

Group C: 5%

If Group C is 5% of the dataset but 20% of real users, the model will underperform for that group.

Group A: 94%

Group B: 89%

Group C: 62%

The large gap for Group C indicates potential bias that should be investigated.

A speech recognition dataset should include:

• Multiple languages
• Multiple accents
• Different ages
• Different genders
• Different background noise conditions

Fraud detection example: Non-fraud 99%, Fraud 1%

• Oversampling — Duplicate minority class examples
• Undersampling — Remove majority class examples
• Synthetic data — Generate new examples for underrepresented groups

Have multiple people label the same data independently and check for agreement before using the labels for training.

Zip code may correlate with race or income. Using zip code as a feature can introduce bias even if race is not explicitly included in the data.

• Does the model perform fairly across all groups?
• Could the model cause harm to vulnerable populations?
• Are there less biased alternatives?
• Who is responsible when a biased prediction causes harm?
• Is the decision process transparent and explainable?

Hiring — Models may deprioritize candidates from certain groups based on historical patterns

Healthcare — Models trained on limited demographics may be less accurate for others

Finance — Loan models may perpetuate past discriminatory lending patterns

Education — Scoring models may produce lower results for certain student groups

Facial Recognition — Some systems have much higher error rates for people with darker skin tones

• Historical data reflects past inequalities that cannot be fully corrected
• Bias can enter at collection, labeling, or preprocessing stages
• Some bias is invisible in aggregate accuracy metrics
• Removing one form of bias can sometimes introduce another
• Human judgment is required, and human judgment itself can be biased

A recommendation model is monitored monthly to check whether some user groups receive less relevant recommendations over time.

If disparities grow, the model may need to be retrained on more balanced data.

• Data bias occurs when training data does not fairly represent the real world
• Types of bias include representation, historical, measurement, label, and feedback loop bias
• Bias can produce unfair predictions that harm real people
• High average accuracy can hide serious bias against specific groups
• Detecting bias requires examining performance at the group level
• Reducing bias involves collecting better data, balancing datasets, reviewing labels, and removing proxy features
• Bias monitoring should continue after deployment