Data cleaning is the process of identifying and fixing problems in a dataset before using it to train a machine learning model.
If these problems are not fixed, the model may learn incorrect patterns.
Clean Data → Better Learning → More Reliable Predictions
Messy Data → Confused Learning → Incorrect Predictions
A house price dataset with incorrect prices and missing locations will produce unreliable predictions.
Duplicate records are rows that appear more than once in the dataset. They can cause the model to overfit to repeated examples.
Customer ID 101 appears 3 times with the same name, email, and purchase history.
Problem — The model overweights this customer's patterns
Same record entered twice by mistake. Should be removed.
Two different people with the same name at the same address. Should be kept.
Missing values occur when some data points are incomplete. They can cause errors during training if not handled properly.
Name: Alice
Age: (missing)
City: Toronto
Problem — Missing age prevents the model from using that feature for this record
If location is missing for many records, the model cannot learn how location affects price.
Remove the row — Simple, but reduces dataset size
Fill with mean — Good when there are no major outliers
Fill with median — Better when outliers are present
Fill with mode — Most common value, good for categories
Predict the value — Use another model to estimate the missing value
Mean — $52,000 (affected by high earners)
Median — $48,000 (not affected by extremes)
If one person earns $2,000,000, the mean becomes much higher than typical.
Recommendation — Use median when the data has significant outliers
Problem — Color column has 40 rows with no value
Option 1 — Replace with most common color
Option 2 — Create a new "Unknown" category
Creating an "Unknown" category is often better because it preserves the fact that color was missing.
Inconsistent formatting occurs when the same type of information is stored in different ways.
Fix — Standardize all values to "United States"
Capitalization — spam, Spam, SPAM → spam
Extra spaces — "Canada " → "Canada"
Different units — 5 feet and 152 cm in same column
Different date formats — 2026-06-09 and 06/09/2026
Not every outlier is a mistake. Some extreme values are genuine and should be kept.
Age listed as 999
This is impossible and should be removed or corrected.
A CEO's salary that is genuinely much higher than others
This is real data and may be important to keep.
Prices: $200,000 / $220,000 / $215,000 / $3,000,000
The $3,000,000 house may be a luxury home (keep) or a data entry error (remove).
Always investigate before deciding
An image of a cat is labeled as "dog."
Problem — This teaches the model incorrect information
Label problems can happen from rushed labeling, unclear guidelines, or human disagreement.
If you remove all records where income is missing, and lower-income people had more missing data, the remaining dataset will be biased toward higher-income examples.
Always check whether data removal creates imbalance
Explore the data — Check for obvious problems and understand the dataset
Find duplicates — Identify and remove exact duplicate records
Handle missing values — Fill, estimate, or remove missing data
Fix incorrect values — Correct or remove values that are clearly wrong
Standardize formatting — Apply consistent formats across the dataset
Handle outliers — Investigate and decide whether to keep or remove
Fix incorrect labels — Review and correct mislabeled examples
Validate — Re-check the dataset before training begins
| Data Cleaning | Data Preprocessing |
|---|---|
| Fixes problems in the data | Transforms data for model use |
| Removes duplicates | Scales numerical features |
| Handles missing values | Encodes categorical features |
| Corrects incorrect values | Tokenizes text |
| Standardizes formats | Resizes images |
| Fixes labels | Splits into train/val/test |
Pandas (Python) — Remove duplicates, fill values, filter rows
NumPy — Detect and handle outliers
OpenRefine — Visual data cleaning for non-programmers
Excel / Google Sheets — For small datasets
Calculate the mean of the entire dataset including test data, then use it to fill missing values.
Test data was used to inform training — this is data leakage.
Calculate the mean using only the training data. Apply that same value to fill missing data in both training and test sets.
Ask the AI if you need help understanding data cleaning steps, handling missing values, fixing inconsistent formats, or identifying outliers.