Data preprocessing is the process of transforming raw data into a format that a machine learning model can understand and learn from effectively.
Raw Data — Collected from various sources
Preprocessing — Transform into clean numeric form
Model Training — Model learns from preprocessed data
Predictions — Model makes predictions on new data
Machines need numbers. Text, dates, and categories must be converted before most models can use them.
Date text — "2026-06-09" → timestamp number
Yes/No — "Yes" → 1, "No" → 0
Categories — "Red", "Blue", "Green" → 0, 1, 2
Normalization scales values to a fixed range, usually between 0 and 1. This ensures no single feature dominates simply because of its scale.
50 → 0.00
100 → 0.33
150 → 0.66
200 → 1.00
Formula: (value - min) / (max - min)
Feature 1 — Age, range 0–100
Feature 2 — Income, range 20,000–200,000
Without normalization the model may overweight income simply because the numbers are larger.
(25 - 0) / (100 - 0) = 0.25
Standardization transforms data so the mean is 0 and the standard deviation is 1. It does not require knowing the min and max values.
Score — 90
Class mean — 80
Standard deviation — 10
Standardized — (90 - 80) / 10 = 1.0
A value of 1.0 means this score is 1 standard deviation above the class average.
| Normalization | Standardization |
|---|---|
| Scales to 0–1 range | Centers around mean 0 |
| Requires min and max | Uses mean and standard deviation |
| Best for non-normal distributions | Best for roughly normal distributions |
| Good for KNN, Neural Networks | Good for Logistic Regression, SVM |
| Affected by outliers | More robust to outliers |
Machine learning models cannot directly use text labels. Categories must be converted to numbers.
Red = 0, Blue = 1, Green = 2
Simple but may imply ordering between categories.
Best for — Ordinal categories (Small, Medium, Large)
Red → [1, 0, 0]
Blue → [0, 1, 0]
Green → [0, 0, 1]
Best for — Non-ordinal categories (Color, Country)
Tokenization splits text into smaller units called tokens. This is an essential step in natural language processing.
Sentence — "The cat sat on the mat"
Tokens — ["The", "cat", "sat", "on", "the", "mat"]
Convert text to lowercase
Remove punctuation and special characters
Tokenize into individual words
Remove stop words (the, a, is, and)
Apply stemming or lemmatization
Original — "The cat sat on the mat"
After removing stop words — ["cat", "sat", "mat"]
Cuts words to a base form (may not be a real word)
running → run
studies → studi
Faster but less accurate
Converts to the proper dictionary form
running → run
studies → study
Slower but more accurate
Images must be preprocessed before most models can use them. Images come in different sizes, and models need a consistent input size.
Original image — 1920 × 1080 pixels
Resized to — 224 × 224 × 3 (height × width × color channels)
Pixel values range from 0 to 255
Example — 128 / 255 = 0.50
Normalizing to 0–1 helps training stability and speed
Training (70%) — 7,000 examples — used to teach the model
Validation (15%) — 1,500 examples — used to tune the model
Test (15%) — 1,500 examples — used to evaluate final performance
Calculate normalization values using the entire dataset including test data.
Problem — Test data has influenced training setup (data leakage)
Calculate normalization values using only the training set. Apply those same values to transform validation and test data.
Handle missing values
Encode categorical features
Normalize or standardize numerical features
Tokenize text (if applicable)
Split data into train, validation, and test sets
If you normalized age between 18 and 80 during training, you must apply the same formula (same min and max values) when a new user's age is entered at prediction time.
Inconsistent preprocessing can cause silent failures in production.
Ask the AI if you need help understanding normalization, standardization, encoding, tokenization, image preprocessing, or how to avoid data leakage during preprocessing.