Cross-validation is a model evaluation technique that tests a model across multiple data splits. Instead of using only one train-test split, the model is trained and evaluated multiple times.
1
Split data multiple ways
2
Train and test multiple times
3
Average the results
4
Get a more reliable performance estimate
Cross-validation helps answer how well a model performs across different parts of the data.
Why Cross-Validation Matters
A single train-test split can sometimes give a misleading result. One split may be easier or harder than another.
Easy Test Split
Model looks too strong — inflated score.
Hard Test Split
Model looks too weak — deflated score.
Cross-validation tests across multiple splits for a more reliable estimate.
Single Train-Test Split
A basic evaluation approach is to split the data once into training and test sets.
Example Split
Training Data: 80%
Test Data: 20%
Problem: The result depends heavily on how the data was split. An unlucky split may not represent the model well.
Problem with One Split
One split may not show the full picture.
Example
Dataset: 1,000 examples
Training: 800 examples
Test: 200 examples
If the test set has mostly easy examples, the model may look too strong. If it has mostly hard examples, the model may look too weak.
How Cross-Validation Works
Cross-validation splits the data into multiple groups called folds. The model trains on some folds and validates on another fold.
1
Split data into folds
2
Train on some folds, validate on one fold
3
Repeat until each fold was used for validation
4
Average the scores
What Is a Fold?
A fold is one part of the dataset used in cross-validation. Each fold gets a turn as the validation set.
5-Fold Cross-Validation
Dataset: 1,000 examples
Fold 1 = about 200 examples
Fold 2 = about 200 examples
Fold 3 = about 200 examples
Fold 4 = about 200 examples
Fold 5 = about 200 examples
K-Fold Cross-Validation
K-fold cross-validation divides the dataset into K equal or nearly equal parts. K is the number of folds.
Common Values
5-fold cross-validation
10-fold cross-validation
In 5-fold cross-validation, the model trains and validates 5 times. Each time, a different fold is used for validation.
5-Fold Cross-Validation Example
Rounds
Round 1 — Validate on Fold 1, Train on Folds 2, 3, 4, 5
Round 2 — Validate on Fold 2, Train on Folds 1, 3, 4, 5
Round 3 — Validate on Fold 3, Train on Folds 1, 2, 4, 5
Round 4 — Validate on Fold 4, Train on Folds 1, 2, 3, 5
Round 5 — Validate on Fold 5, Train on Folds 1, 2, 3, 4
The final score is the average of the 5 validation scores.
Cross-Validation Score
Each fold produces a score. These scores are averaged to estimate model performance.
Example Fold Scores
Fold 1 Accuracy: 86%
Fold 2 Accuracy: 88%
Fold 3 Accuracy: 84%
Fold 4 Accuracy: 87%
Fold 5 Accuracy: 85%
Average Accuracy: 86%
This average is usually more reliable than one split alone.
Why Average Scores?
Averaging scores reduces the effect of random data splits. If one fold is unusually easy or hard, the average helps smooth out the result.
Example
One fold score: 70%
Other scores: 85%, 86%, 87%, 88%
The low fold may reveal that the model struggles on certain examples. The average shows overall performance.
Standard Deviation in Cross-Validation
Cross-validation often reports both the average score and the variation across folds. Variation shows how consistent the model is.
Model A
Average accuracy = 88%
Scores are close together
More reliable and stable
Model B
Average accuracy = 88%
Scores vary widely
Less stable across splits
High Variation Across Folds
High variation means the model performs very differently depending on the data split.
Example
Fold Scores: 95%, 92%, 60%, 89%, 58%
Possible causes: dataset is small, model is overfitting, data is not representative, some folds contain harder examples, or classes are unevenly distributed.
Low Variation Across Folds
Low variation means the model performs similarly across different folds. Stable performance is usually a good sign.
Example
Fold Scores: 86%, 87%, 85%, 88%, 86%
The model is more stable across splits.
Cross-Validation and Overfitting
Cross-validation can help detect overfitting when the model performs much better on training folds than validation folds.
Overfitting Sign
Training Accuracy: 99%
Cross-Validation Accuracy: 68%
The model performs well on familiar data but poorly on held-out folds.
Cross-Validation and Underfitting
Cross-validation can also help detect underfitting when the model performs poorly across all folds.
Underfitting Sign
Fold Scores: 55%, 57%, 54%, 56%, 55%
Possible causes: model is too simple, features are weak, training is insufficient, or data quality is poor.
Cross-Validation vs Test Set
Cross-validation is usually used during model development. A final test set should still be kept separate when possible.
Evaluation Method
Purpose
Cross-Validation
Helps tune and compare models
Test Set
Final unbiased evaluation
Cross-Validation vs Validation Set
A validation set is one fixed part of the data used for tuning. Cross-validation uses multiple validation sets and is often more reliable.
Validation Set
One split — simple but can be misleading if the split is unlucky.
Cross-Validation
Multiple splits — more reliable, especially for small datasets.
When to Use Cross-Validation
Use Cross-Validation When
The dataset is small or medium-sized
You want a more reliable performance estimate
You want to compare models
You want to tune hyperparameters
You want to detect unstable performance
A single split may be misleading
When Cross-Validation May Be Expensive
Cross-validation requires training the model multiple times, so it can be expensive for large models.
Cost
5-Fold Cross-Validation: Train 5 models
10-Fold Cross-Validation: Train 10 models
This can be slow for large neural networks or very large datasets. For deep learning, teams often use a fixed validation set instead.
Stratified Cross-Validation
Stratified cross-validation keeps class proportions similar in each fold. This is useful for classification with imbalanced classes.
Example
Dataset: 90% not fraud, 10% fraud
Each fold should also have: ~90% not fraud, ~10% fraud
Goal: Keep folds balanced and representative.
Why Stratification Matters
Without stratification, some folds may not represent the full dataset.
Example
Fraud Dataset: Only 20 fraud cases
Bad Fold: No fraud cases in validation
Problem: The model cannot be properly evaluated for fraud on that fold.
Stratified cross-validation helps keep each fold more balanced.
Leave-One-Out Cross-Validation
Leave-One-Out Cross-Validation (LOOCV) uses one example as validation and all other examples as training.
LOOCV
Dataset: 100 examples
Train 100 times, each time leaving out 1 example
Use case: Very small datasets
Downside: Can be very slow
Repeated Cross-Validation
Repeated cross-validation runs cross-validation multiple times with different random splits for an even more reliable estimate.
Example
5-fold cross-validation repeated 3 times
Total model trainings: 15
Benefit: More reliable estimate
Tradeoff: Takes more time
Cross-Validation for Classification
For classification tasks, cross-validation can evaluate classification metrics across multiple folds.
Classification Metrics
Accuracy
Precision
Recall
F1 score
ROC-AUC
Example: A fraud detection model can be tested across multiple folds to see whether it consistently detects fraud.
Cross-Validation for Regression
For regression tasks, cross-validation can evaluate numerical prediction errors across multiple folds.
Regression Metrics
MAE
MSE
RMSE
R-squared
Example: A house price model can be tested across multiple folds to see whether its errors are consistent.
Cross-Validation and Hyperparameter Tuning
Cross-validation is often used to tune hyperparameters, which are model settings chosen before training.
Hyperparameter Examples
Tree depth
Number of neighbors
Learning rate
Regularization strength
Number of estimators
Goal: Choose settings that work well across multiple data splits, not just one validation set.
Example: Choosing Tree Depth
Decision Tree Tuning
Tree Depth 3: Average CV Accuracy = 82%
Tree Depth 8: Average CV Accuracy = 88%
Tree Depth 20: Average CV Accuracy = 75%
Best Choice: Tree Depth 8
Tree depth 8 performs best across folds in this example.
Cross-Validation and Data Leakage
Data leakage can happen if preprocessing is done incorrectly before cross-validation.
Bad Approach
Normalize the full dataset first, then run cross-validation.
Problem: Information from validation folds leaks into training folds.
Better Approach
Fit preprocessing only on the training folds inside each round.
Benefit: Keeps evaluation fair.
Cross-Validation Pipeline
A safe cross-validation pipeline includes preprocessing inside each fold to avoid data leakage.
1
Split training folds and validation fold
2
Fit preprocessing on training folds only
3
Transform training folds
4
Transform validation fold using training preprocessing
5
Train model, evaluate on validation fold, store score
Example: Spam Detection
Spam Detection
Problem: Classify emails as spam or not spam
Why cross-validation: A single split may place many easy spam examples in the test set
Metrics: Accuracy, Precision, Recall, F1 score
Result: Average score gives a better estimate of model performance
Example: House Price Prediction
House Price Prediction
Problem: Predict house prices
Why cross-validation: A single split may place expensive houses mostly in training or mostly in validation
Metrics: MAE, RMSE, R-squared
Result: Shows whether price prediction errors are consistent across splits
Example: Medical Prediction
Medical Prediction
Problem: Predict whether a patient has a condition
Why cross-validation: Medical datasets may be small; one split may not include enough positive cases
Better approach: Use stratified cross-validation
Metrics: Recall, Precision, F1 score, ROC-AUC
Common Mistakes
Common Mistakes
Using only one train-test split when data is limited
Forgetting to shuffle data before splitting
Not using stratification for imbalanced classification
Preprocessing the full dataset before cross-validation
Using the test set for hyperparameter tuning
Looking only at the average score and ignoring variation
Comparing models using different splits
Using random cross-validation when the data has time order
Cross-Validation with Time Series Data
Time series data has a natural order, so regular random cross-validation can create leakage.
Time Series Examples
Stock prices
Weather readings
Monthly sales
Website traffic over time
Better approach: Train on past data, validate on future data. This matches real-world prediction.
Limitations of Cross-Validation
Cross-validation is useful, but it does not guarantee perfect real-world performance.
Limitations
Takes more computation
Can be slow for large models
Must avoid data leakage
May not work well with time-dependent data
Still depends on data quality
Does not replace a final test set when one is available
Summary
Key Takeaways
Cross-validation tests model performance across multiple data splits.
It gives a more reliable estimate than one train-test split.
K-fold cross-validation divides data into K folds.
Each fold gets a turn as the validation set.
The final score is usually the average across folds.
Variation across folds shows model stability.
Stratified cross-validation helps with imbalanced classification.
Cross-validation is useful for model comparison and hyperparameter tuning.
Preprocessing must be done inside each fold to avoid data leakage.
A separate test set should still be used for final evaluation when possible.
Practice Prompt
A dataset has 1,000 examples and you use 5-fold cross-validation. Explain how many examples are in each fold, how many rounds of training happen, and why the average score is more reliable than one train-test split.
Need Help?
Ask the AI if you need help understanding cross-validation, folds, K-fold cross-validation, stratified cross-validation, data leakage, model comparison, or hyperparameter tuning.