4 / 13

Validation Data

What Is Validation Data?

Validation data is a portion of the dataset used during model development to evaluate and improve the model — without touching the final test set.

Three Data Roles

Training Data — Used to teach the model

Validation Data — Used to evaluate and improve during development

Test Data — Used only once at the very end

Why Validation Data Is Needed

A model's performance on training data can be misleading. A model that memorizes training data may score very highly on training examples but fail on new data.

Overfitting Example

Training Accuracy: 98%

Validation Accuracy: 72%

Problem — The model memorized training examples rather than learning generalizable patterns

Validation data shows how the model performs on data it has not seen during training.

The Machine Learning Workflow

Collect and prepare data

Split into training (70%), validation (15%), and test (15%)

Train the model on training data

Evaluate on validation data

Adjust the model based on validation results

Repeat Steps 3–5 until satisfied

Evaluate the final model once on the test set

Data Split

10,000 Examples Total

Training: 7,000 (70%) — Largest share for learning

Validation: 1,500 (15%) — Used repeatedly during development

Test: 1,500 (15%) — Never seen until the final step

Model Selection with Validation

Comparing Three Models

Model A — Validation Accuracy: 82%

Model B — Validation Accuracy: 89%

Model C — Validation Accuracy: 85%

Selected — Model B (highest validation accuracy)

Hyperparameter Tuning

Hyperparameter Examples

Learning rate
Number of layers in a neural network
Maximum depth of a decision tree
Number of trees in a random forest
Batch size
Dropout rate

Hyperparameters are not learned from data — they must be set manually. Validation data helps choose the best combination.

Detecting Overfitting and Underfitting

Overfitting

Training: 99%

Validation: 70%

Large gap — model memorized training examples but cannot generalize

Underfitting

Training: 58%

Validation: 56%

Both low — model has not learned enough patterns

Early Stopping

Early stopping stops training when validation performance stops improving. This prevents overfitting.

Early Stopping Example

Epoch 1 — Validation loss: 0.50

Epoch 5 — Validation loss: 0.30

Epoch 10 — Validation loss: 0.25 ← Best

Epoch 15 — Validation loss: 0.38 ← Getting worse

Training should stop at Epoch 10 and save that model version

Model Selection

Three Models on Validation Set

Logistic Regression — 82%

Decision Tree — 85%

Random Forest — 91%

Selected — Random Forest for final test evaluation

The Danger of Reusing the Test Set

Bad Approach

Evaluate on test → Adjust → Evaluate on test again → Repeat many times

The model is now indirectly fitted to the test set.

Test accuracy is an overestimate of real performance.

Better Approach

All tuning decisions use validation data.

The test set is evaluated only once at the very final step.

Training Data vs Validation Data

Training Data	Validation Data
Used to teach the model	Used to evaluate the model
Model learns from it directly	Model does not learn from it directly
Used every epoch	Used after each epoch to track progress
High accuracy is expected	Lower accuracy signals overfitting

5-Fold Cross-Validation

Cross-validation creates multiple train/validation splits from the same dataset. Useful when the dataset is small.

5-Fold Cross-Validation

Split 1: Train on Folds 2–5, Validate on Fold 1

Split 2: Train on Folds 1, 3–5, Validate on Fold 2

Split 3: Train on Folds 1, 2, 4, 5, Validate on Fold 3

Split 4: Train on Folds 1–3, 5, Validate on Fold 4

Split 5: Train on Folds 1–4, Validate on Fold 5

Final score — Average across all five folds

Validation Metrics

Problem Type	Common Validation Metrics
Classification	Accuracy, Precision, Recall, F1 Score
Regression	MAE, MSE, RMSE, R²
Object Detection	mAP, IoU
NLP	Accuracy, BLEU, Perplexity

Real-World Examples

Spam Detection

Training: 7,000 | Validation: 1,500

Logistic Regression F1: 0.82 | Random Forest F1: 0.91 | SVM F1: 0.88

Selected — Random Forest based on highest F1 score on validation set

House Price Prediction

Linear Regression MAE: $32,000 | Decision Tree MAE: $25,000 | Random Forest MAE: $18,000

Selected — Random Forest with lowest validation MAE

Common Mistakes

Mistakes to Avoid

Using the test set during tuning instead of validation data
Not using a validation set at all
Calculating preprocessing statistics using the full dataset before splitting
Making the validation set too small to give stable results

Data Leakage in Validation

Bad Approach

Scale all data using stats from the full dataset, then split into training/validation/test.

Problem — Validation and test data influenced the scaling (leakage)

Better Approach

Split first. Calculate scaling stats from training data only. Apply those stats to validation and test data.

Good Validation Data

Characteristics

Diverse — Covers the full range of examples
Representative — Reflects the real distribution
Large enough for stable, meaningful results
Properly held out — Never used during training

Key Takeaways

Summary

Validation data is used during development to evaluate and improve the model
It sits between training and test data — used many times, but never teaches the model directly
Overfitting is detected when training accuracy is high but validation accuracy is much lower
Validation data guides hyperparameter tuning and model selection
The test set should remain untouched until the final evaluation
Cross-validation is useful for small datasets to get more stable estimates

Need Help?

Ask the AI if you need help understanding validation data, hyperparameter tuning, early stopping, cross-validation, overfitting, or data leakage during model development.