Test data is a portion of the dataset that is held out and used only at the very end to evaluate the final performance of a trained model. It represents examples the model has never seen during training or development.
Training Data — Used to teach the model
Validation Data — Used during development to tune and select models
Test Data — Used once at the very end for final evaluation
Test data gives an unbiased estimate of how the model will perform on new data in the real world.
Flashcards — Training data
Practice tests during studying — Validation data
The actual exam — Test data (taken only once at the end)
Test email — "Limited time offer — click here now!"
Model prediction — Spam
True label — Spam
After testing 10,000 emails: 9,600 correct → Test Accuracy = 96%
Collect all available data
Split into training (70%), validation (15%), and test (15%) sets
Train the model on training data only
Evaluate on validation data and tune the model
Select the best model version
Evaluate the final model on test data once
Report the test performance as the real-world estimate
Training: 7,000 (70%) — Largest share; model needs many examples to learn
Validation: 1,500 (15%) — Used to tune the model during development
Test: 1,500 (15%) — Held until the very final evaluation
Use the test set to tune the model, then evaluate on the same test set.
Problem — The model was indirectly fitted to the test data. The evaluation is no longer honest.
Use validation data for all tuning decisions.
Use test data only once at the final evaluation.
Training: 92%
Validation: 90%
Test: 89%
Small differences — the model learned real patterns
Training: 99%
Validation: 76%
Test: 72%
Large gap — the model memorized training data
Training accuracy: 99% | Test accuracy: 70%
The model performs well only on data it has already seen — it memorized instead of learning.
Training accuracy: 58% | Test accuracy: 56%
The model performs poorly even on training data — it has not learned enough.
| Data Split | When Used | Purpose | How Often Used |
|---|---|---|---|
| Training | During training | Teach the model | Every epoch |
| Validation | During development | Tune and select models | Many times |
| Test | After training | Final evaluation | Once only |
| Problem Type | Common Metrics |
|---|---|
| Classification | Accuracy, Precision, Recall, F1 Score |
| Regression | MAE, MSE, RMSE, R² |
| Object Detection | mAP, IoU |
| NLP | Accuracy, BLEU, Perplexity |
Accuracy: 92% | Precision: 88% | Recall: 95% | F1 Score: 91%
MAE: $20,000 — On average, predictions are off by $20,000
Trained and tested only on sunny highway roads.
Real-world conditions also include rain, snow, city streets, and night driving.
Problem — The model passes the test but may fail in real deployment
One or two errors can change accuracy by 10–20%.
Results are unreliable and may not reflect real performance.
Each error only changes accuracy by 0.01%.
Results are more stable and reliable.
Apply normalization using statistics from the entire dataset before splitting.
Test data statistics influenced training — leakage.
Split the data first. Calculate preprocessing statistics using only training data. Apply those same statistics to the test set.
A fraud detection model trained in 2020 performs well on its 2020 test set.
By 2023, new fraud tactics have emerged that the model has never seen.
Result — Test accuracy from 2020 no longer reflects current performance
Models should be monitored in production and retrained when drift is detected.
Training: 7,000 | Validation: 1,500 | Test: 1,500
Test accuracy: 93% — correctly classifies 93% of emails it has never seen
Training: 8,000 | Validation: 1,000 | Test: 1,000
Results must be reported separately per patient group to check for fairness and bias
Training: 7,000 | Test: 1,500
Test MAE: $25,000 — on average, predictions are $25,000 off on unseen houses
Ask the AI if you need help understanding test data, data splitting, how to measure generalization, evaluation metrics, or the problem of data leakage in test sets.