Training performance measures how well a model performs on the data it learned from. Training data is the dataset used to teach the model.
Training Accuracy — 98%
The model correctly predicts 98% of the training examples.
This may sound good, but training performance alone does not tell us whether the model will work well on new data.
Test performance measures how well a model performs on data it has not seen before. Test data is separate from training data.
Test Accuracy — 82%
The model correctly predicts 82% of unseen test examples.
Test performance is usually more important than training performance because it shows how well the model generalizes.
Comparing training and test performance helps reveal whether a model actually learned useful patterns.
How well the model performs on familiar data it learned from.
How well the model performs on new, unseen data.
A model should not only remember the training examples. It should learn patterns that work on new examples too.
A model may perform extremely well on training data but poorly on test data. This usually means the model memorized the training data instead of learning general patterns.
Training Accuracy — 99%
Test Accuracy — 70%
The model looks strong during training, but does not generalize well.
Generalization is the ability of a model to perform well on new, unseen data. A model with good generalization learns patterns that apply beyond the training set.
Learns common spam patterns and detects new spam emails.
Only remembers exact spam emails from training and misses new spam messages.
Overfitting happens when a model learns the training data too well. Instead of learning general patterns, it memorizes specific examples and noise.
Training Accuracy — 99%
Test Accuracy — 72%
Underfitting happens when a model is too simple or has not learned enough from the training data.
Training Accuracy — 58%
Test Accuracy — 56%
A good fit means the model performs well on both training and test data. The scores are strong and close together.
Training Accuracy — 91%
Test Accuracy — 89%
The model learned useful patterns and generalizes well.
The performance gap is the difference between training performance and test performance.
Training Accuracy — 96%
Test Accuracy — 84%
Performance Gap — 12%
A small gap may be acceptable. A large gap may suggest overfitting.
A large gap between training and test performance is a serious warning sign.
Training Accuracy — 99%
Test Accuracy — 65%
Performance Gap — 34%
The model memorized training data.
Training scores are often higher than test scores because the model has already seen the training examples. It has had many chances to adjust itself to that data.
Familiar examples the model has already studied.
New examples — the model cannot rely on memorization.
Accuracy measures the percentage of correct predictions. Training accuracy uses training data, while test accuracy uses unseen data.
Training Accuracy = 95%
Test Accuracy = 92%
Suggests good generalization.
Training Accuracy = 99%
Test Accuracy = 60%
Suggests overfitting.
Loss measures prediction error. Lower loss means better performance.
Training Loss — 0.10
Test Loss — 0.60
The model makes much larger errors on new data.
When training loss is low but test loss is high, the model may be overfitting.
In classification, the model predicts categories such as spam or not spam.
Strong Result — Training 98% / Test 94%
Weak Generalization — Training 99% / Test 68%
The model may have memorized training emails.
In regression, the model predicts numerical values such as house prices.
Training MAE — $8,000
Test MAE — $30,000
The model performs much worse on new houses.
This may suggest overfitting or poor generalization.
Test data must not be used during training. If the model sees test data during development, the test score becomes misleading.
Train model → check test performance → change model → check test performance again → repeat.
The test set starts influencing decisions.
A better workflow keeps training, validation, and test data separate.
Training Data — Teach the model
Validation Data — Tune the model
Test Data — Evaluate final performance
Validation data is often used between training and testing. It helps improve the model without overusing the test set.
| Dataset | Purpose |
|---|---|
| Training Data | Teaches the model |
| Validation Data | Helps tune the model |
| Test Data | Evaluates final performance |
| Training Score | Test Score | Likely Meaning |
|---|---|---|
| 92% | 90% | Good fit |
| 99% | 70% | Overfitting |
| 60% | 58% | Underfitting |
| 75% | 74% | Generalizes, but may be weak |
If test performance is poor, the solution depends on whether the model is overfitting or underfitting.
If training performance is high but test performance is low, the model may be overfitting.
Goal: Reduce memorization.
If both training and test performance are low, the model may be underfitting.
Goal: Help the model learn stronger patterns.
Cross-validation tests model performance across multiple data splits. This gives a more reliable estimate than one split alone.
Fold 1 — Validation set
Folds 2–5 — Training set
Then the validation fold rotates. Each fold takes a turn as validation.
Result: More reliable performance estimate.
A medical model may have high training performance because it memorized patient records. But if test performance is low, it may fail on new patients.
Training Accuracy — 97%
Test Accuracy — 73%
The model may not be safe enough for real-world healthcare use.
Fraud patterns change over time. A model may perform well on old training data but poorly on new fraud cases.
Training Recall — 95%
Test Recall — 62%
The model finds fraud in training data but misses many new fraud cases.
A recommendation model may learn old user behavior very well, but users' interests may change over time.
Training Data — Old user behavior
Test Data — Newer user behavior
The model may recommend content based on outdated patterns.
A model has 99% training accuracy and 71% test accuracy.
Explain what this suggests about the model. Is it more likely overfitting or underfitting? What changes could improve its test performance?
Ask the AI if you need help understanding training performance, test performance, generalization, overfitting, underfitting, validation data, or performance gaps.