The goal of machine learning is not simply to perform well on training data. A successful model should make accurate predictions on new, unseen data.
Learn patterns from training data
Generalize those patterns to new data
Make accurate predictions in the real world
Generalization refers to a model's ability to perform well on data it has never seen before.
Training Data — 1000 sq ft → $200,000, 1500 sq ft → $300,000
New House — 1800 sq ft
A well-generalized model can accurately predict the new house price even though it has never seen that exact house before.
Underfitting occurs when a model is too simple to learn important patterns in the data. The model performs poorly during training and testing because it never learns enough information.
Training Accuracy — Low
Testing Accuracy — Low
Low performance on both datasets is a strong indicator of underfitting.
Overfitting occurs when a model learns the training data too well. Instead of learning general patterns, it memorizes examples and noise found within the training dataset.
Training Accuracy — 99%
Testing Accuracy — 74%
A large gap between training and testing performance is one of the clearest signs of overfitting.
| Model Type | Training Accuracy | Testing Accuracy |
|---|---|---|
| Underfitting | Low | Low |
| Good Fit | High | High |
| Overfitting | Very High | Lower |
Bias measures how much a model oversimplifies a problem. Variance measures how sensitive a model is to training data.
Model is too simple. Underfitting. Missed patterns and poor predictions.
Model is too sensitive. Overfitting. Memorizes training data instead of generalizing.
One of the biggest challenges in machine learning is balancing bias and variance.
Too Much Bias — Model underfits
Too Much Variance — Model overfits
Balanced — Good generalization
Regularization discourages overly complex models and helps reduce overfitting.
Encourages sparse models. Can eliminate unnecessary features entirely.
Reduces large model weights. Creates smoother and simpler models.
Cross-validation evaluates models using multiple training and validation splits.
Fold 1 → Validate, Folds 2-5 → Train
Fold 2 → Validate, Folds 1,3,4,5 → Train
Repeat until all folds are used.
Benefits — Better evaluation, more reliable estimates, helps detect overfitting
Early stopping ends training before the model begins memorizing the training data.
Training continues while validation performance improves
Validation performance stops improving
Training stops and the best model is saved
Underfitting — Disease prediction using only age, ignoring blood pressure and medical history
Overfitting — Fraud model that memorizes specific transactions but fails on new fraud patterns
Good Generalization — Spam filter that learns characteristics of spam rather than memorizing specific emails
| Category | Underfitting | Overfitting |
|---|---|---|
| Model Complexity | Too Simple | Too Complex |
| Bias | High | Low |
| Variance | Low | High |
| Training Accuracy | Low | Very High |
| Testing Accuracy | Low | Lower |
A fraud detection model achieves 99% training accuracy and 74% testing accuracy.
Determine whether the model is underfitting or overfitting. Explain why and suggest techniques that could improve performance.
Ask the AI if you need help understanding overfitting, underfitting, bias, variance, or any evaluation concept.