Model evaluation is the process of measuring how well a machine learning model performs. After a model is trained, we need to check whether it can make accurate and useful predictions.
A model is not useful just because it performs well during training. It must also perform well on data it has never seen before.
Evaluation matters because machine learning models are built to make predictions in real-world situations. A model may look good during training but fail when used on new data.
Training Accuracy — 98%
Test Accuracy — 65%
The model performed well on examples it studied, but struggled on new examples.
Evaluation helps identify this problem before the model is deployed.
Training performance measures how well a model performs on the data it learned from. Real-world performance measures how well the model performs on new data.
A student memorizes answers to a practice test.
Practice Test Score — Excellent
New Test Score — Poor
The student memorized instead of understanding. A machine learning model can do the same thing.
Generalization is the model's ability to perform well on new, unseen data. Good generalization means the model learned real patterns rather than memorizing training examples.
Learns common spam patterns and detects new spam emails.
Only remembers spam emails from the training set and misses new spam messages.
Evaluation helps measure whether the model generalizes well.
Accuracy measures the percentage of correct predictions. It is useful, but it can be misleading when different types of errors matter.
Accuracy = Correct Predictions / Total Predictions
A high accuracy score does not always mean the model is useful or safe for the real problem.
Accuracy can be misleading when classes are imbalanced. This happens when one category appears much more often than another.
Normal Transactions — 99%
Fraud Transactions — 1%
Model Predicts — Everything is normal
Accuracy — 99%
Fraud Detected — 0%
The model has high accuracy but is not useful.
In some problems, one type of mistake is more serious than another. Evaluation should match the real-world cost of errors.
False Positive — Model says disease is present, but patient does not have disease.
False Negative — Model says no disease, but patient actually has disease.
A false negative may be more dangerous.
In this situation, recall may be more important than accuracy.
Evaluation metrics are measurements used to judge model performance. Different machine learning problems need different metrics.
| Problem Type | Common Metrics |
|---|---|
| Classification | Accuracy, Precision, Recall, F1 Score, ROC-AUC |
| Regression | MAE, MSE, RMSE, R-squared |
Choosing the right metric depends on what the model is trying to solve.
Classification models predict categories. Evaluation checks whether the model chooses the correct category.
Regression models predict numerical values. Evaluation checks how close the predicted numbers are to the actual values.
Models should be evaluated on data they did not train on. This gives a better estimate of how the model will behave in real-world use.
Training Data — Used to teach the model
Validation Data — Used to tune the model
Test Data — Used to evaluate final performance
The test set should stay separate until final evaluation.
Evaluation helps detect overfitting. Overfitting happens when a model memorizes the training data instead of learning general patterns.
Training Accuracy — 99%
Test Accuracy — 72%
The model performs very well on training data but poorly on new data.
A large gap between training and test performance often suggests overfitting.
Evaluation also helps detect underfitting. Underfitting happens when a model is too simple or has not learned enough.
Training Accuracy — 58%
Test Accuracy — 56%
The model performs poorly on both training and test data.
This suggests the model is missing important patterns.
Evaluation helps compare multiple models and choose the strongest option for the problem.
Logistic Regression — Test Accuracy = 84%
Decision Tree — Test Accuracy = 78%
Random Forest — Test Accuracy = 91%
Possible choice: Random Forest
However, the best model is not always the one with the highest score. Speed, fairness, interpretability, and reliability may also matter.
Before deploying a model, developers need evidence that it performs well. Evaluation helps decide whether the model is ready for real users.
Some applications require stricter evaluation because mistakes can cause real harm.
In these cases, evaluation must consider more than overall accuracy.
A false positive happens when the model predicts positive, but the true answer is negative. A false negative happens when the model predicts negative, but the true answer is positive.
Model says disease is present, but patient does not have disease.
Model says no disease, but patient actually has disease.
Evaluation results are only useful if the evaluation data is reliable. Poor evaluation data can produce misleading results.
Data leakage happens when information from validation or test data accidentally affects training. This can make evaluation results look better than they really are.
Normalize the full dataset before splitting.
Information from test data affects training.
Split data first.
Fit preprocessing on training data only.
Apply the same transformation to validation and test data.
A model may perform well overall but poorly for specific groups. Fair evaluation may require checking performance across groups.
Overall Accuracy — 92%
Group A Accuracy — 96%
Group B Accuracy — 70%
The overall score hides a serious performance gap.
Large language models and generative AI systems need special evaluation methods because their outputs may not have one exact correct answer.
Human evaluation is often used when model outputs require judgment. This is especially common for generative AI.
Human evaluation can capture quality factors that automatic metrics may miss.
A fraud detection model has 99% accuracy because it predicts almost every transaction as normal. However, it misses most fraudulent transactions.
Explain why accuracy alone is not enough in this situation. What other evaluation metric might be more useful, and why?
Ask the AI if you need help understanding model evaluation, accuracy, generalization, overfitting, underfitting, classification metrics, regression metrics, or why evaluation matters.