Use Accuracy — Classes are balanced, mistakes have similar costs
Use Precision — False positives are costly
Use Recall — False negatives are costly
Use F1 Score — Both false positives and false negatives matter
Use ROC-AUC — You want to evaluate class separation across thresholds
Example: Spam Detection
Metrics for Spam Detection
Accuracy — Overall percentage of emails classified correctly
Precision — How many emails sent to spam were actually spam
Recall — How many actual spam emails were caught
F1 Score — Balance between catching spam and avoiding false alarms
ROC-AUC — How well the model separates spam from not spam
Example: Disease Detection
Metrics for Disease Detection
Accuracy — Overall percentage of correct diagnoses
Precision — How many predicted disease cases actually had disease
Recall — How many actual disease cases were found
F1 Score — Balance between finding disease and avoiding false alarms
ROC-AUC — How well the model separates disease from no disease
In healthcare, recall may be especially important because missing actual cases can be harmful.
Example: Fraud Detection
Metrics for Fraud Detection
Accuracy — Overall percentage of transactions classified correctly
Precision — How many flagged transactions were actually fraud
Recall — How many fraud transactions were caught
F1 Score — Balance between catching fraud and avoiding false alerts
ROC-AUC — How well the model separates fraud from normal transactions
Because fraud is rare, accuracy alone is usually not enough.
Imbalanced Classification
Imbalanced classification happens when one class appears much more often than another.
Example
Normal: 95%
Fraud: 5%
Problem: A model can get high accuracy by mostly predicting the common class.
Better metrics: Precision, Recall, F1 Score, ROC-AUC
Training Metrics vs Test Metrics
Classification metrics can be calculated on training data or test data. Test metrics are usually more important because they estimate real-world performance.
Training Metrics
Performance on data the model learned from
Test Metrics
Performance on unseen data — better estimate of real-world performance
Overfitting Detection
Overfitting happens when the model performs very well on training data but poorly on test data.
Example
Training F1 Score: 0.96
Test F1 Score: 0.62
Problem: The model memorized training examples instead of learning general patterns.
Underfitting Detection
Underfitting happens when the model performs poorly on both training and test data.
Example
Training Accuracy: 58%
Test Accuracy: 56%
Training F1 Score: 0.40
Test F1 Score: 0.39
Problem: The model is not learning enough useful patterns.
Multi-Class Classification Metrics
Multi-class classification has more than two possible classes. Accuracy, precision, recall, and F1 score can still be used.
Multi-Class Examples
Cat, Dog, Bird
Positive, Neutral, Negative
Low, Medium, High
A, B, C, D, F
For precision, recall, and F1: scores may be calculated per class and then averaged.
Macro, Micro, and Weighted Averages
Average Type
Meaning
Macro Average
Treats each class equally
Micro Average
Combines all predictions before calculating
Weighted Average
Gives larger classes more influence
Common Mistakes
Common Mistakes
Using accuracy alone
Ignoring class imbalance
Confusing precision and recall
Ignoring false positives
Ignoring false negatives
Choosing the wrong metric for the problem
Looking only at training metrics
Ignoring test metrics
Assuming high ROC-AUC means the model is perfect
Ignoring performance across groups
Summary
Key Takeaways
Classification metrics evaluate models that predict categories.
Accuracy measures the percentage of correct predictions.
Precision measures how many predicted positives were actually positive.
Recall measures how many actual positives were found.
F1 score balances precision and recall.
ROC-AUC measures how well the model separates positive and negative classes.
A confusion matrix helps calculate many classification metrics.
Accuracy alone can be misleading for imbalanced datasets.
The best metric depends on the real-world cost of mistakes.
Practice Prompt
A fraud detection model has high accuracy but low recall. Explain why this can happen and why accuracy alone is not enough. Then choose which metric you would focus on and explain why.
Need Help?
Ask the AI if you need help understanding accuracy, precision, recall, F1 score, ROC-AUC, confusion matrices, class imbalance, or how to choose the right classification metric.