A loss function measures how wrong a model's prediction is. It compares the model's prediction to the correct answer and produces a loss value.
Prediction is made
Compare with correct answer
Calculate error
Loss value produced — high if very wrong, low if close to correct
Loss functions matter because models need a way to measure mistakes. A model cannot improve unless it knows how wrong its predictions are.
Prediction: Dog
Correct Answer: Cat
Loss: High
Prediction: Cat
Correct Answer: Cat
Loss: Low
The model uses loss to guide learning.
During training, the model repeatedly makes predictions, calculates loss, and updates its weights.
Receive input data
Make a prediction
Compare prediction to the correct answer
Calculate loss
Update weights to reduce loss
Repeat with many examples
The goal is to reduce loss over time.
Loss is related to error, but it is not always exactly the same thing. Error is the difference between prediction and correct answer. Loss is a mathematical way to measure that error.
Actual Value — 100 Predicted Value — 90
Error — 100 − 90 = 10
Loss — Depends on the loss function used
Some use absolute error, some use squared error, some use probabilities.
Low loss means the model's predictions are close to the correct answers.
Actual Value: 100
Prediction: 98
Loss: Low
Correct Class: Cat
Prediction: Cat with high confidence
Loss: Low
High loss means the model's predictions are far from the correct answers.
Actual Value: 100
Prediction: 40
Loss: High
Correct Class: Cat
Prediction: Dog with high confidence
Loss: High
High loss tells the model that it needs to improve.
A loss function is used during training to guide learning. An evaluation metric is used to judge model performance.
| Concept | Main Purpose | Example |
|---|---|---|
| Loss Function | Helps train the model | Cross-entropy loss |
| Evaluation Metric | Helps humans judge performance | Accuracy |
In neural networks, loss is used to update weights. Forward propagation creates a prediction, and the loss function measures the mistake.
Forward propagation produces prediction
Loss function measures the mistake
Backpropagation uses loss to calculate how weights should change
Weight updates are applied
Without loss, the model would not know how to improve.
Backpropagation uses the loss value to calculate gradients. Gradients show how each weight affects the loss.
Loss — Tells the model how wrong it is
Backpropagation — Tells the model how weights should adjust
Gradient descent — Makes the weight updates
Gradient descent is an optimization method that tries to reduce loss by updating weights step by step.
Start with high loss
Adjust weights using gradients
Lower the loss
Repeat — if training works, loss decreases over time
A loss curve shows how loss changes during training.
Epoch 1 — Loss = 1.20
Epoch 5 — Loss = 0.80
Epoch 10 — Loss = 0.45
Epoch 20 — Loss = 0.25
Decreasing loss suggests the model is learning.
Training loss measures loss on the training data, which is the data the model learns from.
Training Loss — 0.90 → 0.30
The model is making fewer errors on training examples.
Validation loss measures loss on validation data. It helps check whether the model is generalizing beyond the training data.
Training Loss — 0.20 Validation Loss — 0.75
The large gap may suggest the model is overfitting.
Test loss measures loss on unseen test data. It is used after training to estimate final performance.
Test Loss — 0.35
This gives an estimate of how well the model performs on new data.
Important — Test loss should not be used repeatedly for tuning decisions.
Overfitting can be detected by comparing training loss and validation loss.
Training Loss — Very low Validation Loss — High or increasing
Example — Epoch 30: Training Loss = 0.05, Validation Loss = 0.80
The model may be memorizing training data.
Underfitting can happen when both training loss and validation loss are high.
Training Loss — 1.10 Validation Loss — 1.15
The model performs poorly on both training data and new data.
The model may be too simple or not trained enough.
Regression models predict numbers. Regression loss functions measure how far numerical predictions are from actual values.
Common Regression Loss Functions — Mean Absolute Error, Mean Squared Error, Huber Loss
Mean Absolute Error, or MAE, measures the average absolute difference between actual and predicted values.
Formula — MAE = Average of |Actual Value − Predicted Value|
Actual values — 100, 200, 300
Predictions — 110, 190, 280
Absolute errors — 10, 10, 20
MAE — (10 + 10 + 20) / 3 = 13.33
MAE is easy to understand because it uses the same units as the target.
Mean Squared Error, or MSE, measures the average squared difference between actual and predicted values.
Formula — MSE = Average of (Actual Value − Predicted Value)²
Actual values — 100, 200, 300 Predictions — 110, 190, 280
Errors — −10, 10, 20 Squared errors — 100, 100, 400
MSE — (100 + 100 + 400) / 3 = 200
MSE punishes large errors more strongly than MAE.
MSE squares errors, which makes large mistakes much more important.
Error: 2
Squared error: 2² = 4
Small penalty.
Error: 20
Squared error: 20² = 400
Much larger penalty.
This is useful when big mistakes should be strongly discouraged.
Huber Loss combines ideas from MAE and MSE. It acts like MSE for small errors and like MAE for large errors.
Small errors — Treated more like MSE
Large errors — Treated more like MAE
Useful when — The data has outliers, but large errors should still matter
Classification models predict categories. Classification loss functions measure how wrong the model's class probabilities are.
Common Loss Functions — Binary cross-entropy, Categorical cross-entropy, Sparse categorical cross-entropy
Cross-entropy loss is commonly used for classification. It measures how well predicted probabilities match the correct class.
Correct Class: Cat
Cat = 0.90, Dog = 0.08, Bird = 0.02
Correct Class: Cat
Cat = 0.10, Dog = 0.80, Bird = 0.10
Cross-entropy punishes confident wrong predictions strongly.
Binary cross-entropy is used for binary classification problems with two classes.
Correct Answer — Spam
Output 0.90 — Loss: Low Output 0.10 — Loss: High
Binary cross-entropy is often used with sigmoid output.
Categorical cross-entropy is used for multi-class classification.
Correct Class — Dog
Model Output — Cat: 0.20, Dog: 0.70, Bird: 0.10 → Loss: Low
If the model gives low probability to Dog, loss becomes high.
Examples — Cat/Dog/Bird, Red/Blue/Green, Positive/Neutral/Negative
Sparse categorical cross-entropy is similar to categorical cross-entropy, but it uses integer class labels instead of one-hot labels.
Class labels — Cat = 0, Dog = 1, Bird = 2
Correct label — 1 (meaning: Dog)
Use — Multi-class classification with integer labels; avoids needing one-hot vectors
Different cross-entropy variants use different label formats.
Cat: [1, 0, 0]
Dog: [0, 1, 0]
Bird: [0, 0, 1]
Used with: Categorical cross-entropy
Cat: 0
Dog: 1
Bird: 2
Used with: Sparse categorical cross-entropy
Language models predict tokens. Cross-entropy loss measures how much probability the model gave to the correct next token.
Input — "The cat sat on the"
Correct Next Token — "mat"
Model Output — Probabilities for many possible tokens
Loss — Lower if the model gives high probability to "mat"
In image classification, the model predicts class probabilities, and loss checks whether the correct class received high probability.
Correct Label — Dog
Model Output — Cat: 0.10, Dog: 0.80, Bird: 0.10 → Loss: Low
If Cat received 0.90 instead, loss would be high.
Recommendation systems may use different loss functions depending on the task.
Predict rating — Regression loss like MSE
Predict click or no click — Binary cross-entropy
Rank items — Ranking loss
The loss function should match what the system is trying to learn.
The right loss function depends on the type of problem.
| Problem Type | Common Loss Function |
|---|---|
| Regression | MAE, MSE, or Huber Loss |
| Binary Classification | Binary Cross-Entropy |
| Multi-Class Classification | Categorical Cross-Entropy |
| Ranking | Ranking Loss |
The loss function should match the output activation.
Sigmoid output — Binary cross-entropy
Softmax output — Categorical cross-entropy
Linear output — MSE or MAE
Spam detection — Sigmoid + Binary Cross-Entropy
Image classification — Softmax + Categorical Cross-Entropy
House price prediction — Linear + MSE or MAE
Loss often considers how confident the model is in its prediction.
Correct Class: Cat
Cat = 0.60
Higher cross-entropy loss
Correct Class: Cat
Cat = 0.95
Lower cross-entropy loss
Loss can be very high when the model is confidently wrong.
Correct Class — Cat
Model Predicts — Dog = 0.99, Cat = 0.01
Loss — Very high
The model was very confident in the wrong answer. Confident mistakes should be strongly corrected.
Training usually calculates average loss across many examples.
Example losses — 0.2, 0.5, 0.3, 1.0
Average loss — (0.2 + 0.5 + 0.3 + 1.0) / 4 = 0.5
One score summarizes a group of examples — used for a batch, epoch, or full dataset.
A batch is a group of examples processed together. Batch loss is the average loss for that batch.
Batch size — 32 examples
Model predicts all 32 examples. Loss is calculated for each.
Batch loss — Average of the 32 losses
Use — Guides the weight update
An epoch is one full pass through the training dataset. Epoch loss is often the average loss across all batches in one epoch.
Epoch 1 Loss — 1.20
Epoch 2 Loss — 0.95
Epoch 3 Loss — 0.70
If epoch loss decreases, the model may be learning.
Loss curves help show whether training is healthy.
Healthy — Training loss decreases, validation loss decreases, gap stays reasonable
Overfitting — Training loss decreases, validation loss increases
Underfitting — Both losses stay high
The learning rate affects how quickly weights change during training.
Loss may jump around
Training may become unstable
Loss may decrease very slowly
Training may take too long
A good learning rate helps loss decrease steadily.
Loss does not always decrease perfectly at every step. It may go up and down during training.
Look at the overall trend, not only one step.
Low training loss does not always mean good real-world performance.
Training Loss — 0.02 Test Loss — 0.90
The model may have memorized training data.
Better check — Validation loss and test loss
A model predicts a house price of $430,000, but the actual price is $450,000. Calculate the error and absolute error.
Then explain why a loss function is needed before the model can update its weights.
Ask the AI if you need help understanding loss functions, training loss, validation loss, test loss, MAE, MSE, cross-entropy, binary cross-entropy, categorical cross-entropy, or how loss helps update weights.