Gradient descent is an optimization method used to reduce loss. It helps a model find better weights by slowly changing them in the direction that lowers the loss.
Make prediction
Calculate loss
Find direction to reduce loss
Update weights
Repeat — predictions improve over time
Gradient descent matters because machine learning models need a way to improve. At the beginning of training, the model's weights may be random and predictions may be poor.
High loss
Predictions are poor or random
Lower loss
The model is learning better weights
Optimization means finding the best values for something. In machine learning, optimization usually means finding weights that minimize loss.
Which weights give the lowest prediction error?
Gradient descent is one method for searching for those weights.
A loss function measures how wrong the model is. Gradient descent tries to reduce this loss.
Prediction: Dog
Correct Answer: Cat
Loss: High
Prediction: Cat
Correct Answer: Cat
Loss: Low
Weights are learned numbers inside a model. They control how strongly inputs affect predictions.
Feature: Suspicious links — Weight: High
Feature: Normal greeting — Weight: Low
Gradient descent changes these weights during training.
A gradient tells the model how the loss changes when a weight changes.
Gradient — Direction and amount of change needed to reduce loss
Important — The gradient points toward increasing loss
Gradient descent — Moves in the opposite direction, toward decreasing loss
The word descent means moving downward. In gradient descent, the model moves downward on the loss surface toward lower loss.
You are standing on a hill. Your goal is to reach the lowest point.
You look at the slope and step downhill.
Gradient descent uses gradients to step toward lower loss the same way.
A loss surface represents how loss changes for different weight values. Gradient descent searches this surface for lower loss.
Bad weights — High loss
Better weights — Lower loss
Best weights found — Lowest loss the model can reach
The learning rate controls how big each gradient descent step is. It is one of the most important training settings.
High learning rate — Large steps
Low learning rate — Small steps
Good learning rate — Helps loss decrease steadily
If the learning rate is too high, the model may take steps that are too large.
The model keeps jumping over the lowest point instead of settling near it.
If the learning rate is too low, the model takes very small steps.
A low learning rate can work, but it may take too long.
A good learning rate is not too high and not too low. It lets the model move steadily toward lower loss.
Choosing the learning rate often requires testing.
Start with initial weights
Make predictions using forward propagation
Calculate loss
Calculate gradients
Update weights in the direction that reduces loss
Repeat many times
A basic weight update subtracts the gradient multiplied by the learning rate.
New weight = Old weight − Learning rate × Gradient
Old weight — Current value
Gradient — Direction loss increases
Learning rate — Step size
Subtracting the gradient moves the weight toward lower loss.
This example shows one gradient descent weight update.
Old weight — 10
Learning rate — 0.1
Gradient — 4
Calculation — New weight = 10 − 0.1 × 4 = 10 − 0.4 = 9.6
The weight moved slightly in a direction that should reduce loss.
Backpropagation and gradient descent work together during neural network training.
| Process | Main Job |
|---|---|
| Backpropagation | Calculates gradients |
| Gradient Descent | Uses gradients to update weights |
Neural networks can have many weights. Gradient descent updates these weights gradually based on how they affected the loss.
Forward propagation — Makes prediction
Loss function — Measures error
Backpropagation — Calculates gradients
Gradient descent — Updates weights
Batch gradient descent uses the entire training dataset to calculate one weight update.
Training dataset — 10,000 examples
Batch gradient descent — Uses all 10,000 examples before updating weights
Advantage — Stable updates
Disadvantage — Can be slow for large datasets
Stochastic Gradient Descent, or SGD, updates weights using one training example at a time.
Training example 1 — Update weights
Training example 2 — Update weights
Training example 3 — Update weights
Advantage — Faster updates Disadvantage — Updates can be noisy
Mini-batch gradient descent updates weights using a small group of examples. It is commonly used in deep learning.
Training dataset — 10,000 examples Batch size — 32
The model processes 32 examples, calculates average loss, then updates weights.
Benefit — Balances speed and stability
Batch size is the number of examples used before one weight update. It affects training speed, memory use, and stability.
Batch size 1 — Very noisy updates
Batch size 32 — Common mini-batch size
Batch size 10,000 — Full batch update, stable but slow
An epoch is one full pass through the training dataset. Gradient descent may update weights many times during one epoch.
Dataset size — 1,000 examples
Batch size — 100
Batches per epoch — 10
Weight updates per epoch — 10
More epochs mean more chances for the model to improve.
An iteration is one weight update step. If one batch produces one update, then one batch equals one iteration.
Epoch — One full pass through the dataset
Batch — Subset of the dataset used per update
Iteration — One weight update step
Gradient descent happens at each iteration.
A loss curve shows how loss changes during training. It helps show whether gradient descent is working.
Loss decreases steadily
Validation loss also improves
Loss jumps around or does not decrease
Training loss improves but validation loss gets worse
A local minimum is a point where loss is lower than nearby points, but not necessarily the lowest possible loss overall.
Lower than nearby points, but may not be the absolute lowest.
In deep learning, finding a useful low-loss point is often more important than finding a perfect minimum.
A global minimum is the lowest possible point on the loss surface.
Meaning — Lowest possible loss for the model and data
Simple problems — May be easier to find
Deep learning — Loss surfaces are complex; the goal is often good enough weights that generalize well
Gradient descent reduces training loss, but training too long can lead to overfitting.
Training loss — Keeps decreasing
Validation loss — Starts increasing
The model is improving on training data but getting worse on new data.
Possible fix — Early stopping
Underfitting can happen when gradient descent does not reduce loss enough.
Training loss — High Validation loss — High
Possible Causes — Model is too simple, learning rate is too low, not enough epochs, poor features, too much regularization
Regularization helps prevent overfitting and can affect how gradient descent updates weights.
Goal: Encourage simpler patterns that generalize better.
Feature scaling can help gradient descent work better. If features have very different ranges, training may be slower or unstable.
Feature 1 — Age ranges from 0 to 100
Feature 2 — Income ranges from 0 to 200,000
Problem — Different scales can make optimization harder
Solution — Scale features before training
Normalization and standardization are common preprocessing methods that can make gradient descent more efficient.
Scales values to a range, often 0 to 1.
Centers values around mean 0 and standard deviation 1.
Both methods can help gradient descent move more smoothly.
In regression, gradient descent reduces numerical prediction error.
Task — Predict house price
Prediction — $420,000 Actual — $450,000
Loss — Based on price error
Goal — Update weights so future predictions are closer to actual prices
In classification, gradient descent reduces classification loss.
Task — Classify image as cat, dog, or bird
Correct Class — Dog Model Output — Dog = 0.20 Loss — High
Goal — Update weights so the model gives higher probability to Dog in similar examples
Language models use gradient descent to improve token prediction.
Input — "The cat sat on the"
Correct next token — "mat"
If the model gives low probability to "mat", loss is high.
Gradient descent — Updates weights so similar predictions improve later
An optimizer is the algorithm that updates model weights. Gradient descent is the basic idea behind many optimizers.
All optimizers use gradients to reduce loss.
Adam is a popular optimizer in deep learning. It adapts learning behavior for different weights and often works well as a default choice.
Adam is still based on gradient information.
RMSprop adjusts updates based on recent gradient sizes. It can help with unstable gradients and changing loss surfaces.
Uses recent gradient behavior to adjust updates.
Useful when — Training is unstable or gradients change a lot
Momentum helps gradient descent keep moving in a useful direction by remembering past update directions.
Momentum is often combined with SGD.
A common analogy is rolling a ball down a hill. The height represents loss, and the ball's position represents the model's weights.
Height — Loss
Ball position — Weights
Slope — Gradient
Goal — Move downhill toward lower loss
Gradient descent can struggle if the training setup is poor. Training a model requires monitoring and tuning.
Vanishing gradients happen when gradients become very small, causing learning to slow down.
Gradients become tiny
Weight updates become very small
Learning slows down — especially in earlier layers
ReLU, normalization, and better architectures can help reduce this issue.
Exploding gradients happen when gradients become very large, causing unstable training.
Gradients become huge
Weight updates become huge
Training becomes unstable — loss may jump around
Gradient clipping can help control exploding gradients.
Gradient clipping limits gradients when they become too large. This helps prevent unstable updates.
If a gradient is above a chosen limit, reduce it to a safer size.
Useful for — Deep networks, sequence models, training with unstable gradients
A model has an old weight of 8, a learning rate of 0.05, and a gradient of 6.
Use the formula New weight = Old weight - Learning rate × Gradient to calculate the new weight. Then explain why the model subtracts the gradient instead of adding it.
Ask the AI if you need help understanding gradient descent, gradients, learning rate, loss curves, weight updates, batch size, epochs, stochastic gradient descent, mini-batch gradient descent, Adam, or the difference between backpropagation and gradient descent.