Gradient Descent Explained in Plain Words

Gradient descent explained in one sentence: it is the method most AI models use to learn, by adjusting their internal numbers in small steps that reduce a measure of error. Imagine standing on a foggy hillside and trying to reach the valley. You cannot see the whole landscape, but you can feel which way the ground slopes under your feet. You take a step downhill, feel again and repeat. Gradient descent does the same thing with a model: at each step it checks which direction lowers the error and moves a little that way.

This guide explains gradient descent in plain words, with only light arithmetic. It covers the idea of a loss function, what a gradient is, the learning rate, the main variants, why training can stall, how the method scales to models with billions of parameters, common failure signs and what it means for everyday users of AI assistants. It is written for curious readers, students and professionals who hear the term and want to understand what it actually does.

Gradient descent explained with a hill

Every model has numbers inside it, called parameters or weights, which determine what it outputs. At the start they are mostly random, so the model is bad at its task. Training is the process of changing those numbers until the model becomes good. To do that, we need a way to measure how bad the model is and a way to decide how to change the numbers.

Measure the error. A loss function turns the model’s mistakes into one number. Lower is better.
Find the direction. The gradient tells us, for each parameter, whether increasing it would raise or lower the loss, and by how much.
Take a small step. Move every parameter slightly in the direction that reduces the loss.
Repeat. Do this many thousands or millions of times.

In the hill picture, the position on the map is the set of parameters, the height is the loss and the gradient is the slope under your feet. Walking downhill repeatedly brings you to a low point. The article on gradient descent gives the mathematical version of the same idea.

The loss function: what we are minimising

The loss function defines what “wrong” means. For predicting a house price, it may be the average squared difference between predicted and actual prices. For a language model, it measures how surprised the model is by the next word in real text: if the model gave the true next word a high probability, the loss is low, and if it gave it a tiny probability, the loss is high. Choosing the loss is a design decision that shapes behaviour, because the model will get better at exactly what the loss rewards.

Regression loss. Measures numeric error, such as how far a predicted temperature is from the real one.
Classification loss. Measures how confident the model was in the right category.
Language modelling loss. Measures how well the model predicts the next piece of text.
Combined or custom loss. Adds penalties for things you want to avoid, such as overly complex models.

A lower loss on training data is not the same as a useful model, a problem called overfitting. A model can score well on its training data and still fail on new data, so engineers always check performance on examples it has not seen.

The gradient: which way is downhill

The gradient is a list of slopes, one for each parameter. A large positive slope for a parameter means that increasing it would raise the loss a lot, so we should decrease it. A slope near zero means the parameter has little effect right now. Computing all these slopes efficiently in a neural network is the job of an algorithm called backpropagation, which passes the error backwards through the layers and uses the chain rule from calculus to assign each parameter its share. Our explanation of neural networks describes the layers that the gradient flows through.

You do not need the formulas to follow the logic. Gradient descent only needs two things: a way to compute the loss and a way to compute the slopes. If both are available, the same procedure works for a tiny model with two parameters and for a large model with billions.

The learning rate: how big each step is

After finding the direction, you must choose the step size. This is the learning rate, and it is one of the most important settings in training.

Learning rate	What happens	Analogy
Too small	Training is very slow and may stall before reaching a good point	Shuffling downhill an inch at a time
About right	Loss falls steadily and settles near a low point	Steady steps down the slope
Too large	Loss bounces around or rises, because steps overshoot the valley	Leaping across the valley and landing on the other side
Changing over time	Large steps early for speed, small steps later for precision	Striding down the mountain and then walking carefully at the bottom

Practitioners often use a schedule that starts with a short warm-up, then a high rate, then a gradual decrease. Finding a good schedule takes experiments, and large training runs are expensive enough that mistakes here cost real money.

Batch, stochastic and mini-batch gradient descent

Computing the loss on every training example before each step would be accurate and extremely slow for large data sets. Variants trade accuracy for speed.

Batch gradient descent. Uses all the data for each step. Smooth but slow and memory-hungry.
Stochastic gradient descent. Uses one example at a time. Fast, noisy steps, which can even help escape poor spots. The article on stochastic gradient descent covers it in detail.
Mini-batch gradient descent. Uses a small group of examples, such as 32 or 1,024, for each step. This is the common choice because it balances speed, stability and hardware efficiency.

Modern training also uses smarter step rules. Momentum keeps a memory of previous steps so that progress continues along consistent directions and smooths out noise. Adaptive methods adjust the step size for each parameter based on recent gradients. These improvements are variations on the same downhill idea.

Why training can get stuck or go wrong

The loss landscape of a real model is not a smooth bowl. It has flat regions, narrow valleys, saddle points and many low areas of different quality. Several difficulties follow.

Local minima and plateaus. The method can slow down in flat areas or settle in a point that is good but not the best. In very large models, most poor points turn out to be less harmful than people once feared, but plateaus still waste time.
Vanishing gradients. In deep networks, the slopes can become tiny in early layers, so those layers barely learn. Architecture choices and better activation functions reduce the problem.
Exploding gradients. The slopes become huge and the parameters jump to extreme values. Clipping the gradient size is a standard defence.
Poor scaling of inputs. If one input ranges from 0 to 1 and another from 0 to a million, steps can zigzag. Normalising inputs helps.
Noisy or wrong data. Gradient descent faithfully learns whatever the loss rewards, including mistakes and bias in the data, a theme covered in our article on AI bias.

Signs that something is wrong include a loss that never falls, a loss that suddenly jumps to infinity or not-a-number values and a training loss that falls while the loss on new data rises.

How it scales to huge models

Large language models have billions of parameters and train on trillions of words, so each step involves enormous computation. The core algorithm is still gradient descent, run in mini-batches across thousands of chips. Engineers split the model and the data across machines, keep copies of the numbers in lower precision to save memory and save checkpoints so that a failed run can restart. The cost grows with model size and data, which is why training is done by a small number of organisations while many more fine-tune or use the results. Our articles on pretraining, scaling laws and model parameters explain how size, data and compute relate.

What gradient descent means for a user of AI assistants

When you chat with an assistant, no gradient descent is happening. The learning took place earlier, during training, and the parameters are fixed while you talk. What you type does not change them. Your conversation changes only the text the model sees at that moment, which is why the assistant can adapt within a chat and then forget it in a new one unless a memory feature stores notes. Understanding this explains several everyday facts: the knowledge cutoff, the reason that models repeat a mistake in the same way and the reason that improvements arrive as new versions.

Mio is one assistant with Chat, Write, Research, Code and Design modes, and you never pick a model: Mio routes each request to one suited to the task. Chat and Write modes are on every plan, including the free plan with 600 points a month, so you can ask follow-up questions about topics like this one at no cost. Your chats and files remain yours, they are not used to train models and you can export or delete them at any time.

A tiny worked example

Take a model with one parameter w that predicts y as w times x. Our data says that when x is 2, y is 6, so the ideal w is 3. Start with w equal to 0. The prediction is 0, the error is large and the gradient says that increasing w would reduce the error. With a learning rate of 0.1, the first step moves w to a small positive number, perhaps around 1.2. The next step uses the new, smaller error and moves w a little further, and so on. After a handful of steps w is close to 3 and the steps become tiny because the slope flattens near the bottom. If the learning rate were 1.0, w would overshoot past 3 and bounce back and forth. That behaviour, at small scale, is exactly what happens inside real models.

Frequently Asked Questions

Is gradient descent the same as machine learning?

No. Machine learning is the broad field of systems that learn from data, and gradient descent is one very common method for the learning step, especially in neural networks. Some methods, such as decision trees, learn without gradients. For deep learning, gradient descent and its variants are the standard tool.

What is the learning rate in simple terms?

It is the size of each step downhill. Too small and training crawls, too large and it overshoots and becomes unstable. Practitioners often start with a recommended value, watch the loss curve and adjust, sometimes using a schedule that begins with larger steps and shrinks them as training progresses.

Why is stochastic gradient descent used so much?

Because using all the data for every step is too slow. Using small random groups of examples gives a noisy but useful estimate of the direction at a fraction of the cost. The noise can even help the model avoid poor spots. Mini-batches also fit well with the parallel hardware used for training.

Does gradient descent always find the best solution?

No. It finds a good low point, but not necessarily the lowest one, and the result depends on the starting values, the data, the learning rate and luck. In practice, for large networks, many different low points give similar performance, which is why the method works so well despite lacking a guarantee.

Does an assistant learn from my chat using gradient descent?

Not during the chat. The model’s parameters are fixed when you use it, and the conversation only provides context for the current reply. Whether a provider later uses conversations for training is a separate policy question. With Mio, chats and files are your own and are not used to train models.

The Bottom Line

Gradient descent explained simply: measure the error, find which way is downhill, take a small step and repeat until the model is good enough. The learning rate sets the step size, mini-batches make the process practical and tricks such as momentum and clipping keep it stable. Understanding it helps you read news about training, costs and model behaviour with more confidence. To explore AI concepts with an assistant, start on the free plan and compare options on the Ask Mio pricing page.

Gradient Descent Explained: How Models Learn Step by Step