Hyperparameters Explained: The Settings You Choose Before Training

Hyperparameters are the settings you choose before a machine learning model starts to train. They are not learned from the data. A person, a script or a search procedure picks them, and they shape how the learning happens: how big the steps are, how many times the model sees the data, how large the network is and how strongly it is discouraged from memorising. The model’s weights, in contrast, are the numbers it learns during training. Getting hyperparameters wrong can make a good idea fail, and getting them right can turn a mediocre model into a useful one, which is why practitioners spend so much time on them.

This guide explains hyperparameters in plain words. It covers how they differ from parameters, the most common ones, how they interact, how people search for good values, how to evaluate fairly, what goes wrong, how the idea shows up in the settings of AI assistants, and how to learn more with an assistant such as Ask Mio. No mathematics is needed.

Hyperparameters versus parameters

The two words sound alike and are easy to confuse. Parameters are the internal numbers a model adjusts while it trains, such as the weights in a neural network. A large model can have billions of them. Hyperparameters sit one level above: they are set from outside and control the training process or the model’s structure. Our article on AI model parameters covers the first group, and this one covers the second.

A cooking analogy helps. The recipe’s ingredients and their proportions are the model’s design, the oven temperature and baking time are hyperparameters, and the taste after baking is the result. You choose the temperature and time before you start, and you judge them by the outcome. If the cake is burnt, you do not blame the flour; you change the settings and try again.

The common hyperparameters

Hyperparameter	What it controls	If too low	If too high
Learning rate	Size of each update step	Training is slow or stalls	Training bounces or diverges
Batch size	Examples per update	Noisy updates, slow use of hardware	Smoother but may generalise worse and need more memory
Number of epochs	Passes over the data	Model underfits	Model overfits, time wasted
Number of layers and units	Model capacity	Cannot capture patterns	Memorises, costs more
Regularisation strength	Penalty for complexity	Overfitting	Underfitting
Dropout rate	Share of units switched off during training	Little protection	Learning becomes hard

Other hyperparameters include the choice of optimiser, the shape of the learning-rate schedule, the number of trees in a forest, the depth of a tree and the number of neighbours in a nearest-neighbour method. The names differ by method, but the idea is the same: a setting that you fix before training.

Learning rate: the setting people tune first

If you could tune only one hyperparameter, it would usually be the learning rate. It decides how far the model moves its weights at each update, as described in the methods behind gradient descent. With a very small value, progress is slow and training may take days longer than needed. With a large value, the model overshoots good solutions and the error jumps around or grows. Many practitioners start with a standard value for their optimiser, try a few values across orders of magnitude, and use a schedule that lowers the rate as training goes on, so early steps explore and later steps refine.

• Warm-up: start with a small rate and increase it for the first steps to avoid early instability.
• Decay: reduce the rate over time so the model settles.
• Monitoring: watch the loss curves. A smooth decline is healthy, while wild swings suggest the rate is too high.

Batch size, epochs and training time

Batch size sets how many examples the model sees before it updates its weights. Small batches give noisier updates, which can help escape poor solutions, but they use hardware less efficiently. Large batches are faster per example and steadier, but they need more memory and sometimes interact badly with the learning rate. The number of epochs, or full passes over the training data, determines how long the model learns. Too few and it has not learned enough, too many and it begins to memorise noise. A common practice is early stopping: monitor the error on held-out data and stop when it no longer improves.

Model size and regularisation

The number of layers and units sets the model’s capacity. A small model may miss patterns, and a huge one may memorise the training set. Regularisation methods, including penalties on large weights and dropout, push the model toward simpler solutions that generalise better. These settings are strongly linked to the problem of overfitting, where a model scores well on training data and poorly on new data. The right balance depends on the amount and quality of data, which is why tuning is done by experiment and not by formula.

How people search for good values

There is no universal best setting, so practitioners search. The methods range from simple to sophisticated.

1. Manual tuning. Change one setting at a time, based on experience and on the training curves.
2. Grid search. Try every combination from a small list of values. It is thorough but grows expensive quickly as you add settings.
3. Random search. Sample combinations at random. Research has shown that this often finds good settings faster than a grid, because only a few hyperparameters usually matter much.
4. Bayesian optimisation. Use the results so far to choose the next promising setting.
5. Automated tuning tools. Software that runs many experiments, stops weak ones early and records the results.

The overview of hyperparameter optimization describes these methods in more detail.

Evaluating fairly: train, validation and test

When you tune hyperparameters, you repeatedly look at results, and that can contaminate your judgment. Practitioners split the data into three parts. The training set is used to learn the weights. The validation set is used to compare hyperparameter choices. The test set is touched only at the end, to estimate how the final model will perform on new data. If you tune on the test set, the final score is too optimistic, because the choices have been fitted to it. Cross-validation, which rotates the validation role among several slices of the data, gives more reliable comparisons when data is limited.

• Never choose settings by looking at the test score.
• Record every experiment: settings, data version, random seed and result.
• Repeat promising runs with different seeds, because results vary by chance.

What goes wrong in tuning

• Tuning too many things at once. You learn nothing about which change mattered.
• Chasing tiny gains. Differences within random noise are not real improvements.
• Leaking test data. Using it for decisions inflates results.
• Ignoring cost. A tiny improvement may need ten times the computing time.
• Forgetting the data. Cleaner or more data often beats clever tuning.

Hyperparameters you see in AI assistants

The idea also appears in the settings of tools you use. When an assistant or an API offers controls such as temperature, which adjusts how random the next-word choice is, or a limit on output length, these are settings chosen at use time and not weights learned in training. They are related in spirit, though not identical. Our guide to AI temperature explains one of the best-known examples. In Ask Mio, you do not need to adjust such settings: the pricing page says Mio routes every request to the best model for the task, so you never have to choose one. Behind the scenes, the people who train and deploy models make many of the decisions described in this article.

Hyperparameters in large language models

For the very large models behind chat assistants, tuning is far more expensive, because a single training run can cost a great deal and cannot be repeated dozens of times. Teams therefore tune settings on smaller versions of a model, look at how results change with size, and extrapolate carefully. The choices include model width and depth, the learning-rate schedule, batch size in tokens and the mix of training data, described in our guides to AI pretraining and neural networks. The general definition is summarised in the article on a hyperparameter in machine learning. The lesson for ordinary users is simple: behind a smooth answer lie many design decisions that were tested, measured and adjusted long before you typed a question.

A worked example: tuning a small classifier

A team trains a model to sort support emails into five topics. They start with a standard learning rate and batch size and see training accuracy at 99 percent but validation accuracy at 81 percent: a sign of overfitting. They add dropout and a weight penalty, lower the number of epochs with early stopping and try three learning rates. Validation accuracy rises to 88 percent. They run random search over dropout, penalty and layer size for twenty experiments, find a configuration at 90 percent, and confirm it once on the held-out test set, which shows 89 percent. They record all settings and the random seed. The work took a day of experiments and avoided a bigger model that would have cost more without helping.

Learning more with an assistant

An assistant is a patient tutor for this topic. Ask Mio to explain hyperparameters with a cooking or sports analogy, then to list the settings of a model you use and what each does. Ask it to design a tuning plan for a small project, including which settings to try first and how to split the data. The features page lists a calculator for exact arithmetic and, on plans with the sandbox, a code runner for Python, so you can ask for a short script that compares learning rates on a toy problem and prints the loss. Chat mode is on every plan, including the free plan with 600 points a month, and a normal reply costs 1 point, according to the pricing page.

Frequently Asked Questions

What are hyperparameters in simple words?

They are the settings you choose before training a model, such as the learning rate, the batch size, the number of epochs and the size of the network. They are not learned from data. They control how learning happens, so they strongly influence whether the model trains well and generalises to new examples.

How are hyperparameters different from parameters?

Parameters, such as weights, are the numbers a model learns during training. Hyperparameters are set from outside before or around training and control the process or the structure. A model can have billions of parameters but only a handful of hyperparameters that a person tunes.

Which hyperparameter should I tune first?

The learning rate is usually the most important, because it decides how big each update step is. After that, batch size, the number of epochs and regularisation strength are common choices. Change one thing at a time, watch the validation results and keep a record of every experiment.

Is grid search or random search better?

Random search often finds good settings faster, because only a few hyperparameters usually matter a lot and a grid wastes effort on unimportant ones. Grid search is still reasonable for a small number of settings with a few values each. More advanced methods use previous results to choose the next trial.

Do I need to set hyperparameters when I chat with an assistant?

No. Those hyperparameters belong to the training stage, which the provider handles. Some tools expose settings such as temperature or output length. In Ask Mio, the pricing page says Mio routes each request to the best model for the task, so you do not need to choose settings yourself.

The Bottom Line

Hyperparameters are the settings chosen before training that steer how a model learns, from learning rate and batch size to regularisation and model size. Tune them with a validation set, change one thing at a time, keep the test set untouched and record every run. Ask Mio to explain any of them with small examples, and try the free plan or compare options on the Ask Mio pricing page.