| # | LR | Nodes | Accuracy |
|---|---|---|---|
| Start a search to see trials. | |||
Parameters are learned from data; hyperparameters are chosen before training starts. Searching that second set well is often worth more than any change to the model itself.
Grid search tries every combination on a predefined grid. With 5 learning rates and 5 layer widths that is 25 runs, and adding a third hyperparameter with 5 values makes it 125. The cost is exponential in the number of hyperparameters.Random search samples combinations at random from ranges you specify. Counterintuitively it usually finds better settings for the same budget, and the reason is sharp: not all hyperparameters matter equally. If learning rate dominates and layer width barely matters, a 5×5 grid tests only 5 distinct learning rates across 25 runs. Random search with 25 samples tests 25 distinct learning rates. You get five times the resolution on the axis that counts, at no extra cost.Bayesian optimisation goes further, building a model of which regions look promising and sampling there. It is more efficient per trial and worth the complexity when each run takes hours.
Parameters are learned by gradient descent. Hyperparameters are the settings you choose before training starts, and they decide how well that learning goes.
Not all of them matter equally, and the difference is large enough that tuning them in the wrong order wastes most of the effort:
| Hyperparameter | Impact | Typical range |
|---|---|---|
| Learning rate | Enormous | 1e-5 to 1e-1, log scale |
| Batch size | Moderate | 32–256 |
| Number of epochs | Moderate | Decided by early stopping |
| Weight decay | Moderate | 1e-5 to 1e-1, log scale |
| Architecture depth / width | Moderate | Problem-dependent |
| Dropout rate | Small to moderate | 0.0–0.5 |
| Optimiser choice | Small | AdamW by default |
| Adam's β values | Negligible | Leave at defaults |
The learning rate deserves its position. It affects results more than the number of layers, the width, or which optimiser you pick — and it is the cheapest to find, via a short exponential sweep.
This is the most useful practical result in the topic, and it is counterintuitive.
A grid search over 3 learning rates and 3 dropout rates tries 9 combinations — but only 3 distinct learning rates. If the learning rate is what matters and dropout barely does, you spent 9 runs to sample 3 values of the important parameter.
Random search with 9 trials samples 9 distinct learning rates. For the same budget it explores the important dimension three times as thoroughly, and the result generalises: when only a few hyperparameters matter, random search finds better configurations than grid search at equal cost.
Bayesian optimisation goes further by modelling the relationship between settings and score, and choosing each next trial where the expected improvement is highest. Optuna and similar libraries implement this, and they add another useful feature — pruning, which abandons a trial early once it is clearly losing.
import optuna
def objective(trial):
lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
wd = trial.suggest_float("weight_decay", 1e-6, 1e-1, log=True)
dropout = trial.suggest_float("dropout", 0.0, 0.5)
return train_and_validate(lr=lr, weight_decay=wd, dropout=dropout)
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=50)
print(study.best_params)Note log=True on the learning rate and weight decay. Both span orders of magnitude, and sampling them uniformly wastes almost every trial in the top decade.
Two habits that matter more than the search algorithm: change one thing at a time so you can attribute the effect, and fix the random seed so a 1% difference is a real difference rather than run-to-run variation.
Hyperparameters are not equal -- a few dominate and the rest are noise. This measures which is which on a real training loop, then compares three ways of searching.
Learning rate should be sampled logarithmically, not uniformly. Sampling uniformly from 0.0001 to 0.1 puts 90% of the samples above 0.01, leaving the small-rate region — where the answer usually is — almost untested.
Sample the exponent instead: draw uniformly from −4 to −1 and use 10x. That gives equal attention to 0.0001–0.001, 0.001–0.01, and 0.01–0.1. The same applies to regularisation strength and any other quantity that spans orders of magnitude.
Hyperparameters sit outside gradient descent, so the only way to evaluate them is to train and measure — which makes the search strategy itself worth thinking about. Random search beats grid search at equal budget because it spends its samples on more distinct values of whichever hyperparameter actually matters, and anything spanning orders of magnitude should be sampled on a log scale. Whatever the strategy, select on validation data and keep the test set untouched.
Every hyperparameter decision made by looking at a score consumes some of that set's independence. Search fifty configurations against a validation set and keep the best, and its score is optimistic — you selected the luckiest of fifty.
The disciplined arrangement is three splits: train fits the parameters, validation chooses the hyperparameters, and test is looked at once, at the end, to produce the number you report.
For a rigorous estimate, nested cross-validation puts the hyperparameter search inside an outer loop whose test folds never participate in tuning. It costs k times more compute, and it is what you want when the number will inform a real decision.
The pragmatic middle ground — cross-validation for tuning, plus one genuinely untouched hold-out set for the final number — is what most projects should do.
Hyperparameter search has diminishing returns, and it is worth knowing roughly where they set in.
The first few trials usually find most of the available gain, because they fix the learning rate. Trials ten to fifty refine it. Beyond a hundred, improvements on a fixed dataset are typically fractions of a percent — at which point better data, better features or a different architecture is a far better use of the compute.
Two shortcuts worth knowing. Tune on a subset — 10–20% of the data — to find a promising region cheaply, then verify the best few configurations on the full dataset. And use early pruning, so a configuration that is clearly losing after two epochs does not consume twenty.
What should I tune first? The learning rate, always. It dominates.
How many trials do I need? Twenty to fifty random or Bayesian trials cover most of the available gain for a handful of hyperparameters.
Grid or random search? Random, or Bayesian. Grid search wastes its budget on dimensions that do not matter.
Should I tune the architecture? Only after the training recipe is right, and prefer a proven architecture over designing one.
Do I need to tune Adam's betas? Almost never. The defaults are robust.
Why do my results change between runs with the same settings? Random initialisation, shuffling and non-deterministic GPU kernels. Fix the seeds, and treat differences smaller than the run-to-run variation as noise.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Two kinds of number”?
What does this module say about “Grid search, random search, and why random usually wins”?
Experiment with automated search strategies. Use Grid Search to systematically scan parameters or Random Search to find high-performance zones efficiently.
AI.Cruz builds and maintains AI.Cruz. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.