Grid Search against Random Search
Same budget, two ways of spending it. Grid search tries nine values of everything; random search tries every value of the one that matters.
Overview
The setup
The background shading is the true score surface, and it is deliberately lopsided. The horizontal axis is a hyperparameter that matters a great deal — a learning rate, a regularisation strength — and the vertical axis is one that barely does. The optimum is a narrow vertical band.
This lopsidedness is the normal case, not a contrived one. In most models one or two hyperparameters dominate and the rest are close to noise, and you usually do not know in advance which is which.
Grid Search against Random Search
Worth knowing
Grid Search against Random Search
Why the same number of trials finds a better answer when you stop being systematic.
Count the distinct values
Set the method to Grid with 16 trials. Sixteen points appear in a 4×4 lattice. Read the line under the chart: 16 trials, 4 distinct values of the parameter that matters.
Every point in a column shares an x. The other three trials in that column tell you nothing new about the axis you care about, because they only vary the axis you do not.
Now switch to Random, same 16 trials. Sixteen points scattered, and 16 distinct values of the parameter that matters. Same cost, four times the resolution where resolution counts.
Push the budget to 64. Grid gives 8 distinct values from 64 trials; random gives 64. The gap widens as the budget grows, and it widens much faster as dimensions are added — a grid over five parameters at four values each is 1,024 trials and still only four values of each.
The argument in one line
A grid of *n* points per axis over *d* axes costs *n^d* trials and buys *n* distinct values of each parameter. Random search of the same *n^d* trials buys *n^d* distinct values of every parameter.
If only one axis matters, grid search has wasted all but *n* of its trials on that axis. This is Bergstra and Bengio's 2012 result, and it changed default practice.
Watch the luck
Move the seed slider with random selected. The best point found moves, and the best score wobbles. Random search is random; a single run can be unlucky.
That is a real cost and a smaller one than it looks. Because the surface is broad along the unimportant axis, most draws land somewhere reasonable, and the distribution of outcomes is much better than grid's guaranteed coarseness. But it does mean a single random run is not a strong claim, and comparisons should allow for the variation.
When grid search is still right
Few parameters, genuinely discrete. Three kernels, four values of *k*: a grid is exhaustive and the exhaustiveness is worth having.
Reproducibility matters more than efficiency. A grid is deterministic and easy to describe in a paper.
You are refining. Random search to find the region, a small grid to comb it. This combination is better than either alone.
What replaced both
Bayesian optimisation builds a model of the score surface from the trials so far and proposes the point most likely to improve. Optuna and Hyperopt do this, and on expensive objectives — where one trial takes an hour — it is clearly better than either method here.
Successive halving and Hyperband attack the cost differently: start many configurations cheaply, kill the worst, give the survivors more budget. When a trial's early performance predicts its final performance, this is dramatically more efficient.
For anything cheap, random search remains an excellent default. It has no hyperparameters of its own, it parallelises perfectly, and it is very hard to use wrongly.
Why random search reaches the good region
Random search's advantage over a grid is that it does not spend trials re-measuring the axis that does not matter, and its chance of landing in the good region follows a simple curve. If the top 5% of the space is what you want, n independent random draws miss it only if *all* of them do:
| random trials | P(at least one lands in the top 5%) |
|---|---|
| 10 | 0.40 |
| 20 | 0.64 |
| 60 | 0.95 |
Sixty trials find the good region 95% of the time, with no assumption about which hyperparameter matters. A grid of the same 60 trials over two axes gives about 8 distinct values of each, so if only one axis matters it has spent all but 8 of its trials varying the one that does not — and over five axes at four values each, a grid is 1,024 trials and still only four values per parameter. That is Bergstra and Bengio's result, and it is why random beat grid as the default. The one place it needs care is the sampling scale: a learning rate must be drawn log-uniformly, because a uniform draw between 0.0001 and 0.1 puts 90% of its samples above 0.01 and starves the small values that often matter most.
What replaced both methods for expensive objectives is Bayesian optimisation — Optuna, Hyperopt — which builds a model of the score surface from the trials so far and proposes the point most likely to improve, clearly better when a single trial takes an hour. Successive halving and Hyperband attack the cost from another angle, starting many configurations cheaply and killing the worst, which wins when early performance predicts final performance. For anything cheap, random search remains an excellent default: it has no hyperparameters of its own, it parallelises perfectly, and the one way to misuse it — a linear scale on a parameter that should be logarithmic — is the same trap noted above.
Where it goes wrong
Sampling scale. A learning rate should be sampled log-uniformly. Uniform between 0.0001 and 0.1 puts 90% of the draws above 0.01.
Tuning on the test set. Same error as everywhere else. Tune on validation.
Searching a range that excludes the answer. No search finds what is outside its bounds. Check whether the best value sits at the edge of the range; if it does, widen it and search again.
Not searching preprocessing. Put the whole [pipeline](ml_pipelines.html) in the search, so imputation and scaling choices are tuned alongside the model.
Why random search finds more with fewer fits
Grid search spends its budget evenly across every parameter, including the ones that do not matter. Random search does not, and on the same budget it usually wins.
Check yourself
0 of …Answer without scrolling back up.
Why does a 4x4 grid of 16 trials explore an important hyperparameter poorly?
How should a learning rate be sampled in a random search?
What should you check if the best value found sits at the edge of the search range?
Grid Search against Random Search
The background shading is the true score surface, and it is deliberately lopsided. The horizontal axis is a hyperparameter that matters a great deal — a learning rate, a regularisation strength — and the vertical axis is one that barely does. The optimum is a narrow vertical band.