Choosing k: Elbow and Silhouette
Inertia always falls as k rises, so it cannot pick one. The silhouette can, and the two often disagree.
Overview
Why inertia cannot answer the question
Inertia is the sum of squared distances from every point to the centroid of its cluster. K-means minimises it directly, so it is the obvious thing to look at.
It is also useless on its own, and the reason is structural: inertia falls at every increase in k, always. With k equal to the number of points, every point is its own centroid and inertia is exactly zero. Watch the left chart as the spread control moves — the curve descends monotonically no matter what the data looks like.
So "choose the k with the lowest inertia" always answers "as many as possible". The metric measures how tightly points sit around their centres, not whether the grouping is any good.
Choosing k: Elbow and Silhouette
Worth knowing
Choosing k: Elbow and Silhouette
Two ways of picking the number of clusters, what each actually measures, and why they disagree.
The elbow
The elbow method looks for the k where the curve stops falling steeply — where extra clusters stop buying much. On well-separated blobs there is a genuine bend at the true number, because up to that point each new cluster splits a real group, and after it each new cluster splits a group that was already coherent.
The problems are practical. The bend is judged by eye and different people pick different points. Very often, on real data, there simply is no bend — the curve is smooth, and reading an elbow into it is wishful. Raise the spread control until the blobs overlap and the bend disappears in front of you.
There is a formalisation, the kneedle algorithm, which finds the point of maximum curvature. It is better than eyeballing, and it still assumes a knee exists.
The silhouette
The silhouette measures something different: not tightness, but separation relative to tightness.
For each point, let *a* be its mean distance to the other members of its own cluster, and *b* its mean distance to the members of the nearest other cluster. The silhouette is:
s = (b - a) / max(a, b)Near +1 the point is much closer to its own cluster than to any other. Near 0 it sits on a boundary. Negative means it is closer to a different cluster than to its own — it is probably misassigned.
Averaging over all points gives one number per k, and unlike inertia it does not improve automatically as k grows. Adding clusters eventually forces points close to a neighbouring cluster, which drives *b* down and the score with it. So the silhouette has a real maximum, and that maximum is a defensible choice.
The right-hand chart shows this. The peak is at the true number of blobs, and it is a peak rather than a slope.
The cost is computation: it needs pairwise distances, so O(n²), against inertia's O(n).
When they disagree
They frequently do, and the disagreement is informative rather than a problem to resolve.
Inertia asks *are the clusters tight*. The silhouette asks *are they separated*. A dataset can have tight clusters that sit right next to each other, and the two metrics will point in different directions.
Switch the data to Two crescents and watch both. The silhouette does not recover the two crescents either — because it is still built on distances to cluster members, and k-means has not produced the crescents in the first place. That is the honest limit of both metrics: they evaluate the clustering you gave them, and cannot tell you the algorithm was wrong.
The answer that is usually right
Neither metric knows what the clusters are for.
If you are segmenting customers so a marketing team can write one message per segment, k is roughly how many messages they can write. If you are compressing colours, k is the palette size you can afford. If clusters feed a downstream model, the number that makes that model best is the number.
Both metrics are worth computing, and the application usually decides.
The silhouette of one point
Where inertia only ever falls as k grows — reaching zero when every point is its own cluster — the silhouette has a real maximum, because it measures separation against tightness rather than tightness alone. For one point, let a be its mean distance to its own cluster and b its mean distance to the nearest other cluster; s = (b - a) / max(a, b):
| a (own) | b (nearest other) | silhouette | reading |
|---|---|---|---|
| 1.0 | 3.0 | 0.67 | comfortably inside its cluster |
| 2.5 | 3.0 | 0.17 | near the boundary |
| 3.0 | 1.0 | −0.67 | closer to another cluster — misassigned |
Averaging s over all points gives one number per k, and unlike inertia it does not improve automatically: add too many clusters and points are forced close to a neighbouring cluster, which drives b down and the score with it, so the curve peaks at a defensible k rather than sliding forever toward "more". The cost is O(n²) pairwise distances against inertia's O(n). Both still evaluate only the clustering they were handed, though — on two interleaved crescents that k-means never separated, the silhouette cannot recover them either, which is the honest shared limit: these scores judge a partition, they do not know the algorithm was the wrong one.
The honest last word is that neither metric knows what the clusters are *for*. If you are segmenting customers so a team can write one message per segment, k is roughly how many messages they can write; if you are quantising colours, k is the palette you can afford; if the clusters feed a downstream model, the k that makes that model best is the answer. The elbow and the silhouette are both worth computing — the elbow reads tightness, the silhouette reads separation, and their disagreement is informative rather than a problem to resolve — but the application usually decides, and a formalisation like the kneedle algorithm only helps once you have accepted that a knee exists at all.
Where it goes wrong
Choosing k by inertia alone. It will always say "more".
Seeing an elbow in a smooth curve. Check whether the bend survives a change of scale on the axis.
Trusting a silhouette on non-spherical clusters. It is distance-based and inherits the same assumption k-means makes.
Forgetting to scale. Both metrics are distances.
Four ways to pick k, on data where you know the answer
The same four selection scores run twice: once on clean, well-separated blobs where they all agree, and once on overlapping ones where they do not. The disagreement is the useful part.
Check yourself
0 of …Answer without scrolling back up.
Why can inertia alone never choose k?
What does a negative silhouette score for a point mean?
Why does the silhouette have a genuine maximum while inertia does not?
Choosing k: Elbow and Silhouette
Inertia is the sum of squared distances from every point to the centroid of its cluster. K-means minimises it directly, so it is the obvious thing to look at.