Defining the Labels
Imagine you're training a model to classify the sentiment of a tweet as 'Negative', 'Neutral', or 'Positive'. How you tell the model the "correct" answer for each tweet is where labelling strategy comes in.
Hard Labelling (One-Hot Encoding)
This is the most common approach. You are 100% certain about the class. The correct class gets a value of 1, and all other classes get a 0. It's a "winner-takes-all" method.
Example Vector: For a 'Positive' tweet, the hard label is [0, 0, 1]. This tells the model, "This tweet is positive, and nothing else."
Soft Labelling (Probabilistic)
This method acknowledges that the world is messy and some data points are ambiguous. Instead of picking one class, you assign a probability distribution across all classes.
Example Vector: For a tweet that is mostly positive but has a hint of neutrality, the soft label might be [0.1, 0.2, 0.7]. This tells the model, "I'm pretty sure this is positive, but there's a small chance it's neutral or even slightly negative."
The two ways to say what the answer is
A hard label is a single confident claim: this image is a cat. Written as a vector for three classes, that is [0, 1, 0] — all the probability mass on one class, none anywhere else.
A soft label spreads the claim: [0.05, 0.90, 0.05], or for a genuinely ambiguous image, [0.30, 0.60, 0.10]. It says what the answer probably is and how sure anyone is.
The difference sounds cosmetic and is not, because of what it does to the loss. With cross-entropy loss and a hard label, the model is pushed to output 1.0 for the true class and 0.0 for everything else. It can never reach those values — softmax outputs are strictly between 0 and 1 — so the pressure never stops. The model keeps growing its weights to push the winning logit further from the others, long after the classification itself is correct.
That endless pressure produces a specific and well-documented failure: a network that is 99.9% confident about images it gets wrong. The confidence has stopped tracking the probability of being right.
Where soft labels come from
Soft targets are not something you invent by hand. Four sources supply them naturally:
- Multiple annotators. Ask seven radiologists whether a scan shows a tumour and five say yes. The honest label is 0.71, not 1.0. Averaging the votes into a hard majority throws away exactly the information about which cases are difficult.
- Label smoothing. A mechanical version: take the hard label and move a small amount ε (typically 0.1) off the true class, spread evenly across the others. [0, 1, 0] becomes [0.05, 0.90, 0.05]. One line of code, and it is standard in modern image classification training recipes.
- Knowledge distillation. Train a large model, then use its full output distribution as the target for a small model. The large model's "this is a cat, but there is something dog-ish about it" carries far more information per example than the word "cat", which is why a distilled small model beats the same model trained on hard labels.
- Mixup and similar augmentations. Blend two training images by 70/30 and blend their labels by 70/30 too. The soft label is the honest description of what the blended image contains.
What it buys you, and what it costs
| Aspect | Hard labels | Soft labels |
|---|
| Confidence calibration | Typically overconfident | Much closer to honest |
| Robustness to label noise | Poor — one wrong label is learned absolutely | Better — error is diluted |
| Information per example | One class name | A full distribution over classes |
| Peak training accuracy | Higher | Slightly lower by design |
| Cost to produce | Cheap | More annotators, or a teacher model |
The middle row is the practical headline. If 3% of your labels are wrong — which is normal for large annotated datasets — hard labels demand that the model memorise those mistakes perfectly. Soft labels leave room for the model to remain unconvinced.
The fourth row is the honest cost. Label smoothing usually lowers training accuracy a little, because you have explicitly told the model not to be certain. Test accuracy and calibration generally improve, which is the trade you wanted.
One caveat worth knowing: label smoothing tightens the clusters a network forms for each class, which is good for classification but has been shown to hurt when you later want to distil from that network, or use its embeddings for retrieval. Smooth the student, think twice about smoothing the teacher.
What you throw away when you take argmax
A hard label is a probability with the uncertainty deleted. Keeping the probability lets you abstain, rank, and pass calibrated confidence downstream.
Things to try
Select different tweets from the list to see how these labelling strategies handle different scenarios.
- The Clear-Cut Case: Select the second tweet: "Absolutely stunning. Every detail is perfect. Highly recommend!" This is unambiguously positive. Notice that the hard label [0, 0, 1] and the soft label [0.01, 0.04, 0.95] are very similar. In such clear cases, both methods convey almost the same information. The model is strongly encouraged to predict 'Positive'.
- The Ambiguous Case: Now, select the first tweet: "The design is gorgeous, but the software is a total nightmare to navigate." This tweet contains both positive ('gorgeous design') and negative ('nightmare software') elements.
- The Hard Label is forced to choose one winner. Here, it's classified as 'Negative' ([1, 0, 0]), completely ignoring the positive aspect. This forces the model to be overconfident and lose valuable information.
- The Soft Label, however, captures the nuance perfectly with a vector like [0.65, 0.1, 0.25]. It tells the model: "This is mostly negative, but there's a significant positive component too."
- The Neutral-Leaning Case: Select the last tweet: "Shipping was late, but the box was packed really well." Again, we see a mix of sentiments. The hard label might classify this as 'Neutral' ([0, 1, 0]), but the soft label [0.4, 0.45, 0.15] provides a much richer signal, indicating it leans neutral but has a strong negative element ('late shipping') and a minor positive one ('packed well').
Why Use Soft Labelling?
While hard labelling is simpler, soft labelling offers several key advantages, a technique often referred to as Label Smoothing:
- Better Generalization: By preventing the model from becoming overconfident in its predictions for the training data, soft labelling helps it generalize better to new, unseen data. It acts as a form of regularization.
- Robustness to Noise: Real-world datasets often contain mislabeled examples. Soft labels can account for this uncertainty, making the training process more stable and less susceptible to noisy data.
- Capturing Inter-Class Relationships: Soft labels can implicitly teach the model about the relationships between classes. For example, it can learn that 'sarcastic' is closer to 'negative' than it is to 'positive'.
- Knowledge Distillation: Soft labels are the foundation of knowledge distillation, where the probabilistic outputs of a large, complex model are used as soft targets to train a smaller, more efficient model.
Reading a model's confidence honestly
Once you care about soft labels you inevitably start caring about calibration: does a predicted 0.8 mean the model is right about 80% of the time?
The way to check is a reliability diagram. Bucket the predictions by confidence — 0.5–0.6, 0.6–0.7, and so on — and for each bucket plot average confidence against actual accuracy. A perfectly calibrated model traces the diagonal. A typical modern network traces a curve sagging below it: the 0.9 bucket is right only 75% of the time.
The single number that summarises this is expected calibration error, the average gap between confidence and accuracy across the buckets.
Two standard repairs, both applied after training on a held-out set:
- Temperature scaling. Divide the logits by a single learned number T before the softmax. T > 1 softens every prediction. One parameter, does not change any prediction's ranking, and it fixes most of the problem. This is the default choice.
- Isotonic regression or Platt scaling. More flexible mappings from raw score to calibrated probability, useful for non-neural models. Isotonic needs more data to avoid overfitting the calibration set itself.
Calibration matters wherever the number is used rather than just the argmax: a medical triage threshold, an expected-value calculation, or a downstream system that combines several models' probabilities.
Questions people ask
Should I always use label smoothing? For large image and text classifiers, it is close to a free improvement at ε = 0.1. For small datasets, calibrated-probability-critical tasks, or when you plan to distil from the model, test it rather than assuming.
Do soft labels need a different loss function? No. Cross-entropy already accepts a full target distribution; the hard-label case is just the special case where one entry is 1. Most frameworks accept either.
How do I combine several annotators' opinions? The simple average is a reasonable start. Better methods weight annotators by their measured reliability, which requires overlapping assignments so agreement can be estimated.
Does this apply to regression? The label-smoothing form does not, but the idea does: predicting a distribution rather than a point value — quantile regression, or a model that outputs a mean and a variance — is the regression version of admitting uncertainty.
Can soft labels hide a genuinely bad model? They can flatter the loss, so keep reporting accuracy and a proper calibration measure alongside it. Low confidence everywhere is not the same as being well calibrated.
Recap in one screen
- A hard label asserts one class; a soft label gives a probability across classes.
- Training on hard labels with cross-entropy pushes a model towards unwarranted certainty.
- Soft targets come from multiple annotators, label smoothing, distillation, or mixup-style augmentation.
- They usually cost a little training accuracy and buy calibration and noise tolerance.
- Check calibration with a reliability diagram; fix it with temperature scaling.