Semantic, Instance and Panoptic Segmentation
Three tasks that all colour in pixels, and differ in exactly two questions: do objects get separate ids, and does the background count.
Overview
Two questions
Every segmentation task colours in pixels. What differs is the answer to two questions:
- Do two objects of the same class get separate identities?
- Does the background get labelled?
Switch between the three settings above and watch the two discs and the background. That is the entire taxonomy, visible in one scene.
| Separate ids | Background labelled | |
|---|---|---|
| Semantic | no | yes |
| Instance | yes | no |
| Panoptic | yes | yes |
Semantic, Instance and Panoptic Segmentation
Worth knowing
Semantic, Instance and Panoptic Segmentation
Three tasks, one scene, and the two questions that separate them.
Things and stuff
The vocabulary that makes this coherent is things against stuff.
Things are countable: people, cars, dogs, bottles. Asking "how many" makes sense, so instances make sense.
Stuff is uncountable: sky, road, grass, water. "How many skies" is not a question. Stuff has extent but not instances.
Semantic segmentation treats everything as stuff — it only ever assigns classes. Instance segmentation only handles things — it needs something to count. Panoptic segmentation covers both, which is why it needed a name of its own.
Semantic
Every pixel gets a class label. Two overlapping cars are one connected region labelled "car", and nothing in the output says there were two.
The standard architectures are fully convolutional: U-Net, DeepLab, SegFormer. The metric is mean IoU per class. This is the right task when what matters is area rather than count — how much of this field is diseased, which pixels are road, what fraction of the scan is tumour.
Instance
Every object gets its own mask and identity, and pixels belonging to no object are left unlabelled.
Mask R-CNN is the canonical approach: detect boxes first, then predict a mask inside each. That ordering has a consequence — masks can overlap, and a pixel can belong to two instances, because nothing forces a single answer per pixel.
This is the right task when counting matters: how many cells, how many people, which pallet is which.
Panoptic
Every pixel gets exactly one class, and pixels belonging to things also get an instance id. Stuff regions get a class and no id.
The word means "everything visible", and the point of the 2018 paper that named it was that the semantic and instance communities had been solving halves of the same problem with different metrics and different architectures.
The constraint that makes it harder than running both is exactly one label per pixel. Instance methods produce overlapping masks and semantic methods ignore instances, so combining them requires resolving conflicts. Panoptic FPN and Mask2Former do it in one model instead.
Its metric, PQ, multiplies a segmentation quality term (mean IoU over matched segments) by a recognition quality term (an F1 over whether segments were matched at all), which prevents a method from scoring well by getting the pixels roughly right while missing objects entirely.
Choosing
Ask what the output is for.
Area, no counting — semantic. Land cover, tumour extent, drivable surface.
Counting or tracking individuals — instance. Cell counting, retail stock, people in a queue.
A complete scene description — panoptic. Autonomous driving needs both "where is the road" (stuff) and "which car is which" (things), and needs them consistent.
PQ, worked
Panoptic quality is the one metric that spans both halves, and seeing it computed shows why it resists the easy failures. It factors into segmentation quality times recognition quality: SQ is the mean IoU over *matched* segments, RQ is an F1 over whether segments were matched at all, TP / (TP + ½FP + ½FN).
Take a scene where 8 predicted segments matched ground truth at mean IoU 0.80, 1 prediction matched nothing (a false positive), and 2 ground-truth objects were missed (false negatives):
| term | value |
|---|---|
| SQ (mean IoU of matches) | 0.80 |
| RQ = 8 / (8 + 0.5 + 1) | 0.84 |
| PQ = SQ × RQ | 0.67 |
The two factors guard different failures. A method that segments the pixels it finds beautifully but misses half the objects has a high SQ and a low RQ, so PQ falls; one that finds every object but traces them sloppily has the reverse. Only getting both right — the pixels *and* the object count — scores well, which is exactly the point of a metric built to end the split between the semantic community, who measured pixels with mIoU, and the instance community, who measured objects. It is also why a semantic mIoU and a panoptic PQ are not comparable: they are answering different questions, and only one of them can tell two touching cars apart.
The taxonomy behind the metrics is two yes/no questions: do two objects of the same class get separate identities, and does the background get labelled. Semantic says no and yes — every pixel gets a class, two overlapping cars are one "car" region — so it is right when area matters and counting does not: tumour extent, drivable surface, land cover. Instance says yes and no — each object gets its own mask, background is left unlabelled — which is what counting needs, and because Mask R-CNN predicts a mask per detected box, its masks can overlap. Panoptic demands exactly one label per pixel *and* an instance id for the countable things, which is the constraint that makes it harder than running the other two and merging their outputs, since that merge needs a conflict rule.
The vocabulary that makes the split coherent is things against stuff: things are countable — people, cars — so instances make sense, while stuff is uncountable — sky, road — and has extent but no instances. Semantic treats everything as stuff, instance handles only things, and panoptic covers both, which is exactly why it needed a name of its own.
Where it goes wrong
Using semantic segmentation to count. Two touching objects are one region. Post-hoc connected components will merge them.
Expecting instance segmentation to label the background. It does not; that is not an oversight.
Comparing mIoU with PQ. Different metrics on different tasks.
Ignoring the one-label constraint. Merging separate semantic and instance outputs into a panoptic one requires a conflict rule, and the rule affects the score.
Three tasks that all look like colouring in
Semantic, instance and panoptic segmentation produce pictures that look alike and answer different questions. This builds the same scene under all three, computes their metrics, and shows the specific query each one cannot answer.
Check yourself
0 of …Answer without scrolling back up.
What is the difference between semantic and panoptic segmentation?
Why does instance segmentation not label the background?
What makes panoptic harder than running a semantic and an instance model together?
Semantic, Instance and Panoptic Segmentation
Semantic segmentation treats everything as stuff — it only ever assigns classes. Instance segmentation only handles things — it needs something to count. Panoptic segmentation covers both, which is why it needed a name of its own.