Anchor Boxes
Detectors do not invent boxes. They start from a fixed set at every position and learn how to nudge them.
Overview
Why detectors do not just predict boxes
A detector must output an unknown number of boxes at unknown positions and sizes. Neural networks are much better at producing a fixed-size output than a variable-length one, and much better at making a small correction than at producing a coordinate from nothing.
Anchors convert the hard problem into the easy one. Place a fixed set of reference boxes at every position of a feature map, and ask the network two things per anchor: is there an object here, and how should this box be adjusted to fit it?
The output is now a fixed-size tensor, and the regression target is a small offset instead of an absolute position.
Anchor Boxes
Worth knowing
Anchor Boxes
The prior that turns 'find every object' into 'adjust these boxes slightly', and the assignment rule that trains it.
What the visualisation shows
The dashed grid is the feature map: one set of anchors per cell. The orange rectangle is a ground-truth object. Drag the cell control to move the anchor set around, and the size control to change the scale.
Three anchors are drawn at each position, one per aspect ratio — wide, square and tall — because objects come in shapes and one square box fits few of them. Real detectors use three scales as well, so nine anchors per position is a common arrangement.
The readout gives the best IoU against the ground truth and what that IoU means for training.
The assignment rule
Every anchor gets a label before training, decided by IoU:
| IoU with the best ground-truth box | Label |
|---|---|
| Above ~0.5 | positive — predict this object, regress the offset |
| Below ~0.3 | negative — predict background, no box loss |
| In between | ignored — contributes nothing to the loss |
The ignore band exists because those anchors are genuinely ambiguous, and forcing them either way teaches the network something untrue.
The page opens on a positive anchor. Drag the size control down and the readout walks through all three labels — positive, then ignored, then negative — without the ground-truth box moving at all. That transition is where an anchor stops being useful for an object, and it is why the anchor set has to match the objects you expect.
The imbalance nobody warns you about
A detector places tens of thousands of anchors on an image. An image with three objects makes almost every one of them negative — ratios of a thousand to one are normal.
Trained naively, the classification loss is dominated by easy background anchors and the model learns to say "background" very confidently and nothing else. Three responses became standard:
Hard negative mining — keep only the worst-scoring negatives so the ratio stays around 3:1.
Focal loss — down-weight examples the model already gets right, so the easy negatives stop dominating. This is what RetinaNet introduced, and it is why one-stage detectors caught up with two-stage ones.
Two-stage detection — a region proposal network filters down to a few hundred candidates before classification, which sidesteps the imbalance.
Choosing the anchor set
Anchors are a prior, and a wrong prior is expensive. Scales and ratios tuned for everyday photographs do badly on aerial imagery, where objects are small and often square, or on text detection, where boxes are extremely wide.
YOLOv2 introduced picking them by running k-means over the ground-truth box shapes in the training set, using IoU as the distance. That replaced a hand-tuned guess with a fitted one, and it is still the sensible default when a dataset's objects are unusual.
Anchor-free detectors
FCOS, CenterNet and DETR drop anchors entirely. FCOS predicts, for each point inside an object, the four distances to the box edges. CenterNet predicts object centres as a heatmap. DETR predicts a fixed set of boxes directly and matches them to ground truth with a Hungarian assignment.
They remove the anchor hyperparameters, which is a real simplification. What they do not remove is the assignment problem — deciding which prediction is responsible for which object — which is simply solved differently.
The assignment, by IoU
Every anchor is labelled before training by its IoU with the nearest ground-truth box, and working three cases shows the bands. Anchor at [0,0,4,4] against three possible objects:
| ground-truth box | IoU | label |
|---|---|---|
| [0,0,4,5] | 0.80 | positive — regress the offset |
| [1,1,5,5] | 0.39 | ignored — too ambiguous to train on |
| [3,3,7,7] | 0.03 | negative — predict background |
The ignore band (roughly 0.3-0.5) exists because forcing an ambiguous anchor either way teaches the network something untrue. The deeper problem the numbers expose is imbalance: a detector places tens of thousands of anchors, and an image with three objects makes almost all of them negative — ratios of 1000:1 are normal, so a naively trained model just says "background" very confidently. That single fact drove three fixes — hard-negative mining to hold the ratio near 3:1, focal loss to down-weight the easy negatives (which is what let one-stage detectors catch up), and two-stage proposal networks to filter first. And because anchors are a *prior*, a set tuned for photographs fails on aerial or text data, which is why YOLOv2 fits them by k-means over the training boxes.
Anchor-free detectors — FCOS, CenterNet, DETR — drop the reference boxes entirely: FCOS predicts, for each point inside an object, the four distances to the box edges; CenterNet predicts centres as a heatmap; DETR predicts a fixed set of boxes and matches them to ground truth with a Hungarian assignment. They remove the anchor hyperparameters, a real simplification, but not the underlying assignment problem of deciding which prediction owns which object — that is simply solved differently. And too many anchors is its own cost: each one is memory and computation at every position of every feature map, and past a point they overlap so heavily that extra ones add nothing but expense.
Where it goes wrong
Default anchors on unusual data. Check the IoU distribution of your anchors against your ground-truth boxes before training.
Ignoring the imbalance. Without focal loss or mining, the model predicts background everywhere and the loss looks fine.
Too many anchors. Each one costs memory and computation at every position of every feature map, and past a point they overlap so much that extra ones add nothing.
Guessing boxes so the network only has to correct them
An anchor is a box the detector guesses before seeing the image, so the network predicts a small correction instead of raw coordinates. This builds an anchor grid, measures which real objects it can and cannot reach, and finds that the thing limiting coverage is not always the thing you would reach for first.
Check yourself
0 of …Answer without scrolling back up.
What does the network predict for each anchor?
Why do anchors with middling IoU get ignored rather than labelled?
Why does focal loss exist?
Anchor Boxes
A detector must output an unknown number of boxes at unknown positions and sizes. Neural networks are much better at producing a fixed-size output than a variable-length one, and much better at making a small correction than at producing a coordinate from nothing.