Receptive Field
One pixel deep in a network sees a patch of the original image. Work out how big that patch is, and why stacking beats widening.
Overview
The question
Take one number in the output of a convolutional network's third layer. Change a pixel in the input image. Does that number change?
For most pixels, no. A convolution is local — each output depends only on a small window of its input — and stacking local operations gives an output that depends on a larger, but still bounded, region of the original image. That region is the unit's receptive field, and it is the honest answer to "what can this feature possibly be detecting".
A unit with a 7×7 receptive field cannot detect a face in a 224×224 photograph. It has never seen a face. It has seen a 7×7 patch, and whatever it responds to has to be visible in one.
Receptive Field
Worth knowing
k − 1 to the receptive field. Growth is linear in depth.Receptive Field
How much of the original image a single deep unit can actually see, and why the answer shapes architecture.
Counting it, one layer at a time
Drag the layers control above and watch the highlighted band widen as it moves down the diagram. Each row shows how many input pixels the layer above can be influenced by.
With stride 1 the rule is simple. A single unit sees 1 pixel of its own input. One k×k convolution makes that k. Another adds k - 1 more, because the window's centre already covers what the previous layer covered and each side extends by (k-1)/2. So:
r = 1
for each layer:
r = r + (k - 1)Three 3×3 layers: 1 → 3 → 5 → 7. The growth is linear in depth, and that is slow. A network of twenty 3×3 layers at stride 1 has a receptive field of 41 pixels — less than a fifth of a 224-pixel image.
Stride changes the arithmetic
Set the stride control to 2 and the widening accelerates sharply.
The reason is that stride changes the *spacing* between the positions a layer looks at, measured in original pixels. Call that spacing the jump. At stride 1 the jump stays 1 forever. At stride 2 it doubles every layer, so a step of one unit in layer three corresponds to a step of four pixels in the input.
r = 1; jump = 1
for each layer:
r = r + (k - 1) * jump
jump = jump * strideNow the additions themselves grow, and the receptive field expands geometrically rather than linearly. This is the real reason architectures downsample. Pooling and strided convolutions are usually explained as reducing computation, which they do — but the more important effect is that they are the only affordable way to get a deep unit to see the whole image.
Why 3×3 won everything
Two stacked 3×3 convolutions have the same 5×5 receptive field as one 5×5 convolution. Three stacked have the same 7×7 as one 7×7. So why not just use the big kernel?
Count the weights, per input/output channel pair:
| Arrangement | Receptive field | Weights |
|---|---|---|
| One 5×5 | 5×5 | 25 |
| Two 3×3 | 5×5 | 18 |
| One 7×7 | 7×7 | 49 |
| Three 3×3 | 7×7 | 27 |
The stack is cheaper, and it has a second advantage that matters more: there is a non-linearity between the layers. One 7×7 convolution is a single linear function of its 49 inputs. Three 3×3 convolutions with ReLUs between them is a composition of three linear functions separated by non-linearities, which can represent things a single linear map cannot.
VGG made this argument explicitly in 2014 and effectively ended large kernels in general-purpose vision architectures. Everything since is 3×3 stacks, with the occasional [1×1](one_by_one_convolutions.html) for channel work.
Effective versus theoretical
The number the formula gives is the *theoretical* receptive field: the set of pixels that could in principle affect the output.
The effective receptive field is smaller and softer. Contributions from the edge of the theoretical field pass through fewer paths than contributions from the centre, and the number of paths falls off roughly like a Gaussian. In practice a unit is strongly influenced by the middle of its field and barely influenced by the rim.
The practical consequence is that a network usually needs a theoretical receptive field noticeably larger than the objects it has to recognise, not merely equal to them. Dilated convolutions exist largely as a way to buy receptive field without buying downsampling, which matters when the output has to stay at full resolution — segmentation, most obviously.
Why two 3×3s beat one 5×5
Stacking small kernels reaches as far as one big kernel for fewer weights and more non-linearity, which is why 3×3 stacks won. The receptive field of a stride-1 stack grows by k-1 per layer, so:
| arrangement | receptive field | weights (per channel pair) |
|---|---|---|
| one 5×5 | 5×5 | 25 |
| two 3×3 | 5×5 | 18 |
| one 7×7 | 7×7 | 49 |
| three 3×3 | 7×7 | 27 |
Same reach, fewer weights — and, more importantly, a ReLU sits between the stacked layers, so three 3×3s are a composition of three non-linear functions where one 7×7 is a single linear map of its 49 inputs. That representational gap is the argument VGG made in 2014, and it ended large kernels. The table also explains why depth alone is not enough: at stride 1 the field grows only linearly, so twenty 3×3 layers still see just 41 pixels — under a fifth of a 224-pixel image. Growing it affordably needs stride, which makes the per-layer jump compound geometrically, or dilation, which is how segmentation networks reach far while keeping full resolution. And the *effective* field is smaller and softer than this theoretical one, falling off like a Gaussian from the centre, so a network usually needs a theoretical field noticeably larger than the objects it must see.
Where it goes wrong
Assuming depth alone is enough. Twenty stride-1 layers of 3×3 still only see 41 pixels. Without downsampling or dilation, a deep network can be blind to anything large.
Forgetting the input resolution. A receptive field of 100 pixels covers half a 224-pixel image and a twentieth of a 2000-pixel one. Resizing the input silently changes what the architecture can see.
Reading detection failures as a data problem. If the model consistently misses large objects, check the receptive field before collecting more images.
How much of the image one output pixel can see
A single neuron deep in a network responds to a patch of the original image. Computing that patch's size layer by layer explains most architectural choices -- kernel sizes, depth, dilation and pooling.
Check yourself
0 of …Answer without scrolling back up.
With stride 1, how does the receptive field grow as layers are added?
Why are three stacked 3x3 convolutions usually preferred over one 7x7?
Why is the effective receptive field smaller than the theoretical one?
Receptive Field
For most pixels, no. A convolution is local — each output depends only on a small window of its input — and stacking local operations gives an output that depends on a larger, but still bounded, region of the original image. That region is the unit's receptive field, and it is the honest answer to "what can this feature possibly be detecting".