Dilated Convolutions
Spread the same nine weights further apart and the receptive field grows without any downsampling and without any extra parameters.
Overview
The problem it solves
A network has two ways to grow its [receptive field](receptive_field.html): stack more layers, which is slow because growth is linear, or downsample, which is fast but throws away resolution.
For classification, throwing away resolution is fine — the answer is one label. For segmentation it is not, because the output has to be the same size as the input, and detail destroyed by pooling has to be reconstructed by something.
Dilated convolution is the third way. It grows the receptive field geometrically, adds no parameters, and does not reduce resolution at all.
Dilated Convolutions
Worth knowing
Dilated Convolutions
How to see further without downsampling, and the gridding artefact that comes with it.
What dilation does
Take a 3×3 kernel. Instead of reading nine adjacent pixels, read nine pixels spaced d apart, leaving gaps between them. The weights are unchanged; only where they are sampled from moves.
The effective size of a k×k kernel at dilation d is:
effective = k + (k - 1) * (d - 1)So a 3×3 at dilation 1 spans 3, at dilation 2 spans 5, at dilation 4 spans 9. Drag the dilation control and the readout confirms it: nine weights, spanning whatever you asked for.
That is the whole trick, and its appeal is that everything about the layer's cost is unchanged. Same weights, same number of multiplications, wider view.
Stacking
Drag the layers control up. Two dilated layers reach further than two ordinary ones, and the growth compounds, because each layer's reach is measured in the already-expanded units of the layer beneath it.
The standard arrangement uses a rising sequence — dilation 1, then 2, then 4, then 8 — which gives exponential receptive-field growth at constant resolution. A stack of four such layers sees a 33×33 region using 36 weights per channel pair and no pooling at all.
The gridding problem
There is a defect, and it follows directly from the gaps.
Stack several layers all at dilation 2, and some input positions are never sampled by any of them. The kernel taps land on even offsets at every level, so odd positions in between are simply not read. The output develops a checkerboard-like inconsistency, and the network is blind to fine detail that happens to fall in the gaps.
The fix is to vary the rate so that the gaps of one layer are covered by another. Rates like 1, 2, 5 or 1, 2, 3 are chosen so their sampling patterns interlock, and the *hybrid dilated convolution* literature is about picking such sequences. The simple version of the rule: do not repeat the same dilation rate several times in a row.
Where it is used
DeepLab made dilated convolution central to semantic segmentation, and its ASPP module runs several rates in parallel and concatenates them, giving the network several scales at once from one feature map.
WaveNet used dilation in one dimension over audio, where the receptive field has to span thousands of samples and downsampling would destroy the waveform.
Dense prediction generally — depth estimation, optical flow, anything whose output is an image — uses it for the same reason segmentation does.
The reach a dilated stack buys
The whole point is receptive field without downsampling, and the arithmetic shows how fast it grows. A dilated k×k kernel spans k + (k-1)(d-1), and stacked layers accumulate (k-1)·d at each step. Take four 3×3 layers at the standard rising rates 1, 2, 4, 8:
| layer | dilation | reach added | running receptive field |
|---|---|---|---|
| 1 | 1 | 2 | 3 |
| 2 | 2 | 4 | 7 |
| 3 | 4 | 8 | 15 |
| 4 | 8 | 16 | 31 |
Four layers reach a 31-pixel window — effectively 33×33 counting the kernel — using 4 × 9 = 36 weights per channel pair and *no pooling at all*. A stride-1 stack of ordinary 3×3 layers would need fifteen layers to see as far, because its reach grows by 2 per layer instead of doubling. That exponential growth at constant resolution is why DeepLab put dilation at the centre of segmentation. The catch is in those same gaps: repeat one rate and some input positions are never sampled by any layer, producing the gridding artefact, which is why the rates are varied (1, 2, 5) so their sampling patterns interlock.
Where dilation earns its place is dense prediction — segmentation, depth, optical flow — anything whose output must be the same size as the input, so resolution cannot be thrown away by pooling. DeepLab's ASPP module runs several rates in parallel and concatenates them, giving the network several scales from one feature map, and WaveNet used the one-dimensional version over audio, where the receptive field has to span thousands of samples. The one cost that is not free is memory: a network that never downsamples holds full-resolution activations at every layer, which is usually the binding constraint on a segmentation model, so the parameters are free but the memory is not.
The gridding fix is worth stating as a rule of its own: never repeat the same dilation several times in a row. Rates chosen so their sampling patterns interlock — 1, 2, 5, or 1, 2, 3 — cover the positions any single rate would skip, and the hybrid-dilated-convolution literature is largely about picking such sequences so that every input position is read by some layer in the stack.
Where it goes wrong
Repeating one rate. Gridding artefacts, and detail that falls in the gaps is never seen.
Using it where pooling would do. For classification, downsampling is cheaper and reduces computation. Dilation keeps the feature map large, and large feature maps cost memory at every layer.
Assuming it is free. The parameters are free; the memory is not. A network that never downsamples holds full-resolution activations throughout, which is often the binding constraint on segmentation models.
Skipping pixels to see further for the same price
A dilated convolution reads the same number of weights from a wider spread of positions. That one change buys receptive field growth that is exponential rather than linear -- and brings a gridding artefact that the fix for it is named after.
Check yourself
0 of …Answer without scrolling back up.
How wide does a 3x3 kernel at dilation 3 reach?
What causes gridding artefacts?
Why is dilation preferred over pooling in segmentation?
Dilated Convolutions
A network has two ways to grow its [receptive field](receptive_field.html): stack more layers, which is slow because growth is linear, or downsample, which is fast but throws away resolution.