Depthwise Separable Convolution
Split a convolution into the part that mixes neighbours and the part that mixes channels. The saving is roughly a factor of nine.
Overview
What a full convolution is doing
A 3×3 convolution from 64 channels to 128 does two jobs at once. At every output position it looks at a 3×3 spatial neighbourhood, and it combines all 64 input channels. Each of the 128 output channels needs its own set of weights for all of that, so the count is:
k * k * Cin * Cout = 3 * 3 * 64 * 128 = 73,728The question that leads to separable convolutions is whether those two jobs have to be done together.
Depthwise Separable Convolution
Worth knowing
k² · Cin · Cout.k×k filter per input channel, mixing neighbours only.1×1 convolution, mixing channels only.k²·Cin + Cin·Cout, which for a 3×3 with many channels is about a ninth.Depthwise Separable Convolution
One convolution split into two, and why almost every model that runs on a phone is built from them.
Doing them one at a time
Depthwise. Run one k×k filter per input channel, and keep the channels apart. Channel 7 is convolved with filter 7 and produces channel 7 of the output. Neighbours are mixed; channels are not. Cost: k * k * Cin, so 9 * 64 = 576.
Pointwise. Follow it with a [1×1 convolution](one_by_one_convolutions.html), which combines all the channels at each position and has no spatial extent at all. Channels are mixed; neighbours are not. Cost: Cin * Cout, so 64 * 128 = 8,192.
Together: 8,768 against 73,728 for the same input and output shape. Drag the controls and the ratio in the readout stays close to 8–9× wherever you put them.
Where the ratio comes from
The saving is
k^2 * Cin * Cout
--------------------- = 1 / ( 1/Cout + 1/k^2 )
k^2 * Cin + Cin * CoutWhen Cout is large the 1/Cout term nearly vanishes and the ratio approaches k^2 — nine for a 3×3. Set the kernel control to 5 or 7 and the saving rises toward 25 and 49, which is why separable convolutions matter more the larger the kernel.
Set output channels to 8 while leaving input at 64 and the ratio falls, because with few output channels the pointwise step is no longer cheap relative to the whole. The technique earns its keep in wide layers, which is where the cost was.
The cost of the saving
It is not free, and pretending otherwise is the usual mistake.
A full convolution can learn any function of a 3×3×Cin neighbourhood. The separable pair cannot: it is restricted to functions that factor into a spatial part and a channel part. That is a strictly smaller family, and on the same architecture a separable model usually reaches slightly lower accuracy per layer.
What makes it worth doing is that the saving is much larger than the loss. With eight times fewer parameters you can afford more layers, wider layers, or a model that fits on the device at all — and the resulting network usually beats the full-convolution one at equal cost.
Where it is used
MobileNet is built almost entirely from these pairs, and was the paper that made the technique standard for on-device vision.
Xception applied the same argument to Inception, arguing that Inception modules were already approximating a separable convolution and that the extreme version worked better.
EfficientNet uses inverted residual blocks that expand channels with a 1×1, do the spatial work depthwise, and project back down — the same decomposition with an expansion around it.
A note on speed
Parameter count and wall-clock time are not the same thing. A depthwise convolution does very little arithmetic per byte of memory it touches, so it is memory-bound, and hardware optimised for dense matrix multiplication does not reach anything like its peak throughput on one.
An eight-times parameter reduction is therefore often two or three times faster in practice rather than eight. Still worth having, and worth measuring rather than assuming.
Where the saving comes from, and how big
Splitting a convolution into a depthwise step (mix neighbours, keep channels apart) and a pointwise 1×1 (mix channels, ignore neighbours) costs a fraction of doing both at once. The ratio is 1 / (1/Cout + 1/k²), which approaches k² as the output grows wide:
| kernel | separable saving (Cout=128) | approaches |
|---|---|---|
| 3×3 | 8.4× | 9 |
| 5×5 | 20.9× | 25 |
| 7×7 | 35.4× | 49 |
So a 3×3 to 128 channels drops from 73,728 weights to 8,768, and the larger the kernel the more it saves, which is why separable convolutions matter most in wide layers with big kernels. The cost is representational: the separable pair can only learn functions that factor into a spatial part and a channel part, a strictly smaller family than a full convolution, so it reaches slightly lower accuracy per layer — but the saving is so much larger than the loss that the freed budget buys more or wider layers that win overall. And the parameter saving does not translate one-to-one into speed: a depthwise convolution does little arithmetic per byte it touches, so it is memory-bound, and an eight-times parameter cut is often only two or three times faster in practice.
The technique earns its keep in wide layers, which is where the cost was, and it underpins the on-device architectures: MobileNet is built almost entirely from these pairs, Xception argued Inception modules were already approximating them and pushed the idea to its limit, and EfficientNet wraps them in inverted residual blocks that expand channels with a 1×1, do the spatial work depthwise, and project back down. Two failure modes are worth naming. Using a separable convolution in the first layer saves almost nothing — three input channels leave little to factor — and costs accuracy where the model can least afford it; and the depthwise and pointwise steps need a non-linearity between them, or the pair collapses toward a single linear map rather than the two-stage decomposition it is meant to be.
Where it goes wrong
Expecting the speed-up to match the parameter count. Measure it.
Using it in the first layer. With three input channels there is almost nothing to save, and the restriction costs accuracy where the model can least afford it.
Forgetting the non-linearity. The depthwise and pointwise steps need an activation between them, or the pair is closer to a single linear map than the two-stage decomposition it is supposed to be.
Splitting one convolution into two cheaper ones
A normal convolution mixes space and channels at once. Doing those separately costs a fraction as much, which is the whole idea behind MobileNet -- and this measures both the saving and what it gives up.
Check yourself
0 of …Answer without scrolling back up.
What does the depthwise step mix?
Why does the saving approach k squared for wide layers?
Why is the wall-clock speed-up usually smaller than the parameter saving?
Depthwise Separable Convolution
A 3×3 convolution from 64 channels to 128 does two jobs at once. At every output position it looks at a 3×3 spatial neighbourhood, and it combines all 64 input channels. Each of the 128 output channels needs its own set of weights for all of that, so the count is: