Autoencoders
Force a signal through a narrow gap and out again. What survives is what the network decided mattered.
Overview
The apparently useless task
An autoencoder has two halves. The encoder maps an input to a smaller representation; the decoder maps it back. The loss is the difference between the output and the input.
Copy the input to the output. Stated like that it sounds like a null task — the identity function scores perfectly.
Except it cannot, because the representation in the middle is smaller than the input. The network has to throw information away, and the training objective forces it to throw away the least costly information. What survives the bottleneck is what the data was mostly made of.
Autoencoders
Worth knowing
Autoencoders
A network trained to reproduce its input, and why that pointless-sounding task is useful.
Watch the squeeze
The grey line is the input; the orange line is what comes back out.
At a latent size of 1 the reconstruction keeps only the largest structure and misses everything else. Raise it and the copy tightens. Past a certain point extra dimensions buy almost nothing, because the remaining detail is noise rather than signal.
That flattening is the useful observation: it says roughly how many numbers the data really needs, which is its intrinsic dimensionality.
The relationship to PCA
A linear autoencoder trained with squared error finds the same subspace as [PCA](../machine_learning/pca.html). Not something similar — provably the same span, though the individual directions need not be the principal components in order.
So a linear autoencoder is PCA with extra steps. What makes them worth having is non-linearity: with non-linear activations the encoder can learn a curved manifold, which no linear projection can follow. An image dataset lives on a wildly curved surface in pixel space, and that is the case where an autoencoder earns its keep.
Denoising
Raise the noise control. Now the input is corrupted and the target is still the clean signal.
The reconstruction error against the *noisy* input goes up, and against the *clean* signal it goes down. The readout gives both.
The reason is the bottleneck again. Noise is, by construction, the part of the signal with no structure to compress, so a narrow code cannot afford to store it. Forcing the network to reconstruct through the bottleneck forces it to discard exactly what noise is.
Denoising autoencoders make this the training procedure: corrupt the input deliberately, ask for the clean version. It also prevents the degenerate solution where a wide code just learns the identity, which is why the trick appears even when denoising is not the goal.
What they are used for
Dimensionality reduction where the structure is non-linear.
Anomaly detection. Train on normal data; anything that reconstructs badly is unlike what was seen. It works because the model has only learned to compress one kind of thing.
Pretraining. Historically the standard way to initialise deep networks before better initialisation and normalisation made it unnecessary.
Generation — but not directly. A plain autoencoder's latent space has gaps, so a random code usually decodes to nothing meaningful. Fixing that is what [the VAE](variational_autoencoders.html) is for.
The elbow that reveals intrinsic dimension
The useful thing an autoencoder measures is *how many numbers the data really needs*, and it shows up as an elbow in reconstruction error against latent size. On data that genuinely lives in d dimensions with a little noise on top, widening the bottleneck past d buys almost nothing:
| latent size | reconstruction error |
|---|---|
| 1 | large — only the biggest structure survives |
| up to the true d | falling steeply, each dimension earns its place |
| beyond d | flat — the remaining detail is noise |
The point where the curve flattens is the data's intrinsic dimensionality, and it is the same number [PCA](../machine_learning/pca.html) would find — a linear autoencoder with squared-error loss provably recovers the PCA subspace. The reason to reach for an autoencoder instead is the non-linear case: real data (an image dataset, say) lies on a curved manifold in a high-dimensional space, and a straight projection cannot follow the curve where a non-linear encoder can. That is also why the reconstruction error reads as pixel agreement rather than perceptual quality, and why autoencoder outputs look softened: the bottleneck keeps what is common to the data and spends nothing on the specific sharp detail that distinguishes one example from its neighbours.
The denoising variant turns the same bottleneck into a training procedure: corrupt the input deliberately and ask for the clean version back. Because noise is by construction the part of a signal with no structure to compress, a narrow code cannot afford to store it, so reconstructing through the bottleneck forces the network to discard exactly what noise is. That also removes the degenerate escape where a too-wide code simply learns the identity, which is why the trick shows up even when denoising is not the goal — and it is the same instinct behind using an autoencoder for anomaly detection, where a model trained only on normal data reconstructs anything unfamiliar badly, and the reconstruction error becomes the anomaly score.
None of the latent dimensions mean anything individually, either: they are not ordered or disentangled unless something in the training made them so, which is a large part of what separates a plain autoencoder from the variational and beta-weighted variants built to encourage exactly that structure.
Where it goes wrong
A bottleneck that is too wide. With enough capacity the network learns the identity and nothing is discovered. Constrain it, or use denoising, or add a sparsity penalty.
Expecting a usable generative model. Sample from a plain autoencoder's latent space and you will mostly get nonsense.
Reading the reconstruction error as a quality score. It measures pixel agreement, not perceptual similarity, which is why autoencoder outputs look blurred.
Assuming the latent dimensions mean anything individually. They are not ordered or disentangled unless something in the training made them so.
Train one, then look at what the bottleneck kept
An autoencoder learns to reproduce its own input through a narrow layer. Training one on data with a known hidden structure shows what a bottleneck forces it to discover -- and where PCA does the job better.
Check yourself
0 of …Answer without scrolling back up.
Why is training a network to copy its input not a null task?
What does a linear autoencoder with squared error find?
Why does a bottleneck remove noise?
Autoencoders
An autoencoder has two halves. The encoder maps an input to a smaller representation; the decoder maps it back. The loss is the difference between the output and the input.