Generative Adversarial Networks
Two networks with opposite goals. Move the generator's distribution and watch the discriminator's gradient appear and vanish.
Overview
The setup
Two networks, one loss.
The generator maps random noise to samples, trying to make them indistinguishable from real data. The discriminator takes a sample and estimates whether it is real, trying to be right.
They optimise the same objective in opposite directions. That makes it a minimax game, and it is why GAN training is unlike anything else: there is no loss that decreases, and a falling generator loss might mean the generator is improving or the discriminator is losing.
Generative Adversarial Networks
Worth knowing
preal / (preal + pfake). Where the distributions do not overlap, it is flat.Generative Adversarial Networks
A generator and a critic pushed against each other, and the specific way that arrangement fails.
The second panel is not learned
For fixed real and generated distributions, the optimal discriminator has a closed form:
D*(x) = p_real(x) / ( p_real(x) + p_fake(x) )The lower panel plots exactly that. It is not a trained network; it is what training drives towards, and its shape is what makes the failure mode legible.
Where the gradient goes
Drag the generator's mean far from the real distribution. The overlap in the readout collapses toward zero, and the discriminator's curve becomes flat — near 1 where the real data lives, near 0 where the fakes are, with a steep cliff between.
Flat means no gradient. The generator improves by following the slope of the discriminator's opinion, and if that opinion is constant everywhere the fakes live, there is nothing to follow. The better the discriminator, the worse this gets, which is a genuinely awkward property: your generator's learning signal degrades as its opponent improves.
Now bring the mean back. Where the distributions overlap the curve slopes, and that slope is the generator's gradient.
At full overlap the discriminator sits at 0.5 everywhere — it cannot tell them apart, which is the equilibrium the whole arrangement targets.
The failures
Vanishing gradients, as above. The non-saturating loss helps; the deeper fix was Wasserstein GAN, which replaces the classifier with a critic estimating earth-mover distance. That distance is informative even for disjoint distributions, so the gradient survives.
Mode collapse. The generator finds a narrow region that fools the discriminator and stays there. Shrink the generator's spread until it is a spike inside the real distribution: it is fooling the discriminator locally while representing almost none of the data. Nothing in the objective directly punishes this, because the discriminator judges samples one at a time and never sees the lack of variety.
Non-convergence. The two can cycle indefinitely, each undoing the other. Two-timescale learning rates, spectral normalisation and gradient penalties are all responses to it.
No usable progress metric. The loss does not say whether samples are improving. FID and Inception Score exist because of this, and both are proxies.
Against the alternatives
Sharp samples. GANs produce them where [VAEs](variational_autoencoders.html) blur, because nothing rewards averaging — a hedged sample is exactly what the discriminator catches.
No likelihood. A GAN cannot say how probable a given sample is, which rules out several uses.
Hard to train, as above.
[Diffusion models](diffusion_models.html) have largely displaced GANs for image generation, trading sampling speed for training stability. GANs remain competitive where inference must be a single forward pass, and the adversarial idea itself survives everywhere — in domain adaptation, in perceptual losses, in super-resolution.
Why a winning discriminator kills training
The vanishing-gradient failure is not vague; it falls straight out of the optimal discriminator D*(x) = p_real(x) / (p_real(x) + p_fake(x)). Put the real and fake distributions as unit Gaussians and slide them apart:
| separation of the means | D* at the fake's centre | D* at the real's centre |
|---|---|---|
| 0.5 (heavy overlap) | 0.47 | 0.53 |
| 4.0 (barely overlapping) | 0.0003 | 0.9997 |
When the two overlap, D* sits near 0.5 everywhere and its curve *slopes* through the region the fakes occupy — and that slope is the generator's entire learning signal. When they separate, D* saturates to 0.0003 where the fakes live: almost perfectly correct, and almost perfectly flat. A flat discriminator has no slope to follow, so the generator gets no gradient exactly when it is doing worst. This is the awkward heart of the original GAN — the better the discriminator, the less the generator can learn — and it is why Wasserstein GAN replaced the classifier with a critic estimating earth-mover distance, which stays informative even for distributions that do not overlap at all.
This is also why a GAN has no usable progress metric and why mode collapse goes unpunished. There is no single loss that decreases — a falling generator loss can mean the generator improved or the discriminator weakened — so FID and Inception Score exist as external proxies for sample quality. And because the discriminator judges samples one at a time, a generator that finds one convincing region and abandons the rest of the distribution is never penalised for the lack of variety; nothing in the objective ever sees more than a single sample at once, so the fix has to come from outside it, in minibatch discrimination or in the Wasserstein reformulation.
None of this makes GANs obsolete. They still win where inference must be a single forward pass, and the adversarial idea itself is everywhere — in domain adaptation, in perceptual losses, in super-resolution — even though diffusion models have taken over image generation by trading sampling speed for the training stability a minimax game can never quite offer.
The practical upshot for anyone training one: watch the samples, not the loss; keep the discriminator from winning outright, since a perfect discriminator gives no gradient; and check the variety of what the generator produces rather than the quality of any single sample, because individually convincing outputs are exactly what mode collapse leaves you with.
Where it goes wrong
Reading the loss curves. They mostly say who is winning, not whether the samples are good. Look at samples.
Letting the discriminator win. A perfect discriminator gives no gradient.
Ignoring mode collapse because the samples look fine individually. Check the variety, not the quality.
Expecting a likelihood. There is not one.
Two networks, one moving target
A GAN trains a generator against a discriminator that is itself learning. This runs that loop on 1-D data, then shows the two failure modes that make GANs hard.
Check yourself
0 of …Answer without scrolling back up.
Why does a very good discriminator stop the generator learning?
What is mode collapse?
Why are GAN samples sharper than a VAE's?
Generative Adversarial Networks
The generator maps random noise to samples, trying to make them indistinguishable from real data. The discriminator takes a sample and estimates whether it is real, trying to be right.