Stand in a doorway taking in a hundred million measurements a second, then decide whether to walk forward. You need almost none of them. Those two or three numbers that matter have a name: latent space.
Adjust input and latent dimensions to see how compression changes.
The squeeze between a camera and a decision — and why everything after the bottleneck is all the rest of the system ever gets.
Stand in a doorway and look into a room. Your eyes are taking in something like a hundred million measurements a second. Now decide whether to walk forward.
For that decision you need almost none of them. How far away is the far wall? Is there a way through, and roughly where? That is two or three numbers, and everything else your eyes collected was, for this decision, decoration. Those two or three numbers have a name. They are the latent description of the scene — the word means hidden, or not directly visible. You will meet the phrase “latent space” in nearly every paper from here on.
Somewhere between the camera and the decision there has to be a squeeze, and whatever comes out the other side is all the rest of the system ever gets. That is the case for latents, and the same fact is the case against them.
The architecture is a narrow opening in the middle of a system. One half turns the picture into a short list of numbers. The other half takes the list and tries to rebuild the picture. Train the pair together and there is only one way to succeed: the short list has to carry what the picture was made of.
The two halves have names that every paper uses. The one that takes the picture in is the encoder, and the one that rebuilds it is the decoder. The narrow bit between them is the bottleneck. The pair together is an autoencoder: it encodes its own input and decodes it again.
Geoffrey Hinton and Ruslan Salakhutdinov made the case for the squeeze in 2006, in Reducing the Dimensionality of Data with Neural Networks. They put the paper in Science rather than a machine learning journal, which tells you who they wanted to win over. Everything since has been that argument with better kit.
A bottleneck creates pressure without naming the answer. Many equally short lists of numbers can rebuild the same pictures, and nothing makes one number mean distance and another mean lighting. You need a bias for that, in the architecture, the data or the task.
Irina Higgins and colleagues at DeepMind made the case for such a bias in 2016. A good representation, they argued, is one whose directions each mean one thing: turn one number and only the lighting changes. Pulling the hidden factors apart, one per axis, is called disentanglement.
Their beta-VAE a year later was an autoencoder with one dial added. The dial (the beta) turns the pressure on the bottleneck up, and past one the axes start lining up with nameable things — the turn of a face or the size of a shape. The cost is that pressing harder makes the pictures go blurry.
By 2018 the squeeze was inside a working agent. David Ha and Jürgen Schmidhuber published World Models that year, an agent that learned a driving game through a model of it. Every frame was crushed to thirty-two numbers, and the parts that predicted and chose never saw a pixel.
The same year Danijar Hafner and colleagues built PlaNet, in Learning Latent Dynamics for Planning from Pixels. It squeezed raw pixels the same way, then planned inside what came out, and never rebuilt a frame to decide anything. At decision time the short list was all it had — the bolder of the two designs.
In 2019 Francesco Locatello and colleagues closed the question Higgins had opened. With no labels and no built-in bias, they proved, disentanglement cannot be done: for any squeeze whose axes mean something, another draws the same pictures just as well with axes that mean nothing.
They then trained thousands of models, and which axes you got came down mostly to the random seed. The paper was Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. It took the best paper award at ICML that year. A lot of latent-space writing has not caught up with it.
Look at who was unaffected. Ha and Schmidhuber’s agent never needed its thirty-two numbers to mean anything a person could name, and neither did PlaNet’s planner. The squeeze carried on, because the agents built on it had only ever asked that it keep what the next step turns on.
Those numbers are coordinates. In a latent space that training has shaped well, they name a point with useful neighbours, directions and distances. Nothing guarantees that; a plain autoencoder can learn a twisted lookup table instead.
Once it has been shaped, you can do things there that make no sense in pixels. You can ask what is nearby, move in one direction and watch the world change smoothly in one way, or take two rooms and stand between them. Halfway between two pictures is a ghost, and halfway between two descriptions is a room.
This matters because prediction, planning and search all move somewhere you have not been, and if every step lands on nonsense none of them work. This is also why the noise in a variational autoencoder (VAE) is deliberate: it blurs where each picture lands so neighbouring points decode to similar things, making the space navigable.
Being small is not the goal in itself. A single number is very small and useless. What you want is that the things which matter are easy to get at, and the things that do not are gone. Yoshua Bengio, Aaron Courville and Pascal Vincent set that down in 2013, in Representation Learning: A Review and New Perspectives.
Three tests from that survey, in rough order of how often they fail:
Rebuilding the picture is a proxy, and proxies drift. The test asks whether the short list can redraw what the camera saw. What you wanted to know is whether it kept what the decision turns on.
The things that take up the most pixels are usually not the things that matter. A description scored on rebuilding pixels spends itself on texture and weather, and loses the small fast object you needed to avoid.
The second cost is subtler. Anything the squeeze drops is gone for good from that point on. Everything after the bottleneck works from the short list and nothing else. That choice is made early, by a part of the system that has no idea what task is coming, and it sets the ceiling on everything after it.
That is why the argument since has been about the middle of the pipe. PlaNet trained the squeeze together with the forward model, and LeCun’s JEPA line asks whether to rebuild pixels at all.
Say the answer out loud before you reveal it — recalling is what makes it stick.
What is the bottleneck in an autoencoder, and why does it force compression?
What did Locatello et al. (2019) prove about disentanglement without labels?
What is beta-VAE and what tradeoff does its beta parameter control?
Why do low-dimensional latent representations help planning and prediction?
Latent space = coordinates after the bottleneck. Autoencoder: Encoder → Bottleneck → Decoder. Beta-VAE adds pressure to disentangle axes. Locatello (2019): unsupervised disentanglement is impossible without labels. Ha & Schmidhuber: 32 latent dims from raw pixels. Good latent space: keeps what the future turns on, smooth (nearby = nearby), easy to roll forward.