LeCun’s argument for predicting embeddings instead of pixels: the leaf problem, the ghosting problem, and why JEPA avoids both by never predicting the raw picture.
Compare pixel prediction vs JEPA embedding prediction. Increase ambiguity to see the ghosting effect.
Predicting embeddings instead of pixels: why the slogan “stop predicting pixels” has a narrower but correct argument behind it.
If you have followed the world model argument online, you will have run into a slogan: stop predicting pixels. It comes from Yann LeCun. He was Meta’s chief AI scientist at the time, and he is a Turing Award winner. He is also one of the people who made convolutional networks work, the kind of network that reads images.
In 2022 he published A Path Towards Autonomous Machine Intelligence — a position paper, meaning a case for how machines should learn rather than a result. One charge in it was aimed at the reflex then governing learned world models: when in doubt, predict the next picture.
LeCun’s smaller charge starts with a camera pointed at a road. Ask what the next second looks like: a leaf at the edge of the frame will move, and which way is unknowable. It is also the least useful fact on offer, since nothing you might do about the road depends on it. A model asked to predict the picture has to predict that leaf anyway.
The leaf is made of pixels, and pixels are what the model is marked on. Multiply that by every leaf, every ripple, every flicker of light on wet tarmac. Most of the model’s capacity has gone to things that are hard to predict and do not matter. That is the leaf problem.
The larger charge is a car coming to a junction. It will turn left or right. You do not know which, and no amount of staring at the current frame will tell you. Now ask a deterministic model — one that gives a single answer every time — for one next picture. Score it by squared pixel error. The best answer in the maths is the conditional average of the two turns: the left and right futures blended at their odds, rather than either one on its own.
The result scores better than either real future. A model that predicted the left turn would be punished half the time. The ghost in the middle is punished a little all the time, and under squared error a little all the time wins. That training signal asks for the mean rather than for a plausible future. When the future is open, the mean is a smear that never occurs.
One complication: the pixel camp had dealt with the ghost by 2017, five years before the position paper. SV2P — Stochastic Variational Video Prediction — from Mohammad Babaeizadeh and colleagues, and a similar model from Emily Denton and Rob Fergus, both learn a spread of futures and draw one at random. The trick is a hidden variable, a few numbers standing in for whatever the frame cannot tell you, such as which way the car turns. Each draw can be sharp even when the model has no idea which turn will happen.
That kills the blur objection, though not the rest of the case. Sampling fixes the form of the output. It does not make useless texture cheap, make rare futures well estimated, or tell a decision maker which possible future matters. Stochastic generation keeps the pixel target; embedding prediction changes it.
LeCun’s proposal is to stop predicting the picture. Encode what you can see into a compact description — an embedding, a short list of numbers that stands in for something bigger. Similar things land near each other in that list. Encode the part you are predicting into one too. Then train the model to predict the second description from the first. Nothing is ever decoded back into pixels.
I-JEPA is the simplest example: a 2023 image model from Mahmoud Assran and colleagues in LeCun’s group at Meta. Hide part of the picture and predict the embedding of the missing piece. V-JEPA 2, from Meta AI in 2025, does the same for video — trained first on plain video with no actions, then trained again on a robot’s actions for control.
Two things follow. The leaf problem goes away, because the description never had to record which way the leaf went. If a detail does not survive the encoding, nothing downstream is graded on it. The ghost shrinks too, because you are no longer averaging pictures. The descriptions of a car turning left and one turning right can share most of their entries.
Predicting pixels has one large virtue that JEPA gives up. The target is fixed, and the model cannot make the next frame easier to predict. Predict an embedding and that stops being true. The target now comes from the encoder, the network that turns pictures into descriptions — and that network is also being trained. It can get a perfect score while learning nothing, by making every description the same. That failure is called collapse.
Most of the difficulty in these methods is preventing collapse, and the fixes form a lineage. BYOL, from Jean-Bastien Grill and colleagues at DeepMind in 2020, keeps a second copy of the encoder that updates slowly — a target encoder that trails the one being trained. VICReg, from Adrien Bardes, Jean Ponce and LeCun in 2021, goes the explicit route: it penalises descriptions whose parts have collapsed or copied each other. That is the EMA target encoder trick done with explicit regularisation terms.
JEPA adopts the BYOL approach. One encoder is trained by gradient descent; a second target encoder is updated as an exponential moving average (EMA) of the first. The predictor learns to match the target encoder’s output. Because the target trails the predictor, it cannot run away. The training signal stays non-trivial, and collapse is avoided without pushing any pairs of embeddings apart.
The loss is also no longer a number you can read. A pixel loss of zero means the model predicted the frame. An embedding loss of zero might mean it understands everything — or nothing at all. Only a probe, a small second network trained to read facts out of the embeddings, can tell you which you got.
None of this makes generation a mistake. Systems that predict pixels are the ones making worlds you can walk around in, and they work. Whatever they waste on leaves, they deliver a picture, and no embedding does that. The awkward counter-example comes from LeCun’s own lab. Kaiming He’s 2021 paper Masked Autoencoders Are Scalable Vision Learners hides most of an image and rebuilds its pixels — the very thing JEPA refuses to do. The descriptions it learns are very good anyway, and it came out the year before the position paper.
The version of the argument that holds up is narrower than the slogan, but it is right. Predicting appearances spends most of its capacity on things no decision depends on. When the future is open, a deterministic predictor is trained towards a picture of something that will not happen. If what you want is a compact state to act on, that is a bad way to get one. If what you want is a picture, it is the only way anyone has.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is the leaf problem, and why does JEPA avoid it?
Why does a deterministic pixel predictor produce ghostly blurred images when the future is uncertain?
What does JEPA predict, and how does it avoid the ghosting problem?
What is the collapse problem in JEPA, and how does the EMA target encoder address it?
JEPA (Joint Embedding Predictive Architecture) predicts embeddings of future states, not pixels. Leaf problem: pixel models waste capacity on irrelevant texture. Ghosting: deterministic pixel predictors average futures into a smear. Fix: encode both current and future state, predict the embedding. Cost: collapse — avoid with EMA target encoder (BYOL trick) or explicit variance penalty (VICReg). I-JEPA (2023): images. V-JEPA 2 (2025): video + robot control.