Ask something to predict what comes next and it has no choice but to build whatever the next moment depends on. Direction, speed, grammar, a game board — stowaways nobody put on the manifest.
Ball in mid-air. Predict N steps ahead to reveal the stowaways the predictor must carry.
Stowaways revealed
Stowaways, the guess-check-adjust loop, and where it stops being enough.
Take a photograph of a ball in mid-air and ask where it will be a moment later. You cannot say. Up, down, sideways, or barely moving at all, the picture looks the same. Everything the answer depends on is missing from a single frame.
Now take two photographs a fraction of a second apart. This time the answer arrives before you have finished asking. Both pictures were equally sharp, so detail is not what changed. Two frames hold something one frame cannot: a direction and a speed. Neither is drawn on either picture. They live in the relationship between the two, and you pulled them out without being told to.
Ask something to predict what comes next, and it has no choice but to build whatever the next moment depends on. It was never asked for those things, and it cannot be right without them. I call them stowaways: the quantities a predictor is forced to carry that nobody put on the manifest. Direction and speed are the first two.
People have been finding larger stowaways ever since, with better instruments each time. Claude Shannon went first, in 1948, with A Mathematical Theory of Communication. Jeffrey Elman followed in 1990 with Finding Structure in Time — he found grammar inside a network asked only to guess the next word. Then in 2022 Kenneth Li and colleagues wrote Emergent World Representations and found a whole game board inside a network asked only to guess moves.
The loop is as small as it sounds. Look at what you have and say what you think is coming. Wait, then compare what arrived to what you said. Change yourself a little in the direction that would have been less wrong, and go again.
None of the steps is clever. The power is in what the loop does not need: nobody has to label anything, because the target is the next piece of the recording. This is all that self-supervised means. Supervised, because the model is told how wrong it was after every guess. Self, because the data did the telling rather than a person with a spreadsheet.
For most of the field’s history a model learned from examples that somebody had labelled. A million images meant a person writing down what was in each one. The labels were the costly part, and the size of a dataset was the size of somebody’s budget. Prediction skips the bill, because any recording of anything is already a training set. That convenience, more than any single result, is why prediction took over the field.
Elman was a cognitive scientist at the University of California, San Diego. In 1990 he trained a small network on one job: read a word and predict the next word. Then he looked inside. The internal states had sorted themselves into groups: nouns and verbs, animate and inanimate, things that can eat and things that can be eaten. Nobody had given the network a single category. It reached them because a word’s category is what its next word depends on. The title, Finding Structure in Time, is literal: the structure was found rather than installed.
For a long time that was where it stopped. The network was small and the grammar was a toy. The finding waited thirty-two years for instruments that could make it one.
In 2022 Li and colleagues took a network since nicknamed Othello-GPT. It was trained only to predict legal moves in Othello — a board game on an eight-by-eight grid. It was never shown a board and never told the rules. They probed its activations: trained a small separate classifier to read one specific fact out of the network’s internal numbers. The state of the board was in there, which on its own is a curiosity.
The intervention is what made it a finding. Change that internal board, and the moves the network goes on to make change with it. So the board is there because the network is using it. The two steps are worth keeping apart, because only the second one settles anything. Neel Nanda and colleagues (2023) studied the same network and showed the board was stored plainly enough for the simplest probe to read — the network stored each square as “mine” or “theirs” rather than black or white.
The same rule sits under both results, thirty-two years apart. Prediction rewards one thing only: making the next moment less surprising. Sometimes the cheapest way to be less surprised is to build the very thing that is making the data.
The loop says predict the next thing. It never says what counts as a thing, and that choice is the design decision. Predict the next pixel and you get a system that is very good at leaves. Leaves are hard to predict and they are also most of the pixels, so that is where the capacity goes. I call that the leaf problem.
Predict the next word and you get grammar, plus a surprising amount of knowledge about the world. In 2019 Alec Radford and colleagues at OpenAI showed how far it goes at scale. Their model was GPT-2, trained on nothing but next-word prediction over a large pile of web text. It came out able to answer questions, summarise, and translate a little — and nobody had trained it for any of those. The model learned those tasks because the next word sometimes depends on them.
Predict the next state of some compact description and you get something a planner can use. So the same loop gives three different machines, and all that changed is what was put on the right-hand side of the guess. Predict the next thing is a family of methods rather than one. Picking the target is the design decision that matters most.
There is a second way to look at this, four decades older than Elman’s result. Shannon was a mathematician at Bell Labs, whose business was pushing more calls down the same wire. Suppose you have to send a message and you are charged by the symbol.
You and the receiver share a predictor. Before each symbol both of you run it and get a list of what is likely next, so you need only send enough to pick the right one. Confident and correct costs almost nothing; confident and wrong costs a lot. Shannon made this exact in 1948: a symbol you gave a probability of one in two costs one bit. One in four costs two. The word “bit” first appeared in print in that paper.
So the length of the shortest message you can write measures how well you predicted. A better predictor is a better compressor. Read backwards, whatever compresses your data hardest has understood the most about it. Marcus Hutter took the reading literally: the Hutter Prize is a long-running cash prize for compressing a snapshot of Wikipedia. Every winner to date has been, in effect, a language model.
The same reading says why the stowaways were never optional. Memorising is the costly way — a lookup table of everything that ever happened is huge and useless on anything new. The cheap option, the one that shortens the message, is to work out the rule that produced the data and keep that instead. The rule is the stowaway.
There are two limits.
Predicting well is not the same as being useful. A system can be excellent at continuing a recording and still hand you nothing you can act on. Saying what comes next is a different question from saying what would come next if you did something different. Only the second supports a decision. Li’s Othello-GPT carries a board and was still only ever asked which moves were legal.
Being unsurprised is not the same as being right. A predictor that has only ever seen calm weather is not surprised by calm weather. That tells you nothing about what it would do in a storm. Low error on what you happened to record is a weaker claim than it sounds, and it is the claim most headline numbers make. Drag the launch speed of a ball past the range the model was trained on and watch the gap open. The calm days were the band, and it never saw a storm.
Neither limit undoes the argument. Prediction is enough to pull structure out of raw experience, and it is the cheapest signal anyone has found. The line from Shannon through Elman to Othello-GPT is people proving that with better instruments.
Say the answer out loud before you reveal it — recalling it is what makes it stick.
What is a stowaway in the context of prediction, and what are the stowaways for a ball in mid-air?
What is self-supervised learning, and why did it take over the field?
What made the Othello-GPT finding more than just a curiosity? What was the critical test?
What are the two limits of prediction as a learning signal?
Stowaways = quantities forced onto a predictor nobody asked for (vx, vy, g for a ball; grammar for words; a game board for moves). Self-supervised: target is the next piece of the recording — no human labels. Guess-check-adjust loop: compare prediction to reality, nudge weights toward less wrong. Better predictor = better compressor (Shannon 1948). Target choice matters: pixels → leaves; words → grammar + world knowledge; compact state → plannable. Two limits: predicting well ≠ being useful (need action-conditioning); being unsurprised ≠ being right (distribution shift).