A dynamics model does one job: take what you know, take what you did, and produce what you know next. The one-step error looks impressive — until you see what happens when you eat your own output for a hundred steps.
See how one-step error compounds over multiple rollout steps.
One job: take what you know, take what you did, produce what you know next. The number a paper leads with is the easy number to be impressed by.
If you have read the last chapter, you have a state: a short list of numbers that stands in for what the camera saw. The next step is the one where that state moves. A dynamics model, also called a transition model, does one job. Take what you know, take what you did, and produce what you know next.
The awkward word there is know, because it covers more than the last picture. It is everything that has happened and still matters, folded into something small enough to carry. A ball behind a wall is still moving. A door you opened four rooms ago is still open.
A dynamics paper leads with its one-step error, and it is the easiest number to be impressed by. That number measures a job the model will never be asked to do. You were going to ask it for a hundred steps, and from step two it stops being handed the truth and starts being handed its own last answer.
The field named this in 2015 and has not solved it since. At 4% error per step, a corrected (teacher-forced) line still hugs the truth after thirty steps. The free-running line, same model, same error rate, ends up thirty-nine times further away. Keep that picture in mind.
The arithmetic is simple: compound 4% error over twenty steps and you do not get 80% — you get (1.04)20 − 1, which is 119%. One percent per step becomes 22% over twenty steps. The one-step number is not a lie. It just measures the corrected run, not the one that will actually be used.
The habit came with the first networks trained on sequences, decades before world models. During training a sequence model is shown the true previous value at every step. That is what the recording contains, and it keeps training stable. The field calls this teacher forcing.
The cost is that nobody teacher-forces you in production. At the moment of use the model gets its own last answer, which is slightly wrong. It has never once been asked to recover from being slightly wrong.
Samy Bengio and his co-authors named that gap in 2015, in Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. Bengio was at Google, working on networks that caption a photo one word at a time, each word fed back as the next input. That is a rollout, and the captions drifted the way the free-running line drifts.
The question underneath the dynamics model is how the past gets carried. Since 1997 it has had two answers.
Carry a summary. Keep one fixed block of numbers. Every time something new arrives, fold it in and throw the old block away. The work per step never changes however long the sequence gets. This is what recurrent networks do. The summary camp’s founding paper is Long Short-Term Memory, from Sepp Hochreiter and Jürgen Schmidhuber in 1997. Their LSTM was a loop built to hold a fact longer than training pressure wanted. It ran speech recognition and translation until transformers arrived.
Keep a window. Store the steps that fit in the context. When a new one arrives, look back over them to decide what matters. This is attention. Transformers have done this since 2017, when Ashish Vaswani and colleagues at Google published Attention Is All You Need. The title meant what it said: the loop went out.
Then the summary came back. State-space models (S4, Mamba) keep the fixed summary with better machinery. Mamba’s summary decides what to keep based on what it is looking at, rather than at write time. Speed is the half of the trade that gets benchmarked. The duller half: a fixed summary has to decide at every step what is worth keeping, without knowing what will be needed later.
The systems that work share one trick, arriving in 2018 with PlaNet. The trick is to carry two things forward rather than one. One part is deterministic: computed the same way every time, holding whatever reliably follows from the last state. The other is stochastic: sampled, holding whatever could still go either way.
A ball’s motion goes in the first. Whether the door opens goes in the second. Keep only the deterministic part and the model has no way to say it is unsure — it confidently invents one future. Keep only the stochastic part and it struggles to remember anything for long. PlaNet ran the comparison and argued for carrying both. Dreamer followed a year later, and the split state went into every Dreamer since, up to the Nature paper of 2025.
Nobody has removed the second-step problem. Instead the field has four ways of not being destroyed by it.
Two models can tie on the usual one-step test and behave in opposite ways when run free. Training on perturbed or self-generated states teaches recovery — it is a separate skill that has to be put into the training objective because the model does not learn it on its own.
Professor Forcing came in 2015 in Yoshua Bengio’s lab, from Alex Lamb and colleagues. It attacks the gap from the other side: instead of changing the inputs, it trains the model until its internal states look the same whether it is teacher-forced or running free. The judge is a second network trained to tell the two kinds of run apart — the model has to fool it.
Everything in this story stayed tied to recorded experience. The next post asks what happens when an agent learns inside the model instead.
Say the answer out loud before you reveal it — recalling is what makes it stick.
Why does one-step error understate how bad a dynamics model is in practice?
What is teacher forcing and why does it create a problem at test time?
What is the key difference between LSTM and Transformer approaches to sequence memory?
Why does PlaNet carry both a deterministic and a stochastic state component?
Dynamics model: (state, action) → next state. One-step error compounds — (1+e)^n not e×n. Teacher forcing: train on true inputs, but run on own outputs (the gap). 4 truces: scheduled sampling, rollout loss, short horizons, look-again. LSTM = summary, Transformer = window. PlaNet: deterministic + stochastic state.