At a junction you run the scene forward before deciding. A dynamics model does the same — take what you have, run it forward, move for what comes next. This chapter covers rollouts, cached answers vs recipes, and why errors compound.
Simulate a 1D dynamics model. Position evolves under a chosen action.
A junction decision, a dynamics model, and thirty years of not trusting the dream.
I want to start at a junction, because I think it is the most ordinary world model there is. You are waiting to turn right. You look right and there is a car coming. Somewhere in the next second you decide: go now, or wait for it to pass.
What you do in that second is run the scene forward. That car is there, going about that fast, so in two seconds it will be here. You commit before anything has happened, on your own forecast of it. Most of the time the forecast is good enough, and the whole thing takes less time than it took to read about. The cases where it is not good enough are the interesting ones. Learner drivers misjudge the speed of oncoming cars, and a wrong forecast here ends in a collision you do not get to undo.
Take what you have, run it forward, and move for what comes next. That is a dynamics model. Of everything the phrase world model has since been stretched to cover, it is the oldest and the one the term was coined for. The coining happened in 1990.
The model itself only predicts what would happen. Something else reads those predictions and chooses, and that something else has its own name, the policy — the part that chooses the action, by hard thinking or by pure reflex. Jürgen Schmidhuber’s 1990 version kept the two as separate objects, and the split holds all the way through this chapter.
So when somebody tells you their system is a world model, ask which half they mean. The first half — the prediction — gets you a great deal, and it also fails in a way nobody has engineered away.
The shape has not changed since 1990: where you are, what you do, what comes next. A dynamics model learns that mapping. It usually learns what the step was worth as well. Written down, with a hat on, it looks like: f̂(s, a) → (s′, r̂). The hat says estimated, and every failure later in this chapter is that hat coming due.
Where you are used to be written down by hand. In the control theory of the 1960s, the state of a thing was its position and speed. The field has spent the years since 2018 letting go of that idea, in three steps.
PlaNet (2018), from Danijar Hafner and colleagues, was the first step. It learns its model from images and plans inside that model — it works out where it is from the pictures and never names any part of that answer. MuZero (2020), from Julian Schrittwieser and colleagues at DeepMind, was the second: its model never rebuilds the scene at all, producing only three number-like things — how good this is, how good it will get, and what to try. Every one wears a hat. The newest systems take the third step, running with no decoder at all: TD-MPC2 (2024), for controlling simulated robots, is one such system.
So a model does not have to look like the world. It only has to keep enough of what the world does. “Is it photorealistic?” is the wrong first question. A model of a chess position that cannot draw a bishop is still a perfectly good model of a chess position.
One prediction on its own is not the useful part. Hand the model a run of actions nobody has taken, let it read its own output, and you get a future nobody has seen. That is a rollout: one imagined run forward — predict, feed the prediction back in, predict again. Its length is the horizon.
The question to put to every new paper is whether you can run it forward under actions nobody took and compare the results. Sebastian Thrun’s system was passing that test in 1990.
Let me put two cars in front of two matching walls. The first brakes when the wall is nearer than some fixed distance. Somebody tuned that distance once, at the speeds they expected, and shipped it. The second carries a model of its own braking. Every tick, it asks where it would stop if it braked now. At the speed the first was tuned for, they do the same thing.
Arrive faster and the trigger still fires at the same distance. The rule has a built-in guess about how long stopping takes, and nothing inside it knows it is guessing. The model car moves its decision because its stopping point moved, and it can see that it moved.
The difference that lasts is about when the thinking happened. A policy settled most of the answer during training — close to a list of answers: in this situation, do that. A model kept a different kind of entry: in this situation, if you did that, this is what you would get. The first is a cache. The second is the recipe for working out a fresh entry once you know the question. Caches are usually right, and that is why so much of biology and engineering is made of them. The trouble starts when the question moves outside the range the answer was prepared for.
A model buys you three things, and all three come down to working on futures that have not happened.
You can try an action before paying for it. A planner writes down a run of actions and plays it through the model. It scores how that turned out, throws it away and writes another. With one candidate it is guessing, with two hundred it is shopping, and nothing in the world changed in between. PlaNet does this on simulated control tasks, walking and balancing, starting from raw pixels. MuZero is how far the idea goes: it matched AlphaZero at chess, Go and shogi, having been told less.
You can practise where mistakes are cheap. Sutton’s Dyna mixed real experience with experience the model made up. Ha and Schmidhuber (2018) trained their controller entirely inside the model’s dream of a DOOM level, then moved it back into the real game, where it held up. Dreamer (2019) put the trick at the centre: imagine long runs, learn the behaviour from the imagined ones. DreamerV3 (2025) ran that recipe across more than 150 tasks, retuned for none of them — the first system to dig up a diamond in Minecraft with no human play to copy.
You can ask what if. What if I brake now, what if I turn instead, what if I push it from the other side. Comparing those without running them is why it matters that the model is told which action you mean. It moves the question from what probably happens next to what happens if I do this. The answer is true inside the model. Given what it has learned, this is what braking earlier would have done. Nobody should mistake that for a report from the world that did not happen.
All of that argues for more: a better model, more candidates, longer runs. Each step makes sense on its own. Together they walk you into the failure this field is organised around.
Errors compound. A model that is slightly wrong once is fine. A model that is slightly wrong and then reads its own answer back in is fine for a while, and then it is nowhere near. Nothing breaks at any step. A one-step model can be accurate enough to trust while a hundred-step fantasy stitched out of that same model is worthless. Tingwu Wang and colleagues (2019) named the limit the planning horizon dilemma: look too short and the planner cannot see the goal; look too long and it is steering by a model that has drifted.
Search goes hunting for the errors. Say the learned map is wrong in one place — it thinks there is a gap in a wall that is solid. Try plans at random and you will almost never go near it, and a weak search never finds it. A thorough one does, because the best route in the whole learned world runs straight through it. The field calls this model exploitation: the better the planner gets at beating the model, the more it drifts toward the places the model was never taught. A model can be wrong by almost nothing on average and still be a dangerous thing to plan inside.
Nobody has fixed this, and I do not think anybody is close. What thirty years has built instead is a family of ways to limit how far imagination gets trusted.
Ask more than one model. Train several instead of one — an ensemble. Where the models disagree, the plan has wandered somewhere nothing supports. PETS (2018), from Kurtland Chua and colleagues, carried their disagreement through the plan.
Do not dream far. The simplest answer, and Janner’s (2019), is to start every imagined run from somewhere the world produced and keep it short. Planning on a short clock does the same job: plan a little, act a little, look again, plan again — the usual name for this is model-predictive control; it ran oil refineries before it ran robots.
Charge for not knowing. Take the score the model hands back and dock it wherever the model is unsure. MOPO (2020), from Tianhe Yu and colleagues, built pessimism in for offline settings — the reward the model promises counts for less wherever the model is unsure.
All three say the same thing, and so did Dyna before them: do not let the planner collect full marks for going somewhere the model has no grounds to be confident about. The best of these systems now hold up across a startling range of tasks: DreamerV3 across more than 150, TD-MPC2 across 104. But they are careful ways of using a model that is wrong.
Say the answer out loud before you reveal it — recalling it is what makes it stick.
What is a rollout, and why is the horizon important?
What is the difference between a policy and a dynamics model?
What is model exploitation, and why is it a problem even when average model error is low?
MuZero never rebuilt the game board. Why did that still work for chess and Go?
Dynamics model: f̂(s,a) → (s’,r̂). Rollout: feed prediction back in for H steps. Policy = cache (settled during training). Model = recipe (computed fresh given question). Buys: action trials, cheap practice, what-if queries. Fails: compounding errors over long horizons; search exploits model errors (model exploitation). Fixes: ensembles (PETS), short rollouts from real states (Janner 2019), pessimistic reward penalty (MOPO 2020). Key systems: PlaNet (latent planning from pixels), Dreamer (behaviour from imagination), MuZero (3 numbers, no board), DreamerV3 (150+ tasks, Minecraft diamond).