A robot arm drops things a million times learning to pick them up. Spend part of that month collecting experience, fit a model, and let it practise inside the model where a drop costs only compute. Richard Sutton wrote that loop in 1991 and called it Dyna.
Animate the real-env and imagination cycle.
The short answer is yes, and Dreamer is the evidence. The longer answer is about what it costs.
A robot arm learning to pick something up will drop it a couple of million times first. Every drop costs something real: wall-clock seconds, wear on the joints. Someone nearby has to reset the scene when the arm knocks the bin over. At one attempt a second, that is a month of doing nothing else.
The idea is simple enough to say in a sentence. Spend part of that month collecting experience, fit a model to it, and let the arm practise inside the model, where a drop costs only compute. Richard Sutton wrote that loop down in 1991 and called the agent Dyna. Sutton co-wrote the standard reinforcement learning textbook. Dyna was tiny — a loop in a grid maze — and about thirty years early.
Danijar Hafner’s Dreamer, which runs the same loop with deep networks, arrived in 2019. Learning inside a model is the biggest practical argument for building one, and Dreamer is the evidence.
Hafner built it as a Toronto PhD student working inside Google Brain, a year after his PlaNet had planned by search inside a learned model. Dreamer dropped the search and trained a policy — the part that picks the action — instead. Dream to Control, the 2019 paper, learned behaviour from long imagined rollouts. Neither the actor nor the critic ever touched the environment while it learned. The actor is the policy under another name. The critic is a second network that guesses how much reward lies ahead.
DreamerV2 followed in 2020 as Mastering Atari with Discrete World Models, and started beating agents that learn straight from the environment on the Atari games. DreamerV3 arrived in Nature in 2025 as Mastering diverse control tasks through world models. It ran with one set of settings across more than 150 tasks. It was the first agent to collect diamonds in Minecraft without human data — twelve steps in order — which is the strongest answer available to whether learning in imagination generalises.
Dyna’s loop has four steps and repeats: act in the world for a while and keep what happened; fit a model to it; train the policy inside that model (compute, not robot contact); take the behaviour back out.
The behaviour that comes back out collects better experience than the last lap’s did. That makes the next model better. The second lap is the part the summaries drop. A model fitted to the flailing of an untrained agent is only good where an untrained agent goes. It only improves once a better policy goes somewhere new.
That is why the month always keeps a day and a half locked. Imagined steps are worth having only while the model can supply them accurately, and it can only do that where it has been.
A training log counts one real transition and one model-generated transition as two rows. The learner cannot count them the same.
Imagined experience carries the model’s blind spots. The discount compounds with every step a rollout takes from a real state. Michael Janner and colleagues priced it in 2019. Their paper, When to Trust Your Model, introduced MBPO: train a policy on imagined rollouts only a few steps long, each starting from a real state drawn from the replay buffer. That trades volume for trust.
A policy trained inside a model is scored by that model. The model is all it can see, so it cannot tell a good action from one that only looks good to its examiner. It will find the second kind, because those are the cheapest points on offer.
David Ha and Jürgen Schmidhuber met this head-on in 2018, in World Models. Their agent was a tiny controller trained entirely inside a dream of a Doom level where the player dodges fireballs. The controller found a way of moving that stopped the dream producing fireballs at all. Inside the model this was an excellent policy. In the real game it was worthless, because the game had not agreed to stop firing. The fireball policy is the student that found the examiner’s blind spot.
Ha and Schmidhuber’s fix was a worse model, on purpose. If the policy is exploiting a quirk, make the quirk unreliable. Turn up the uncertainty in the model’s predictions during training — the dial is usually called temperature. Then the same action stops always producing the same handy outcome, and a trick that needs the dream to roll one way is not worth building on.
Both ends of that dial fail: a confident dream gets exploited, and a noisy one has no task left to learn. What works is the narrow band where neither failure has taken over. Robotics made the same move a year earlier: Josh Tobin and colleagues at OpenAI called the fix domain randomisation, in 2017. OpenAI’s 2019 paper, Solving Rubik’s Cube with a Robot Hand, pushed the idea onto real hardware.
The loop works, and it costs. Three things the Dreamer line did not buy:
Say the answer out loud before you reveal it — recalling is what makes it stick.
What is the Dyna loop and who invented it?
What did DreamerV3 achieve in Minecraft, and why does it matter?
What is the fireball policy and what does it reveal about model-based RL?
Why is imagined experience a different currency from real experience?
Dyna (Sutton 1991): Act → Store → Fit Model → Imagine → Policy Update → repeat. Dreamer (2019): actor-critic in latent space. DreamerV3 (2025): 150+ tasks, first diamond in Minecraft. Fireball policy: model-trained agents exploit model blind spots. Fix: temperature / domain randomisation. Imagined ≠ real experience: short rollouts from real states (MBPO).