The steering-wheel test: what separates watching from controlling. GameNGen, Genie, DIAMOND, Oasis — and why an action input alone does not make a simulator.
Left: passive video model. Right: action-conditioned simulator. Press Left/Right to steer and compare.
Watching vs controlling: why having an action input does not guarantee the model obeys it.
If you have watched any generated video lately, you will have seen somebody walk through a scene with the arrow keys. Until 2024 a video model was something you watched. It chose its own future and you had no say. One extra input changed that.
Take a model that predicts the next frame. Now tell it, along with the frames, which key is being held. From the same opening it can give you one continuation for left and another for right. Jake Bruce and colleagues at Google DeepMind made that move with Genie in 2024 — trained on footage of 2D platform games with no labelled keypresses. The model had to explain each change between frames with one of a handful of codes, and the codes it settled on behaved like a controller.
A model can receive an action and still ignore it, or respond only weakly. It can also let the action change how things look without changing the path underneath. Think of a steering wheel that turns without being connected to anything.
So control has to be measured. Hold the start fixed, vary only the command, and check whether the futures split apart as asked. I call it the steering-wheel test: action fidelity is how far the model obeys the command. The input can be present the whole time and mean very little. A model trained on footage is rewarded for frames that look right. If the footage mostly goes straight, going straight is a good bet whatever the key says.
Good video hides weak control. If three beautiful continuations all go straight, you have been shown video. A planner that asked for three counterfactuals — three what-ifs from one start — has been handed one of them three times.
Scaling did more than most people expected. Let’s take the two systems that mark the change.
GameNGen came from Dani Valevski and colleagues at Google Research in 2024. Their paper was called Diffusion Models Are Real-Time Game Engines. A diffusion model makes a picture by starting from noise and cleaning it up step by step. GameNGen used one to run DOOM, the first-person shooter from the early nineties. It made each frame from the earlier frames and the player’s inputs, fast enough to play. People shown short clips struggled to tell it from the real game.
Genie 3, from Google DeepMind a year later, is where the numbers stopped looking like a demo. It is reported to make interactive video at 720p and 24 frames a second, hold a scene together for several minutes, and take mid-session instructions: change the weather, add a flock of birds. Walk away from something, walk back, and it is roughly as you left it.
You will see the claim made about all of them. OpenAI’s 2024 report, Video generation models as world simulators, put “emergent physics” into common use. The observation behind it is real: train on enough video and the outputs start to respect regularities nobody put in. Dropped objects speed up as they fall, water pools rather than piling up.
But “it learned physics” says the model picked up the rule, and that is testable. Bingyi Kang and colleagues ran one in 2024: they trained video models on simple simulated scenes and asked for speeds the footage never showed. Inside the training band, the generators obeyed the laws. Outside, they broke them — and nothing in the output said which case you were in. I call that the band problem: the models were matching each new scene to the training scenes it most resembled, not applying a rule. Nobody tests outside the band, because that is where the footage runs out.
DIAMOND (2024) from Eloi Alonso and colleagues trains a diffusion world model directly in game environments, producing action-conditioned rollouts for RL agents to plan inside. Oasis, also 2024, does the same for Minecraft, running at interactive speed with keypress conditioning. Both pass a weak form of the steering-wheel test — pressing left does tend to produce a leftward trajectory. Neither claims to have learned physics rules that extend beyond the training distribution.
The distinction that matters is between a model that associates your command with the footage it most resembled, and one that has extracted a rule that would hold in novel situations. The first is a video model. The second would be a simulator. As of now, no demo has settled which you are looking at.
There are three seams where a demo can mislead you.
Consistency is learned, not enforced. Nothing stops the room from changing, and usually it does not. But usually is a statistical property of a trained system, and a guarantee is a property of an engine.
The interesting moments are the vague ones. A model trained on pixels is pushed towards the average of possible futures wherever the future is open. The steering-wheel test is hardest to pass exactly where passing it matters — at decision points, not in the straight corridors.
The scores do not compose. Benchmarks measure visual quality, geometry, action fidelity and persistence one at a time. There is no accepted rule for combining them into one ranking. A claim about improvement belongs to one particular test.
None of that argues against using them. For training other systems a generated world is useful: it avoids contact with the real environment and costs GPU time instead. It resets at once, and it can produce a thousand versions of a scene that would take a week to stage. NVIDIA built Cosmos for this — a set of video generators for making physically-aware training data for robots and vehicles.
For being a world you can move around in, they are already good. No other approach lets you walk into the picture. For being the model an agent plans inside, raw video generators are an awkward shape — a planner may need to score a hundred action sequences, and rendering each one is expensive. Systems can instead distil a learned latent, a short internal code that stands in for the frames, and plan in that.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is the steering-wheel test and why does passing it matter?
What did GameNGen demonstrate, and what were its limits?
What is the band problem in video models?
What separates a passive video model from an action-conditioned simulator?
Steering-wheel test: vary the command from a fixed start; measure how far futures split. Action input ≠ control. GameNGen (2024): DOOM via diffusion at play speed. Genie (2024): latent actions from unlabelled footage. Genie 3 (2025): 720p, minutes of coherence. DIAMOND/Oasis: action-conditioned diffusion for RL. Band problem: emergent physics is pattern-matching within training distribution — not learned rules. Scores don’t compose: visual quality, action fidelity, persistence are separate measures.