What Is a World Model?
A world model is a learned simulator: a neural network that can predict what the world will look like after the agent takes an action, without requiring the real environment to run. Rather than learning purely from trial and error in the actual environment (model-free RL), model-based RL uses a learned model of the environment dynamics to plan, imagine, and train without as much real-world interaction.
The idea is appealing: humans don't need to physically try every option before making a decision. We mentally simulate likely outcomes, predict which actions are dangerous before taking them, and plan complex sequences of actions by reasoning in our heads. World models aim to give AI agents similar capabilities.
DreamerV3: A Modern World Model
DreamerV3 (Hafner et al., 2023) is the current state of the art in model-based RL. Its architecture has three components: a Recurrent State Space Model (RSSM) that learns to predict the next state from current state and action; an encoder that maps observations to the latent state; and a decoder that maps latent states back to observations (for visualization and auxiliary objectives).
DreamerV3's key claim: a single model that works across 150+ diverse tasks spanning Atari, continuous control, 3D environments, and robotics — trained only from reward and visual observations, without task-specific architecture changes. This generality is significant; earlier model-based methods typically required extensive tuning for each domain.
Planning in Latent Space
With a world model, the agent can plan entirely in latent space — imagining trajectories without rendering them, evaluating potential future states cheaply, and backpropagating gradients through simulated trajectories. This dramatically improves sample efficiency: the agent can make thousands of imagined decisions for every real-world step. DreamerV3 typically achieves human-level performance on Atari in 200M imagined steps, corresponding to only 20M real environment steps.
Challenges: Distribution Shift and Compounding Errors
World models have a fundamental challenge: they're trained on the agent's current behavior, but used to plan future behavior that may be very different. Predictions become increasingly inaccurate for plans that explore regions of state space the agent hasn't visited. Small errors in each prediction step compound: after 50 steps, a 1% per-step error rate can produce completely unrealistic imagined trajectories.
Practical mitigations: use short planning horizons (avoid compounding errors over many steps); maintain an ensemble of world models and only use states where models agree; use uncertainty-aware planning that treats high-disagreement regions as dangerous.
Application to Robotics
World models are particularly powerful for robotics, where real-world interaction is expensive and sometimes dangerous. The Threshold Robotics Lab at Meridian AI uses a world model approach: train in simulation, use the world model to bridge the sim-to-real gap, and only collect real-world data for fine-tuning the world model to capture reality's deviations from simulation. This pipeline reduces the required real-world robot interactions by 10-100× compared to purely model-free approaches.