why simulate at all
The road is a bad place to learn. The scenarios that matter most, a car cutting in, a kid stepping off a curb, a truck braking hard ahead, are exactly the ones you can't stage safely or often enough. So the whole field runs on simulation: rebuild the world in software and drive the hard miles there. The interesting part isn't the graphics. It's a spectrum of techniques, each with a real name, each trading realism against control against cost. Here's the map.
log replay
The simplest simulation barely is one. You record a real drive, every sensor stream, and play it back to the stack. This is log replay (Applied ships it as Log Sim). It's maximally realistic because it is real. And it's open-loop: the recorded world cannot react to anything the car does. You're grading the stack against a fixed film. If the car would have braked earlier than it did that day, the log has no answer for what happens next, because that footage was never shot. Open-loop replay tells you what the stack would have decided, never what would have happened.
closing the loop
The real game is closed-loop: let the car's decision change the world, simulate the consequence, feed it back, repeat. The instant the ego deviates from the recorded path, log replay breaks, because the car is now somewhere the recording never went, and you have to synthesize the world it would see from there. Closing the loop is the hard, expensive, valuable thing. Everything below is a different bet on how to do it.
object sim: closing the loop in the symbolic layer
If your stack is modular (perception, then prediction, then planning, sometimes called two-stage), there's a shortcut. The planner never sees pixels; it sees a symbolic world, a list of objects, lanes, and a route. So you can skip rendering entirely and simulate that symbolic layer directly. This is object sim (Applied's Simian): spawn cars, pedestrians, and cyclists as abstract agents with positions and velocities, and let the planner drive against them in closed loop. Cheap, fast, fully controllable. The catch: it only tests the planner. Perception is assumed perfect, because there are no pixels to get wrong.
The mechanism is worth seeing plainly, because it's what makes closed-loop possible at all here. You replay the log's objects, not its pixels. The other agents are seeded from the drive, but the ego is not replayed, it's driven by the planner under test, and a vehicle model integrates its motion. So the ego leaves the recorded path and re-plans against those objects every tick: decide, move, decide again. The loop closes on the ego, and you never had to synthesize a single pixel, because the planner only ever needed objects, and the log already gives you objects. Log-based closed-loop is just object sim seeded from a real drive. This is the whole reason a two-stage stack can close the loop from a log and an end-to-end stack can't: the latter eats pixels, and the moment the ego moves, the logged pixels are from the wrong place.
The underrated payoff is iteration speed. Because it's symbolic and needs no rendering, object sim is deterministic, scriptable, and cheap, so you run thousands of scenarios overnight in CI and A/B a planner change in minutes. Fast, repeatable iteration on the planner is a large part of why modular stacks stay so productive. End-to-end can't buy this cheaply, because its loop needs generated sensors, which are slow and immature.
But there's a limit baked into replay, and it's the natural bridge to the next problem. The replayed agents are non-reactive: they follow their recorded trajectories, blind to the ego. So the moment the ego deviates far enough from what it did that day, it drifts away from where the logged agents are, they keep running their old script oblivious, and the interaction you cared about evaporates. The sim diverges from the log and quietly stops being meaningful. That divergence is exactly the pressure that pushes toward reactive agents.
sensor sim and neural sim: closing the loop in pixels
To test perception too, or to close the loop on an end-to-end stack (one-stage: pixels straight to a trajectory, no symbolic middle), you can't cheat with objects. The model eats raw sensor data, so you have to generate raw sensor data for wherever the car now is. Two ways:
The quiet consequence: end-to-end stacks are the fashionable direction, and they are the hardest to simulate, because closing their loop demands generated sensors, and generated sensors are the least mature piece. Modular stacks get cheap closed-loop testing almost for free. End-to-end stacks don't get it until neural sim grows up.
the agent problem: reactive vs non-reactive
Rendering is not the hard part. The other agents are. How the simulated cars and people behave decides whether a scenario is worth anything, and there's a whole vocabulary for it:
Here's the trap, and it's a good one. Reactive agents are more realistic, so "our agents are reactive" became the thing to brag about. But realism fights control. If you want to test a collision, a reactive agent trained on real driving will always dodge the ego, because not crashing is exactly what the data taught it. You literally cannot stage the crash. So to test the dangerous case you often have to make the agent non-reactive, freeze it on rails so it won't swerve out of the scenario you built. The most realistic agent is useless for the test that matters most. Realism and controllability pull in opposite directions, and good simulation is mostly the art of managing that tension.
world models
The frontier bet is the world model, a generative simulator that learns the dynamics of driving from enormous amounts of real footage and then generates new footage on demand. Wayve's GAIA is the clearest public example: prompt it with text, an action, or a starting frame and it produces realistic multi-camera driving video, controllable down to weather, time of day, and, crucially, safety-critical maneuvers like cut-ins and emergency stops that never show up enough in real logs. NVIDIA's Cosmos is a similar bet. The pitch is that a world model gets you all three things classical sim had to trade against each other: realistic pixels (like neural sim), controllable scenarios (like object sim), and a closed loop for end-to-end stacks (which nothing else offers). Whether it delivers all three at once is the live question of the field.
the landscape in one line
It's a spectrum from real-and-rigid to synthetic-and-flexible: log replay (real, open-loop) → object sim (symbolic, controllable, planner only) → sensor and neural sim (pixels, closes the loop for end-to-end) → world models (generative, the attempt to have it all). Every rung trades some realism for some control or some cost. There's no free lunch, only a well-named set of compromises. And the deepest compromise isn't technical: a simulator realistic enough to trust is a simulator whose agents won't do the dangerous thing you built it to test.