the question
imitation tops out at the teacher. copy the best human, you get the best human, minus a bit. self-play drops the teacher. the model plays itself, and the only limit is the rules of the game.
the loop
how it got here
why it works in games
| needs | chess / go | driving | robot hands |
|---|---|---|---|
| a perfect simulator | yes, the rules | close, sim-to-real gap | hard, contact physics |
| a clear reward | win / lose | "drove well" is fuzzy | sometimes |
| millions of tries | cheap | only in sim | only in sim |
| an opponent | yes | other drivers | mostly just physics |
games check every box. the real world checks almost none. that's why self-play crushed games years before it touched robots.
where it shows up now
- fill a simulator with copies of the policy as every car on the road. they learn to drive around each other.
- apple's gigaflow (2025): ~1.6 billion km of self-play, no human driving data, strong on three driving benchmarks.
- driving has the one thing robot hands don't: other agents. that's what makes it self-play.
- math and code have a checker, like chess has rules. the answer is right or it isn't.
- so you can let a model try millions of times and reward what works. rl with verifiable rewards.
- a big part of why reasoning models got good. alphazero's idea, moved to text.
the missing piece: the simulator
self-play needs a world to play in. for games, someone wrote it. for robots, nobody can. so the bet is to learn the world (a world model, or a frontier llm building digital twins like argon) and then self-play inside that. the sim is the bottleneck, not the algorithm.
links back: foundation model · robotics system design (the data loop)
open questions
- how good does a learned world have to be before self-play in it transfers to real robots?
- what's the reward for "did the job well" when there's no win / lose?
- can contact-rich manipulation ever get the gigaflow treatment, or is it stuck on real data?