the question
what a foundation model is
the term is about how it's made, not size or domain. fable is one. so is alpamayo, a driving model 100x+ smaller. coined by stanford in 2021.
five ways to build a robot's brain
| approach | how | examples | decisions / sec | the bet |
|---|---|---|---|---|
| classical | hand-built perception → planning → control | industrial arms, older av stacks | 100+ hz | engineering beats learning when it must be reliable |
| specialist policy | small net trained on robot data for one job | diffusion policy, comma | 10–100 hz | narrow and fast beats general |
| vla | vision-language model fine-tuned to output actions | rt-2, openvla, π0, alpamayo | 10–50 hz | web knowledge + robot data, one model |
| world model | learn physics by predicting future video, act on it | wayve gaia, nvidia cosmos | varies | knowing what happens next is the core skill |
| llm as brain | frontier model drives through tools or code | saycan, code as policies, fable | ~0.2–1 hz | general intelligence is enough |
the map
how big
why llms are so good
compared against the usual alternative: a specialist policy (or a vla) trained by imitation on robot demos.
| capability | specialist policy | llm | why it matters |
|---|---|---|---|
| prior knowledge | learns from 104 to 106 demos. knows only what's in them | pretrained on 1013+ tokens. already knows objects, physics, how things are used | a cap unscrews counterclockwise. the policy has to see it. the llm already knows |
| generalization | interpolates inside its training distribution. new object, lighting, layout = out of distribution | composes concepts through language. "unscrew" + "jar" works even if never seen together | the real world is mostly out of distribution |
| vision | encoder trained on task images, fixed set of things it recognizes | encoder trained on billions of image-text pairs. open vocabulary: "the mug with the chip in it" | you can name anything, it can find it |
| long horizon | sees a frame or a few, horizon of seconds. small errors compound | 100k+ token context. holds the plan, the history, where it is in the task | multi-step tasks fail on drift, not on any single step |
| recovery | trained on clean demos, rarely sees failures. drifts off and doesn't know how to get back | in-context learning. sees the failure, re-plans. no gradient update | imitation's classic weakness (compounding error) gets handled at test time |
| precision | has to learn geometry implicitly, end to end | calls exact tools: grasp planner, ik solver, segmentation. writes and debugs the code | hands the precise math to classical tools that are already good at it |
vlas sit in between. they start from a vlm, so they inherit some of this. but fine-tuning on narrow robot data overwrites part of what the base knew (catastrophic forgetting). that's why rt-2 kept training on web data alongside robot data.
the core of it: robot data is tiny and expensive, internet data is huge and free. the llm brings the world knowledge, the tools bring the precision. that's why fable can do tasks it was never trained on. the task is new. the world isn't.
the cost: all of that runs through a giant model, one token at a time.
why llms are so slow
how fast models get around it: small enough to run on the robot. no thinking in words, camera in and motor command out. and they output chunks: π0 predicts the next ~50 moves in one pass and runs them while computing the next.
does slow matter? not when the world waits: sorting, assembly, a rubik's cube. yes when it doesn't: catching, balancing, driving, a slipping grip. so the fix is the split. the llm picks what to do about once a second, a fast controller handles how, 100+ times a second.
the real-time wall
a big model can plan. it can't react. contact and balance need feedback every 10–100 ms.
the fight
- the bitter lesson. general methods that scale with compute beat hand-built expertise. chess, go, speech, vision. every time.
- nlp already did this. pre-2020: one model for translation, one for parsing, one for sentiment. then gpt-3/4 beat almost all of them without training for any. robotics looks like the same movie.
- trend vs snapshot. "$100 for a 5 min task" is today. cost per capability has been falling ~10x a year. the generalist improves every release for free. the specialist needs new data to move.
- the long tail. real requests are open ended. a specialist does what it was trained on. the world is mostly edge cases.
- economics. frontier labs spend billions per model. a specialist company is chasing a moving target, and one release can erase its edge.
- physics doesn't scale like software. a car has ~100 w and ~100 ms. tokens get cheaper, but not small enough to run onboard at 10 hz on the same timeline.
- offline and cost. a robot can't stop when the wifi drops. cost per hour matters across a fleet.
- moravec. the rubik's algorithm is easy for computers. gripping a slippery cap is the hard part.
- harness. if a tool says "grasp here," the llm is just sequencing. harness not published.
- benchmark isn't deployment. vendor-run, sim only. sim scores only partly predict real, let alone 10,000 hours in a warehouse.
- incentives. a lot of the loudest bulls are founders and investors whose bet depends on it.
what the benchmark actually shows
the headline says vlas are done. the per-task table says something else. robodojo runs two boards: end-to-end policies (vla, wam) on one, agents (a general model calling tools) on the other.
| kind of task | examples | agents | vla / wam |
|---|---|---|---|
| needs thinking | push a t into place, solve an equation, tic-tac-toe, sort dolls by size, press buttons by number | ~80 to 100% | ~0% on most |
| needs contact | insert tubes, hang mugs, assemble a toy, slide a laptop into a bag | tie or lose | tie or win |
and the #2 agent on the board, agibot's go-3 nano, isn't a pure agent. it's an agent with a wam underneath. the split, already on the leaderboard.
for anyone running fleets: the part that decides what to do is getting cheap fast. the part that keeps the robot alive when a door opens two seconds late is not.
google's version: gemini robotics 2.5
google is pushing the vla right at the last 10 cm: deformables, tools, delicate stuff. and it's a real factory, not a sim.
| role | who does it | what it does |
|---|---|---|
| thinker | gemini robotics er | tracks the workflow, plans the steps |
| hands | gemini robotics 2.5 (vla) | controls the arms and hands |
| data engine | gemini 4 argon (frontier llm) | turns one camera scene into thousands of simulated practice runs, diagnoses failures from video |
the big model isn't driving the robot here. it's building the robot's brain. that's the best answer i've seen to "specialist models are a dead end." the specialist isn't dead, the generalist makes it.
where it probably lands
both camps are describing the same thing from different ends. a specialist trained from scratch probably is a dead end. a specialist distilled from, or trained on data made by, a generalist is just how deployment works. the generalist wins on knowledge, the small model wins on running in real time.
so the real question is narrower: does the small model eventually go away, or is it permanent because of power and latency? for robots and cars, my bet is it sticks around a long time.
what it means for robot labs
- train big general models, head to head with anthropic / openai.
- needs someone to fund the capex.
- win on robots in the field, data, reliability.
- still competes with them, but greenfield. more possible winners.
open questions
- what does the robodojo harness actually expose? primitives or joints?
- how much of a frontier model survives distillation down to ~1b?
- does the sim result hold on real hardware?