01. humanoid kitting on a car line
based on gemini robotics 2.5 × apptronik × mercedes-benz. the real numbers are theirs, the design is my take.
the prompt
design a humanoid that picks parts from bulk bins into trays, in the exact order each car on the line needs them. live factory, people nearby.
requirements
functional
- read the build order for the next car
- find and pick the right part from a messy bin
- place it in the right tray slot
- check the tray, fix gaps and mistakes
non-functional
- keep up with the line. a late tray stops production
- reliable. 96% per pick isn't enough on its own, recovery has to close the gap
- safe around people. not optional, not learned
- no cloud in the loop. wifi drops can't stop the arm
back of the envelope
~10parts per tray (assume)
~60/hrtrays, about one car a minute (assume)
~6 sper pick, all in
~24/hrmisses at 96%. one every ~2.5 min
the takeaway: at line speed, 96% means something goes wrong every couple of minutes. the design is mostly about the 4%. detection and recovery matter as much as the grasp.
architecture
four loops, split by speed
order systemfactory software (mes)
which car is next and what parts it needs, in order
per car~1/min
plannerembodied reasoning vlm
tracks the tray, picks the next part, checks the result, decides to retry. the thinker
~1 hzon robot
policyvla, action chunks
"pick bracket a, place slot 3" → arm and hand motion. the last 10 cm
10–50 hzon robot gpu
controllerclassical
joint torques, compliance so a bump doesn't break anything
500–1000 hzmotor drivers
safety layercertified, independent
sees people, slows or stops the robot. can override everything above it. no ml in the critical path
hard real-timeown hardware
offline, not in the loop: frontier llm as data engine. turns failures into simulated practice, diagnoses from video.
one pick, end to end
ordercar #482
→
planner"bracket a → slot 3"
→
policygrasp + place
→
controllertorques
→
verifycamera on tray
→
next part, or retry
where each piece runs
| piece | size | rate | runs on | why there |
|---|---|---|---|---|
| planner | ~1 to 10b | ~1 hz | robot gpu | slow is fine, but it can't depend on the network |
| policy | ~1 to 3b | 10–50 hz | robot gpu | contact needs fast feedback |
| controller | no ml | 500+ hz | motor drivers | physics, has to be deterministic |
| safety | no ml | hard rt | separate safety plc | has to be certifiable |
| data engine | frontier | offline | data center | too big and slow for the loop, perfect for training |
sizes and rates are my rough guesses for this kind of system, not published numbers.
failure modes
| what goes wrong | how you know | what happens |
|---|---|---|
| missed grasp | hand sensors, nothing in the gripper | policy retries, new angle |
| wrong part | planner checks the tray image | pull it, put it back, re-pick |
| parts tangled in the bin | repeated failed grasps | shake or re-sort the bin, then retry |
| person walks in | safety layer, not the model | slow or stop. resume when clear |
| falling behind the line | planner tracks time vs takt | flag a human early, before the tray is late |
| network drops | heartbeat | nothing. the loop is all on the robot |
| stuck, 3+ retries | planner retry count | call a human, log everything |
the data loop
fleet runs
→
record & replayevery failure logged
→
data enginesim variants, diagnosis
→
retrain policy
→
eval in sim + on a test cell
→
roll out to fleet
this is the flywheel. every miss becomes training data. apptronik said the models improved "dramatically in just a few weeks of post-training" and carried over to production with little drop-off.
follow-ups an interviewer would ask
- why not one end-to-end model? you can't certify it, and you can't tell which part failed. the split gives you seams to test and debug.
- why not an llm for the grasp? seconds per decision. contact needs 10+ hz. see foundation model.
- why a humanoid and not a fixed arm? the line was built for people. a humanoid drops in without rebuilding the station. a fixed arm is cheaper and more reliable if you can rebuild.
- how do you ship a new model safely? shadow mode first, then one robot, then the fleet. the safety layer never changes with the model.
- what do you measure? picks per hour, success per pick, recoveries per hour, human interventions per shift, trays late.