Robotics' End Game: Nvidia's Jim Fan
In a Nutshell
NVIDIA's Jim Fan declares robotics entering its "endgame," mirroring LLMs' progression: pre-train world models on vast video data to predict next physical states (WAMs like DreamZero), align via action fine-tuning, and scale with RL—outperforming VLAs by prioritizing physics/verbs over language. Data shifts from costly teleop to scalable sensorized human egocentric videos (EgoScale: 21k hours pre-training yields dexterous policies) plus real-to-sim synthesis (DreamDojo) and wearables like DexOOI, unlocking neuroscaling laws for dexterity up to 10M hours/year. Predictions: physical Turing Test in 2-3 years, API fleets for factories/labs soon after, full agentic auto-research by 2040.
These notes were generated by AI and may contain inaccuracies.
Andre introduces Jim Fan, who leads the embodied autonomous research group at NVIDIA, known as NVIDIA Robotics. Robots are thrilling; cars are big robots, but excitement is for robots that can lift things. Jim was a standout at last year's AIN.
Jim recounts a summer day in 2016 in this office: a guy in shiny leather jacket with big biceps delivers a large metal tray inscribed, "To Elon and the OpenAI team, to the future of computing and humanity, I present you the world's first DGX1." First time meeting Jensen. As a good intern, Jim signs his name on it (visible in photo), along with Andre. They joke about going to the computer history museum, feeling like dinosaurs. Back then, no clue what they were signing up for.
If you believe in deep learning, deep learning will believe in you.
Deep learning believed in them big time. Three-step functions in six years: (1) GPT-3 pre-training on next token prediction learns grammar and shape of language, simulating how thoughts, code, and strings unfold. (2) 2022 InstructGPT supervised fine-tuning aligns simulation to useful work or reasoning. (3) Auto research accelerates the loop beyond humanly possible.
All labs are at the final boss fight for LLMs, in the end game. OM folks speedrunning AGI on mythical creatures called mythos.
Why can't robotics have fun? Jim copies homework and calls it the great parallel: simulate next physical world state, align through action fine-tuning to a thin slice that matters for real robots, let reinforcement learning carry the last mile. If you can't beat them, join them. New episode: Robotics, the endgame.
How to play: boils down to model strategy and data strategy.
Model Strategy
Last three years dominated by VLAs (visual language action models) like Pi and Groot: pre-training on VLM, graft action head on top. Really LVAs, most parameters on language. Language first-class citizen, then vision and action. Great at nouns/knowledge, weak on physics/verbs; head-heavy in wrong places.
Example from original VLA paper: "Move the coke can to a picture of Taylor Swift." Generalizes without seeing Taylor Swift before, but not ideal pre-training.
Second paradigm: AI video slop. Video models learn to simulate next world state. Probes from V3 show models pick up gravity, buoyancy, lighting, reflection, refraction by predicting next pixels at scale. Visual planning emerges: V3 solves mazes by simulating forward in pixel space.
Favorite example: V3 solves task by not looking (geometry optional). Call this physics slop.
Make useful via action fine-tuning: collapse superposition of futures to robot-relevant slice. Introduce DreamZero: policy model dreams couple seconds into future and acts. Motor actions are high-dimensional continuous like pixels; render jointly.
DreamZero jointly decodes next world states and actions; zero-shot solves unseen tasks/verbs. Visualize robot's dreams during execution; tight correlation: good video prediction → good action; hallucination → failure. Vision and action now first-class citizens.
Fun with DreamZero: wrote robot around lab, prompt random tasks. Not 100% robust, but gets shape of motion correct like GPT-2. First step to open-ended, open-vocabulary prompting for robotics. New model type: world action models (WAMs).
Moment of silence for VLAs. Rest in peace. Long live WAMs.
Data Strategy
NVIDIA chief scientist Bill Dally does teleop (most expensive trajectory ever).
Past three years: teleop golden era—VR headsets, low-latency streaming, complex rigs. Upper bound 24 hours/robot/day (realistically 3 hours, robots throw tantrums).
Better: UMI (universal manipulation interface)—wear robot hand on your hand, collect data directly, bypass rest of robot body. Greatest robotics data paper; spawned unicorns: Generalist (wearable gripper), Sunday (three-finger data gloves).
NVIDIA's DexOOI: exoskeleton with 1:1 mapping to five-finger dexterous hands. Comparison: left—human direct fastest; right—skilled PhD teleop slow, low success; middle—wear exoskeleton, collect data, train policy (zero teleop data). Breaks 24-hour curse; robots happy, out of data loop.
Not solved: Tesla FSD or Waymo drivers contribute to physical data flywheel ambiently. UMI still cumbersome. Need FSD equivalent: data collection fades to background, captures full human dexterity across lives/labors.
Going all-in on human egocentric videos with annotations (hand tracking, dense language). EgoScale: 99.9% training from egocentric videos → end-to-end policy from pixels to 22-DoF dexterous hands (fully autonomous).
Pre-train on 21k hours in-the-wild egocentric human data (zero robot data), predict hand joints/wrist poses. Action fine-tuning: 50 hours high-precision MoCap gloves + 4 hours teleop (<0.1% mix).
Generalizes to dexterous tasks: sorting cards, syringe liquid transfer (robot nurses?), one-shot shirt folding.
Key finding: neuroscaling law for dexterity—clean log-linear relation between pre-training hours and optimal validation loss. Six years after original for language models.
Chart: X=alignment to robot hardware, Y=scalability. Teleop least scalable; data wearables to hundreds of thousands hours; egocentric video to 10M hours/year via FSD flywheel. Left of line: sensorized human data new paradigm.
Predictions (1-2 years): teleop → negligible; ensemble of data wearables for hardware/use cases; main diet = egocentric videos. Moment of silence for teleop. Long live sensorized human data.
Outer ring: scale environments like OM labs' millions of coding envs for RL. RL on real robots pushes tasks to ~100% success (continuous hours assembling GPUs). But 1M envs need 1M robots.
Better: iPhone photo → 3D world scan → extract objects → synthesize interactive in classical physics sim (digital cousins). iPhone = pocket world scanner. Real-to-sim-to-real.
Do better: DreamDojo—video world model as neural simulator. Inputs continuous actions, outputs next RGB + sensor states in real-time (purely data-driven, no physics/graphics). Captures robot mechanics.
New paradigm: massively parallel RL—few real robots + GPUs for world scans + inference for world models. Compute = environments = data.
Great parallel happening now; beginning of endgame. Like Civilization: unlocking achievements on civilizational tech tree. Three more for robotics:
- Physical Turing Test: indistinguishable from human across activities (unit energy in, unit labor out). 2-3 years away.
- Physical API: fleet of robots configured via APIs/command lines (orchestrated by o1 9.0). Enables lysot factories (markdown design → autonomous assembly), wet labs (automate chem/bio/med discoveries).
- Physical auto research: robots design/improve/build next iterations beyond humanly possible.
Timeline: AI from AlexNet 2012 (cat vs dog) to AI ascent 2026 took 14 years; add 14 more to 2040 for agentic auto research. Exponential advance.
With 95% certainty, end of endgame by 2040.
If you believe in robotics, robotics will believe in you.
Our generation born too late to explore Earth, too early for stars, just in time for robotics.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
