Chelsea Finn: This is the State of the Art in Robotics
In a Nutshell
Chelsea Finn's Physical Intelligence has developed general-purpose robotics models that achieve 90%+ reliability on real-world tasks like espresso-making and kitchen cleaning through a self-improving system combining foundation model pre-training, human intervention for dead-end avoidance, and reinforcement learning using a value function trained on diverse robot experience. The PIO7 model demonstrates out-of-the-box performance matching or exceeding task-specific fine-tuned specialists, with compositional generalization across robot platforms and objects never seen in training data. The approach enables long-term autonomous operation (13+ hours) without human supervision by implementing multi-time-scale memory systems that compress extended task histories into actionable text summaries.
These notes were generated by AI and may contain inaccuracies.
Chelsea Finn founded Physical Intelligence two years ago to develop robots capable of performing any task in the real world. One year prior, she presented progress on complex tasks including unloading and folding laundry, and demonstrated robots successfully completing useful tasks in previously unseen environments.
Since the previous presentation, robots have achieved additional tasks including washing a greasy pan, peeling a carrot, making a grilled cheese sandwich, and slicing a zucchini. The focus shifts from showcasing individual capabilities to understanding what enables robots to become useful in real-world applications.
Two critical aspects must be addressed: developing general-purpose models and bringing these models into the real world for actual impact. The discussion begins with the challenges of real-world deployment.
A timeline of major production launches leveraging machine learning reveals key milestones. Early applications included product recommendations and ad ranking. Five years later, deep learning emerged for similar applications, offering advantages because the algorithm could be applied out-of-the-box to complex input-output scenarios. The pivotal moment occurred in 2022 with ChatGPT, the first general-purpose model used by many people in the real world, reaching one million users within five days. More recently, coding agents like Cursor have demonstrated real-world utility.
Generalist models are increasingly deployed for real-world problems. However, a critical distinction exists: in existing machine learning applications, customers make final decisions based on AI recommendations, making occasional mistakes acceptable. Physical AI systems must make decisions that directly affect the physical world, requiring far fewer mistakes and operating fully autonomously.
One year ago, Waymo passed a quarter of a million weekly autonomous rides, demonstrating that machine learning-based systems can operate reliably and autonomously in the physical world. This provides optimism for broader physical AI applications.
For robots to be useful, they must operate autonomously for extended periods rather than relying on humans to interpret model predictions. The espresso-making task serves as a concrete example: a robot must reliably insert the portafilter with precise and forceful control, handle cups with liquid without spilling, and maintain accurate timing. Over 90% reliability is required to avoid constant human supervision.
The standard machine learning process involves collecting datasets, training models, and evaluating performance. Initial attempts rarely achieve reliability. The practical approach involves iteration: collecting more data, improving label quality, making labels more detailed, gathering edge case data, and adjusting dataset balance. Manual iteration eventually reaches limits due to human fatigue.
The optimal approach enables the AI system itself to iterate automatically, seeking out scenarios requiring more data or supervision. This mirrors reinforcement learning algorithms that learn from failures and improve autonomously. More iterations become possible because the process is automated rather than manually driven.
Language models use algorithms like PPO and GRPO that scale to large language models through millions or tens of millions of attempts. Translating this approach to robotics presents significant challenges: one million trajectories of a one-minute robot task would require approximately 700 robot days. Unlike language models running on compute clusters, robotics requires physical hardware operation in the real world.
Reinforcement learning algorithms for language models spend considerable time on dead-end trajectories, which is acceptable when using compute but costly in physical robotics. For example, when constructing cardboard boxes, a robot might grab two flush boxes and attempt to fold them together rather than separating them first. Instead of continuing unproductive attempts, human intervention demonstrates recovery actions, allowing the robot to learn efficient paths and avoid wasting time on incorrect approaches.
PPO and GRPO algorithms make multiple attempts per prompt to estimate good versus bad responses. This cost can be amortized by training a general-purpose value function on diverse robot experience videos. The value function learns to predict what's good or bad across scenarios—for instance, recognizing that accidentally unfolding a shirt represents negative progress, or identifying forward progress when retrieving items from a fridge. This general value model predicts time-to-success and significantly reduces attempts needed for improvement.
The complete algorithm trains a foundation model on diverse data, collects experience with human intervention to prevent dead-end trajectories, trains a general-purpose value function, and uses this to improve the model. This enables fine-tuning foundation models to higher performance levels.
The model directly controls robot joints using camera images as input. The system successfully inserts the portafilter, waits the appropriate time for espresso dispensing, pours steamed milk, and transfers a full latte cup to a coaster with delicate balance control. The policy ran continuously for 13 hours, demonstrating reliability sufficient for real-world usefulness over extended periods.
The same algorithm was applied at Dandelion Chocolate Factory to construct, label, and stack cardboard boxes following their actual workflow. The reinforcement learning approach was also applied to folding and closing tasks across multiple previously unseen environments, with robots operating autonomously for extended periods.
Throughput measurements combining success rate and speed showed a drastic increase across training phases: pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL) post-training. The RL stage alone provided approximately 2x throughput improvement. For the espresso task, over 90% success rate was achieved.
The approach enables long-term autonomy in real workflows that matter to people. However, only a few iterations of the algorithm were run, leaving substantial room for further improvement. Current limitations include remaining mistakes and slower performance compared to humans.
Most state-of-the-art robotics foundation models lack memory or context, operating only on current sensor observations. Short motor skills and repetitive tasks can function without memory, but long tasks involving sequential steps require memory to track completed progress.
Naive memory approaches face computational barriers. Passing 10 seconds of video at 50 Hz across four camera streams with 256 tokens per image requires approximately 500,000 tokens. Even subsampling to one frame per second yields 10,000 tokens, which remains prohibitively expensive.
A solution implements memory at multiple time scales: short-term video memory covering approximately 10 seconds computed efficiently, and longer-term memory spanning minutes to hours represented as text summaries. This compressed textual representation of 10-15 minutes of history is incorporated into the model.
With multi-scale memory, robots can perform non-repetitive 10-15 minute tasks autonomously. A kitchen cleaning sequence includes wiping counters with a sponge, drying with paper towels, disposing of towels, storing mustard in the fridge, putting dishes in cabinets, and washing dirty dishes—requiring tracking of multiple sequential steps.
The goal is creating a single general-purpose model incorporating all previously demonstrated capabilities plus additional functionalities. The evolution of generalist AI provides context: the 2012 ImageNet breakthrough demonstrated deep learning outperforming specialist systems, 2014 saw pre-trained models become standard for fine-tuning, and subsequent transitions moved toward out-of-the-box generalist model usage.
In 2021, the first signs of compositional generalization appeared, exemplified by Dolly. Physical AI three years ago (2023) commonly involved collecting bespoke datasets and training from scratch for each project—analogous to creating ImageNet from scratch for every application.
The goal is to develop a single general purpose model that works out of the box and shows compositional generalization. The first objective involves out-of-the-box models, analogous to the transition from BERT to GPT. Currently, achieving optimal robot performance on specific tasks always requires fine-tuning. Videos showing tasks like unlocking a lock, human-to-robot transfer measurements, and reinforcement learning post-training for tasks like making espresso all required fine-tuning for individual tasks. Fine-tuning prevents achieving a true general purpose model since each task requires separate fine-tuning.
The second goal is compositional generalization, inspired by the Dolly result from 2021. This milestone was significant because when models demonstrate compositional generalization, they show conceptual understanding of individual concepts like "avocado" and "chair" and can combine them into novel outputs exhibiting both concepts simultaneously. This also indicates data efficiency since the training data doesn't need to cover all possible concept combinations. Even though Dolly wasn't perfect in 2021, it demonstrated these two key model attributes.
The established approach for developing these models involves using sufficiently large and diverse datasets combined with models of sufficient capacity. The training approach uses all available data including diverse robot demonstration data with varying quality levels, policy rollout data from reinforcement learning attempts on previous tasks, videos of humans performing tasks, and web data. For model capacity, training requires sufficiently large models, but prompting becomes particularly important for handling heterogeneous data types.
The key innovation involves prompting the model with all necessary context for action prediction. The foundation model takes as input memory, task instructions, subtask instructions indicating the next immediate action, and metadata about data quality, episode lengths, and other characteristics. Optionally, sub-goal images can be included as prompts indicating what the robot should achieve in the coming seconds. This detailed prompting enables the model to utilize much more heterogeneous data effectively.
Deploying the model requires providing subtask construction and sub-goal images. A high-level policy is trained to predict the next subtask instruction, while a world model generates images for sub-goal image conditioning. This approach enables training a single model on diverse data, resulting in the PIO7 model capable of performing tasks like folding collared shirts, precise assembly steps involving screw insertion and drilling into robot arms, and replacing trash bags in trash cans.
The PIO7 model demonstrates out-of-the-box performance that matches or exceeds fine-tuned specialists. When measuring throughput and success rate, the single pre-trained PIO7 model matches or outperforms fine-tuned PIO6 models that used reinforcement learning post-training for specific downstream tasks. This performance parity also holds when comparing against supervised fine-tuning specialists, suggesting a single model can achieve high performance across diverse tasks without task-specific fine-tuning.
Compositional generalization was tested through multiple approaches. The first test involved interacting with rarely encountered appliances like air fryers, specifically testing whether the model could open an air fryer, place a sweet potato inside, and close it. Although the dataset contained only three episodes with air fryers, the robot successfully interacted with this barely represented appliance, combining the skill of operating it with previously unseen objects. A high-level policy can be trained to perform this task fully autonomously.
The second test examined compositional generalization between tasks and robot platforms using a BARM robot, a large industrial platform. Despite having no folding data collected on this platform, the model successfully performed shirt folding using only data from folding clothes on different robot platforms. The robot differs significantly in size, linkage lengths, and joint configurations, yet the model demonstrated successful compositional generalization. The generated sub-goal images shown in the top left guide progress on the folding task.
Performance on folding towels and shirts on previously unseen platforms increases dramatically with more advanced models like PIO7, approaching human teleoperation performance despite lacking robot-specific training data for folding clothes. Ablation studies examined the importance of diverse data and detailed prompting. Removing the most diverse data dramatically decreases performance on held-out tasks, while removing a random 20% of less diverse data only slightly decreases performance, indicating diverse data plays a crucial role in generalization.
Metadata prompting significantly improves performance. When adding lower quality data from 80% to 100% of the dataset without metadata prompting, performance decreases. However, with metadata prompting, performance actually increases when adding low-quality data, suggesting the prompting enables extracting more value from even lower quality data sources.
Physical intelligence has reached a GPT and Dolly-like era for robotics within just a few years. Models are now deployed in real-world circumstances, with YC companies Ultra and Weave using PI models post-trained for tasks like folding laundry and warehouse packaging. The models work across diverse robot embodiments including standard bi-manual platforms, drones, quadcopters, surgical robots, and tractors, demonstrating physical intelligence impact beyond research demonstrations.
The distribution channel for physical models will likely be slower than ChatGPT's rapid user growth because physical robots are required for deployment. The rollout of systems like Waymo demonstrates that physical device deployment takes considerable time. While a single moment with ChatGPT-like distribution may not occur, model capabilities are reaching the point of real-world usefulness, with ChatGPT-level capabilities expected within the next few years.
Starting with generalist policies and fine-tuning them immediately can be highly effective. Open-source generalist policies like PI Zero and PIO5 models are available and have seen significant use. Partners including tractor companies, Ultra, and Weave are developing more powerful models for their applications using recent PI models. The only scenario where generalist policies might not be suitable involves extremely constrained environments, such as surgical robots in operating rooms with no internet connection and limited GPU resources, though local inference remains possible on workstations.
A PhD provides an opportunity to learn handling uncertainty and selecting good problems to work on, since research requires independently identifying problems and determining their achievability across different time spans. This skill becomes particularly valuable in startup environments and at the AI frontier where optimal routes for model improvement remain unknown. The PhD also offers freedom to pursue personally exciting research directions.
Industry opportunities exist across the entire technology stack including software infrastructure, hardware development, machine learning infrastructure, and data infrastructure. A PhD is not necessary for much of this engineering work, though research opportunities also exist in industry settings. The decision remains highly personal, with both paths offering significant opportunities.
The robotics equivalent of internet-scale data involves robot experience operating in real-world circumstances. Physical Intelligence collects robot experience across diverse tasks through teleoperation for initial data, with future data including autonomous robot experience from deployed systems attempting tasks. Similar to how language models now generate synthetic data, robotics data will increasingly come from robots attempting tasks in varied real-world conditions.
Additional useful data sources include videos of people performing tasks like YouTube content, web data, and captioned images providing contextual information. However, robot experience on actual platforms remains irreplaceable since watching humans perform tasks doesn't enable robots to replicate those actions directly. Large datasets will be necessary, with actual robot platform experience serving as a critical component alongside human video data.
The cost of embodied data and hardware could significantly impact model democratization compared to large language models. Obtaining data for even model distillation presents greater challenges in robotics due to hardware requirements.
Getting data for robotics is harder compared to language models because it's not readily available on the internet. Large datasets have been open sourced along with powerful models. It's difficult to predict exactly what will happen in terms of open source development. Companies focusing on closed source models are also doing significant open sourcing, with examples like Gemma and GBT open source. Companies support open source because it helps build the ecosystem around their products. There's optimism for a strong open source community in robotics, though it may not follow exactly the same path as language models.
All models shown output target joint positions specifying the angle for each joint. A PD controller then attempts to reach those target joint positions. The model is also trained to predict target gripper positions in 3D space, which can be used to back out joint positions. Alternative approaches include outputting motor torques directly or voltages and efforts. Controlling joints and controlling in the 3D space of the gripper both work well. Going directly to voltages allows for more or less stiff output, whereas fixed controllers don't allow the model to control that aspect. The current approach doesn't appear to be a bottleneck.
The PIO7 model can imagine what a future image should look like and then attempt to accomplish that goal. This capability leads to quantitative improvement, as demonstrated in the shirt folding example. However, the model performed surprisingly well even without this imagination capability. Being able to predict the future seems like a relevant objective compared to predicting future actions, which should help with learning from available data. It remains unclear whether imagination will be a critical component, though empirically it appears to help, and even without it robots can perform impressive tasks.
Speed improvements have been observed from reinforcement learning. The RL token release demonstrated even faster speed than human teleoperation. A major bottleneck is that human teleoperation, the easiest way to teach robots, tends to be slow. Several projects are in development with promising results for getting fast policies. To improve speed, either make the data faster or figure out how to be faster than the data. Evidence shows it's possible to be faster than the data, with next steps being to go further or make data faster.
During PIO7 development, a policy was trained for assembling a pin wheel where all training data involved picking up the pin with the right hand and the paper with the left hand. The robot initially followed this pattern but then made a mistake, ending up with the paper on the right and the pin on the left. The robot then picked up the paper and used its left gripper to pick up the pin and insert it with its right hand, despite never seeing data of inserting the pin with the left gripper. This demonstrated emergent equivariance between left and right hands, transferring behaviors despite no such data existing in post-training or pre-training datasets. This represents an emergent capability not previously observed.
Rather than focusing on individual tasks, emphasis is placed on capabilities and how to derive the next capability from models. Recent work involves robots using knives to slice vegetables, which opens possibilities once safe knife usage is achieved. The most exciting aspect is seeing robots accomplish any task, even if not done before.
There is substantial software engineering work within robotics, making it possible to join robotics companies as a software engineer. An example career path involves Jenny, who worked in algorithmic trading and then at Harvey doing legal work before becoming interested in robots. She purchased a cheap robot and experimented with it in her bedroom, fine-tuning an open source model. After sharing her work and sending a cold email expressing interest in working in the lab, she demonstrated genuine excitement and hands-on experience. She now works at Physical Intelligence. The recommended approach involves getting hands-on experience, learning from it, and using that experience to share with others and include on resumes. Abundant open source resources are available to get started.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
