Robots Are Finally Starting to Work
In a Nutshell
Physical Intelligence (PI), co-founded by Kuang Vang, is building foundation models for robotics—the "GPT-1 moment"—enabling a single model to control any robot for any physically possible task via scaling cross-embodiment data, cloud inference with real-time chunking, and open-sourcing models like PI-0. Key breakthroughs include emergent generalization from projects like Open X-Embodiment (50% better than specialists), real-world deployments with partners like Weave (laundry folding) and Ultra (warehouse logistics), and overcoming data scarcity through mixed autonomy and scrappy hardware. This unbundles vertical integration, slashing startup costs to spark a Cambrian explosion of robotics companies targeting every menial job with quick data collection, economic break-even, and incremental scaling.
These notes were generated by AI and may contain inaccuracies.
The equation for starting a robotics business has changed and will continue to change at an accelerating pace because the upfront cost is not that high anymore. Everyone's spending a lot of time in the digital world and now is the time to start thinking about the world of atoms. This is the playbook for how to build a vertical robotics company. The mission from the start is to create that Cambrian explosion.
Welcome to another episode of the Light Cone. Special guest: Kuang Vang, co-founder of Physical Intelligence (PI), the robotics AI lab that might bring the GPT-1 moment for robotics. Mission: build a model that can control any robot to do any task that is physically capable of, at a high level of performance useful to people in all walks of life. GPT-1 for robotics is the ChatGPT moment for robotics. Perspective: build a really intelligent model and a platform to externalize that intelligence to the world for building interesting applications in all sorts of verticals in robotics.
Peeling an onion analogy: Start from a strong base model with common sense knowledge that works to some extent on your robot. Then a mixed autonomy system, similar to autonomous driving cars today. Deploy to do a real job; it's okay if it makes mistakes. Over time, expose to real-world complexity and edge cases; system gets incrementally better every day. One day, fully autonomous system providing tremendous value.
Robotics problem simplified to three pillars: semantics (unlocks from language models ported into robotics), planning, and control (real-time interaction with changing environment).
Seminal papers from PI robotics team starting in 2024 signaling GPT-1 moment near:
- SayCan: First demonstration of language model bringing common sense knowledge into robotics, reducing need for robot-specific data. Example: Ask language model steps to go to YC office to record a podcast; it shows the plan incredibly well. Language models infiltrate robotics starting at planning/semantic level.
- Still control problem: Need mechanism to convert plan into low-level actions to actuate robot.
- PaLM-E and RT-2 (Robotics Transformer 2): Start from powerful vision language model, adapt with robotic data to "speak robot language." Shows transfer from vision-language knowledge to low-level actions.
- Favorite example: Picture of celebrity on table (Taylor Swift, Queen of England); ask robot "pick up the Coke can and move it to Taylor Swift"—works even though Taylor Swift not in robot data.
- Spatial reasoning: "Move the dinosaur next to the red car"—unseen objects in robot data.
RT-2 and PaLM-E are single-embodiment exercises (work for very specific robot).
In robotics, scaling question: How to scale data collection? Insight: Data from one robot not that different from another. If enough robots in training data, model learns abstract general notion of controlling any robotic platform, better at any particular platform.
- Open X-Embodiment and Robotics Transformer X (RT-X): First paper showing potential scaling laws in robotics by training across multiple hardware kinds—never done before. Research labs previously trained on specific sensors/actuators/motors, very finicky.
Key result: Take 10 different robot platforms, collect data, train specialized policy optimized for each (10 policies). Absorb data into high-capacity generalist model. Generalist 50% better than specialists.
Open X possible due to huge collaboration across robotics community. Joke in robotics grad school: Add two years to PhD by working on new robot platform (takes 1-2 years to get platform up and running for data collection). 10 platforms = 20 years.
Open X dataset impact similar to ImageNet for vision (huge, first large dataset across multiple hardware, collaboration), but ImageNet more impactful because it enabled reproducible evaluation. Open X made data available but didn't solve evaluation (hard in robotics). Open X now a drop in the bucket compared to current community data scale/volume/diversity.
GPT-1 moment: Alec Radford proved scaling laws with neuron on specific input/output. Biggest robotics problem: data scarcity.
- Language bootstrapped off internet sum total (quite a lot). Robotics lacks internet of data.
Data scarcity two problems:
- Data generation.
- Data capture: Lots of robotic data generated but no incentive to capture/digest for training. Open X goal: Capture existing data.
Robotics data collection: Operationally heavy. Will it scale? Napkin math: Solve robotics (model controlling any robot/any task) contributes 10% to $24T US GDP—massive, warrants investment.
Focus on cross-embodiment: Infrastructure to consume data from many robot sources scales easier than optimizing single hardware platform (harder to manufacture 1000 units). No company scaled that way.
Inventory shocked: No two robot platforms the same. Debate multi-robot vs single-robot: Single simpler to scale? Not in practice—even single platform drifts (hardware/software changes every 3 months), hard to reuse old data. Need many samples from distribution to generalize.
Hypothesis: Many platforms in fleet → model learns abstract "how to control a robot, not any particular robot" → ingests data from slightly different robots better.
Seeing emergent properties in robot large production models: Transfer between data sources. Zero-shot tasks (no data collected) that last year required hundreds of hours.
- Examples: Precision tasks, reasoning with multiple objects—general property, not luck on one test.
Current state: For tasks where robot mistakes okay and mixed autonomy possible (human takeover/corrections), performance level makes scaling deployment sensible.
Partnerships with YC companies Weave and Ultra (blog post):
- PI primarily research-focused on best model, but partners with deployment companies. Treat as same team, free info flow, design for best performance on their tasks.
Weave Laundry Folding
Video: System folding diverse laundry items in real laundromat (people walking outside). Difficult: Infinite observation space, deformable clothing, unseen in training data.
- Weave shipping first robots into homes for household tasks. Inspired by PI's first laundry demos.
- Took ~2 weeks from goal to good-enough model/system.
- Laundry folding: Turing test for robotics (pre-AI, no deterministic programming due to infinite space). Now possible; robots can do everything humans can, matter of improvement.
- PI-Zero publish: People called PI "laundry company." Home tasks (deformable objects) intentional: Relatable, easy to test generalization.
Ultra Logistics Task
Video: 100 minutes at 4x speed, bright outside to sun set. Robot insensitive to lighting/environment changes.
- Ultra: Makes adapting robots to new tasks easy, focusing on logistics (labor shortage).
- Task: From Amazon soft pouches—pick item from tray, place in pouch (machine closes), put pouch ready for shipping. Real e-commerce warehouse, real products to real customers.
- Hard: Many object types, narrow opening → robot nudges item in (precise motion, scene understanding). Runs full day with minimal human intervention—autonomy at scale, not demo station.
Approach shifts from difficult engineering per task to operational: Identify use cases, collect data scalably across many tasks.
Past: Robots need real-time on-device compute.
Companies ask: What compute unit on robot? Expensive, increases BOM cost, goes out of fashion as models change/get bigger.
PI insight: Almost all robot evaluations (complicated demos: coffee, laundry, mobile navigation) hosted in cloud (real data center, not office server). In high-frequency control loop, robot queries API endpoint hosting model, sending images and language command
PI couples system hardware and model development tightly to solve integration challenges. Key insight: bury inference time within the robot control loop. Robots have enough actions to execute for the next 100 milliseconds, so no need to wait until execution finishes to query the model. Query for next actions when 50 milliseconds remain, ensuring seamless continuation.
Real-time chunking designs inference to handle cloud query delays. An action chunk is a sequence of actions executable for 100 milliseconds. At 50 milliseconds in, predict the next chunk for transition after current one ends, ensuring consistency (e.g., smooth continuation of motion). Solution: precompute (preemptively compute) next chunks.
You can preempt compute and that's one of the algorithmic improvements that we've made to make inference using model hosted in the cloud possible.
This simplifies robot systems, avoiding clunky dual operating systems, embedded OS, complex compute, and power needs (e.g., early Waymo ran servers in the trunk, unaffordable for general-purpose robotics).
Much compute can happen in the cloud; some local compute is needed. The top-left video feed appears to be a dumb camera streaming to the cloud with minimal local processing. PI's collaboration with Weven Ultra: speaker has never seen the robot in person, knows little about its workings or data collection—intentional to test parachuting into existing systems without deep hardware knowledge, decoupling hardware control from semantics and planning.
Initially expected real deployment only 5 years in; at 2 years, scaling robot numbers is a serious consideration. Pace of progress faster than expected.
Robotics traditionally hard due to vertical integration: own customer relationships, hardware, autonomy stack, safety certification—high barriers.
PI provides foundation of physical intelligence for community to onboard autonomy quickly, focusing on other problems.
Recipe for Vertical Robotics Business
- Deep understanding of existing workflows; robot must fit in.
- Meticulous identification of insertion points for biggest impact.
- Scrappy hardware and data collection: no need for expensive, precise robots—models are reactive, compensate for inaccuracies.
- Collect data and run real-deployment evaluations.
- Build mixed autonomy system to reach economic break-even, enabling scaling (historical challenge: long payback periods prevent growth).
Upfront costs now lower: cheaper hardware, data collection/evaluation, use case understanding. No need for proprietary classical autonomy stacks. Focus on differentiation.
Unbundling vertical integration enables Cambrian explosion of companies like Ultra targeting every menial job: deep customer understanding, custom robots, mixed human-machine to full autonomy, per sector.
I believe there's going to be a Cambrian explosion of robotic company across the entire world and across many different vertical just because it's just so much cheaper to build and it doesn't require someone with 20 years of experience in robotics to start anymore.
Requires scrappy, quick system integration and customer focus.
Analogy to personal computing: industrial robotics like mainframes/minicomputers (e.g., 1970s Digital Equipment Corp)—expensive, specialized, enterprise. Progress to personal computing via Altair, Apple I/II, IBM PC/XT. Traditional advice: dirty/dangerous jobs (e.g., Tesla Gigafactory). Profitability enables non-extreme cases first.
This is already happening: many companies seeking autonomy, okay with mistakes, desperate for robots. Playbook works across verticals.
Identify where the robot can fit in, focus on cheaper hardware, collect data, run evaluation, mix autonomy, break even, scale robots.
PI wants Cambrian explosion; failure if robotics takes 50+ years. Publish research, open-source PI-0 and PI-05 (same pre-trained model weights as internal use—no differences).
Success: models on any robots performing useful tasks, even unknown ones, benefiting end consumers. Focus on cross-embodiment.
Larger-than-average founding team from Google Robotics team (great for ideas, relationships). Met Locky while starting company (business expertise). Adnan (hardware lead from Anduril) handles cross-embodiment fleet: building/improving/scaling heterogeneous robots—divide and conquer.
Co-founders: Brian, Chelsea, Sergey, speaker (Quan), Locky, Adnan.
We really enjoy each other's company... any one of us could have started a company and be successful but the problem is just so incredibly hard and the chances of success is just so much higher that we band together and we can divide and conquer.
First startup for many; enjoy work relationships.
Surprises in Startup vs. Academia/Big Tech
No infrastructure for large-scale general-purpose robots: data collection devices/management/annotation/visibility, evaluation, operational processes. Built much software in-house. Opportunity: services like remote teleop, data collection/annotation (don't repeat per company).
Tight collaboration loop: task selection, data collection hardware/quality/visibility, training, evaluation (scales super-linearly, e.g., 2-min vs. 20-min task >10x harder), distilling learnings to improve.
Side project idea: automated robotic research scientist—ingests multimodal data, analyzes failures (data/annotation/training causes), suggests/tests hypotheses.
Joke: record meetings, train model for predictions.
LLMs help simple failure modes if precisely described, but limited: not core models taking physical actions/seeing consequences. Missing: fundamental physical world understanding.
Internal: Claude (skill?) as pre-training on-call babysits large runs, remedies errors—50% compute utilization improvement.
PI focused on giving foundation models to world for Cambrian explosion. Robotics costs decreased.
Costs will continue to dramatically decrease and require a very different kind of scrappy skill set that young startups need. The goal is to enable an explosion of many different robotic use cases. Always reach out if you want to collaborate.
Thanks man. Thanks so much. Thank you.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
