Why Robotics Still Isn't Solved - But Could Be Soon | YC Paper Club
In a Nutshell
The core message is that robotics faces four fundamental scaling barriers—physics modeling gaps, deformable object dynamics, sensory-motor deficits, and embodiment drift—preventing the "next year" predictions that have persisted for a decade. The most important breakthroughs presented include memory-augmented policies that decompose tasks into high-level text memory and low-level visual execution, self-supervised bootstrapping of embodiment-specific reasoning that prunes useless annotations while improving out-of-distribution performance, and simulation-trained goal-conditioned policies that achieve zero-shot tool manipulation without teleoperation data. These advances collectively suggest that solving robotics requires architectural decomposition and selective reasoning rather than pure scaling of existing vision-action models.
These notes were generated by AI and may contain inaccuracies.
The discussion opens with a 10-year history of robotics being predicted as solved 'next year.' AlphaGo's release prompted claims that scaling the algorithm would solve robotics. MuJoCo enabled training robots to walk in 3,000 iterations, reinforcing the prediction. The ALOHA system was hailed as a breakthrough, with demonstrations of watering plants, fixing bikes, and operating a Keurig. The diffusion policy paper and VALAS were cited as evidence that 2026 would be the year of robotics. However, halfway through 2026, only pre-orders for Neo 1X are available, with no consumer access to Pi or Figure robots yet.
Four primary barriers to scaling robotics are identified. The first is physical real-world modeling. Video models trained on games like Doom fail to respect physics when deployed in real-world scenarios, such as driving a car into a grocery store that magically transforms into a highway. The sim-to-real gap remains unsolved.
The second barrier involves deformable objects and action-conditioned dynamics. Estimating the transition function from state to state-plus-one becomes significantly harder when conditioned on actions, requiring substantially more data. Feature pyramid networks developed in 2013 for robotic policy bot war simulations addressed representation learning, but action space representation remains unsolved for rapid learning.
The third barrier is the sensory-motor gap. Humans possess nerve endings detecting normal force, tangent force, moisture, temperature, vibration, and friction coefficient across the entire body. Current robots have at most one coin force-torque sensor per fingertip and a wrist camera. Neuroscientists note humans build world models without vision, such as identifying backpack contents by touch alone. The absence of an artificial epidermis prevents robots from achieving comparable tactile capabilities.
The fourth barrier is embodiment drift. Actuators accumulate dust, corrosion, and rust, particularly in marine environments. Toyota Prius throttle response varies over time, and battery power output degrades. These changes render teleoperation data stale, requiring complete retraining and new data collection.
Marcel, a Stanford PhD student and former Physical Intelligence intern, presents Multiscale Embodied Memory (MAM). Milan Gennai, a Stanford PhD student affiliated with Marco Pavone, Clark Barrett, and currently at Waymo, discusses self-supervised bootstrapping of embodied reasoning. Tyler, a PhD student with Janette and Karen, is introduced. Nico, a friend from 2012 double EE master's program and founder of Rerun, addresses data deployment practicalities. Bill and Guan Ming, founders of General Instinct, a Y Combinator company, present real-time world action models with prior DeepMind experience.
Physical Intelligence trained policies for robot Olympic tasks including unlocking blocks, folding inside-out clothes, and making peanut butter sandwiches. These tasks last approximately 2 minutes. Longer-horizon tasks such as manufacturing, cleaning entire bedrooms, or cooking full meals with kitchen cleanup require memory integration.
Current policies including PIO5 and Groot lack memory, processing each timestep without historical context. Demonstrations show memoryless robots washing dishes indefinitely and burning grilled cheese due to absent temporal awareness.
Adding memory creates effectiveness and efficiency problems. Distribution shift and data scarcity cause performance degradation. Extended context increases training duration and inference latency.
The proposed architecture decomposes policies into high-level and low-level components. High-level policy determines next steps; low-level policy executes actions. Memory splits into short-context dense visual frames for dexterous manipulation fed to low-level policy, and long-context compressed language representations of recent minutes fed to high-level policy.
The encoder modifies ViT architecture by incorporating temporal attention layers while dropping all tokens except the current frame. This preserves necessary information through temporal attention while achieving substantial compression. Benefits include straightforward ViT weight initialization, reduced token count, and accelerated inference.
Flagship demonstration: autonomous grilled cheese preparation requiring appropriate waiting periods to prevent burning. Additional tasks include unloading groceries from bags using wrist camera views requiring memory of item locations and quantities, and window cleaning without indefinite repetition.
High-level policy generates recurrent memory scratchpad predictions tracking task history. Memory predictions feed back into the policy for continued context. Text representations require fewer tokens than images, improving training efficiency. The policy can dynamically update memory based on observed events, reducing distribution shift susceptibility.
Long-horizon demonstration: preparing recipe ingredients over tens of minutes. Memory string predictions display in green boxes during execution. The system outperforms baselines including no-memory and alternative memory configurations.
Memory enables in-context adaptation absent in current vision-action models. Memoryless policies repeat identical mistakes indefinitely, such as attempting to open a refrigerator or grasp chopsticks without adjustment. Memory-augmented policies observe errors and modify subsequent actions, such as lowering gripper height or switching refrigerator sides.
High-level policies trained via supervised fine-tuning require manual data annotation specifying information to track. Future directions include reinforcement learning in memory space to autonomously determine relevant information. During inference, short text memory remains in RAM and feeds as standard text tokens to the vision-action model.
Task descriptions include detailed specifications of item locations and identities. Vision-language models pretrained on internet data enable generalization with limited robot data. Low-level policies receive concise text descriptions of immediate actions.
Text-based memory imposes representation constraints compared to latent embeddings. However, text provides strong supervision bias, debuggability, and practicality. Human memory likely operates beyond text space, suggesting future latent embedding exploration.
Vision-action models continue training vision-language models including Gemini or Qwen on scarce robotics datasets comprising teleoperation or self-driving recordings. Models accept image or perception features plus language instructions, outputting executable actions such as steering commands or end-effector positions.
Embodied reasoning augments limited robotics data through chain-of-thought style annotations. Text-based reasoning enables human-readable decision traces for debugging and understanding robot choices.
Two core problems exist: grounding problem lacks oracle reasoning data sources comparable to internet text or image captions for robotics; model source problem prevents accessing human decision-making processes from perception to motor commands. Verbosity concerns include whether step-by-step planning or exhaustive object listing creates unacceptable latency in robotics applications.
R&B encore stands for refine and bootstrap embodiment specific chain of thought reasoning. This is a self-improving pre-training cycle for embodied reasoning vision-action models. The insight treats reasoning as an unobserved latent variable for observed context and action, leveraging variational inference as the theoretical framework.
Two high-level components exist in this approach. The first is a reasoning proposer, which acts as an annotator model proposing various types of reasoning for a given context or demonstration. It evaluates whether visible object and move reasoning, gripper position reasoning, plans reasoning, or visible object and subtask reasoning constitutes useful annotation data before actions. The second component is a reasoning validator, which serves as a scoring metric based on variational inference theory.
The validator score contains three main parts. The first is concision, ensuring reasoning traces remain short and not verbose. The second is non-trivial, encouraging generalization in reasoning behavior. The third and most important is action predictiveness, which ensures the reasoning trace or annotation is grounded in that embodiment.
All reasoning traces are scored and resampled to produce a new dataset of synthetic but action-aligned and embodiment-aligned reasoning. This reasoning dataset enables analysis of which reasoning types are effective, and more importantly allows retraining the embodied reasoning vision-action model to produce a better and more robust policy.
The pre-training cycle was tested on multiple embodiments. For manipulation, embodied reasoning manipulation vision-action models were pre-trained, finding that move reasoning and gripper position reasoning proved very useful while perceptual reasoning was not useful and could be pruned out. This improved success rates through action-aligned reasoning.
Perceptual reasoning proved ineffective due to distracting objects in many scenes. Pruning through this cycle improves annotation data quality by focusing on task saliency of traces and data used for collection and annotation, improving both success rate and object criticality rate.
Hardware manipulation vision-action models tested with this cycle showed improved out-of-distribution performance on novel target objects and cluttered scenes. Legged locomotion navigation models were also pre-trained, finding that reasoning about structural affordances and movements proved more important than reasoning about terrains and counterfactuals.
The cycle was applied for refining human annotations for self-driving, where substantial garbage data exists. The approach removes human annotations that are not useful, pruning hallucinated experiences while preserving meta action, invisible objects, and perceptual reasoning. This resulted in lowered L2 path errors and improved performance metrics including collision rates.
Three main takeaways emerged. First, selective reasoning is more important than exhaustive reasoning, as even valid reasoning is not necessarily useful. Second, self-supervised bootstrapping works, addressing the chicken and egg problem of needing an oracle source of model and reasoning data. Third, the approach generalizes across embodiments including manipulation, navigation, and driving, as well as vision-action model sizes from 1 billion to 30 billion parameters.
The core argument is that embodied reasoning is not a fixed template applied uniformly but rather a resource that should be discovered, deployed, and budgeted carefully. Areas of interest include data quality, recovering from failures, adapting to novel scenes, and specializing in them.
Questions addressed non-textual reasoning capabilities where models navigate continuous space of understanding unhindered by textual format, requiring good architecture and theory for developing continuous reasoning. The approach leverages priors from large language models and behavior language models, where massive data exists, for data-scarce regimes like robotics.
For autonomous vehicles, the streaming latency problem is addressed by using reasoning to introduce priors into the model through co-training data. Textual reasoning serves as good annotations alongside robotics demonstrations. Action forcing allows dropping reasoning during inference time, eliminating inference latency while still extracting enhancements from textual annotations and reasoning.
Reasoning is based on the current image and current instruction, so state space drift where reasoning becomes irrelevant is not a fundamental problem. Regarding pruning counterfactuals, certain reasoning types do not need to happen at every moment. During self-driving where nothing novel occurs, such as driving in a straight line with no interesting counterfactuals to explore, planning at every single step would be redundant.
Tyler Lum presented Sim Tool Reel. A single policy works zero-shot across tools and tasks never seen during training, including brushes, hammers, and dextrous tasks like screwdriver spinning that require multi-fingered hands rather than parallel jaw grippers. The policy runs at 60 Hz controlling both a 22 degree of freedom hand and 7 degree of freedom arm.
Teleoperation for dextrous hands has been used to collect demonstrations for imitation learning, but highly dextrous actions remain difficult to demonstrate reliably and at scale due to embodiment mismatch and limited force feedback. Rather than learning from teleoperation, the policy trains entirely in simulation using sim-to-real reinforcement learning.
GPU-accelerated simulation runs tens of thousands of robots in parallel, generating experience approximately 1,000 times faster than real time. This scales data collection with compute rather than human effort, collecting decades of interaction data in days. The result is extreme dexterity through optimizing for reward-maximizing behaviors rather than imitating demonstrations.
Prior works learn separate policies for each skill, requiring new reward design, task-specific engineering, and training rounds. The goal is training a single policy that controls both hand and arm through full sequences like grasping a brush, reorienting it within the hand, and using it to sweep objects.
The key insight unifies dextrous tool manipulation as goal reaching. The policy needs no task label such as sweeping or hammering, only moving an object from its current pose to the desired pose. A goal-conditioned policy trains to move arbitrary objects through sequences of desired goal poses.
At training time, primitive objects are procedurally generated in simulation, random goals are sampled, and a goal-conditioned policy trains with massively parallel reinforcement learning in simulation. At inference time, the policy conditions on human video demonstrations. Foundation Pose and SAM extract sequences of goal poses to track, which the reinforcement learning policy follows one by one in a 60 Hz control loop.
The human video provides no robot actions and is not used to train or fine-tune the policy. It only specifies the desired object trajectory for the frozen policy to track. The policy takes proprioception, current object pose, a bounding box of where it should be grasped, and current goal pose, running through an LSTM policy outputting joint position targets for the full hand and arm.
The single policy works zero-shot across 12 unseen tools and target behaviors, achieving substantial task progress across every tool family, particularly those with long handles. Performance is weaker on heavier tools, which are easier to drop, and smaller objects where pose tracking encounters occlusion problems.
Training and test correlation was measured by evaluating checkpoints throughout training. As the policy improves on generic goal reaching with primitive objects, performance on unseen tools and human-demonstrated trajectories sharply rises, validating that the training objective with random objects and random goals effectively generalizes to real-world tools and tasks.
Sim Tool Reel was compared against two baselines. The fixed grasp baseline can grasp objects but must rotate using only the arm, resulting in table collisions. This highlights the importance of in-hand reorientation ability, as most prior approaches assume acquiring and maintaining a fixed grasp is sufficient, which fails for many cases.
Kinematic retargeting imitates human video demonstrations by transferring human fingertip motion to the robot but fails to reason about contact forces, failing to even grasp objects. Failure mode analysis found pose tracking failures dominate, followed by objects dropped after being grasped, and failed grasps. The policy demonstrates strong recovery behaviors, immediately regrasping dropped objects like hammers and completing tasks.
All code, assets, and policy weights are open sourced with an interactive demo running in the browser and on phones.
A follow-up work called Play to Perfect addresses precise assembly requiring sustained contact and millimeter-level precision. The framework argues that before learning precise assembly, the easier problem of playing with objects in free space must be learned first.
This motivates a pre-training fine-tuning paradigm. A shared dextrous prior is learned through task-agnostic play similar to Sim Tool Reel training, then fine-tuned on sparse reward contact-rich assembly tasks and deployed zero-shot in the real world. This enables diverse contact-rich assembly behaviors including tight insertion and multi-part assembly.
Recovery behaviors were not specifically trained for but emerged from domain randomization in simulation where random forces were added to objects, knocking them out of hands and providing experience picking them back up. Without this randomization, recovery would likely be less effective.
Primitive tools used in simulation are cylinders and cuboids that can be reasonably approximated as bounding boxes of graspable regions. This approximation works for spatulas and brushes but may not suit objects like bowls or scissors. Simulation runs two to three times faster with simple objects.
Generalization beyond tools to articulated objects presents challenges, as many things cannot be simulated well including water, zippers, or cracking open cans. The approach would involve transferring priors into the real world and continuing fine-tuning with real-world experience.
The policy functions as an inverse dynamics model in pose space, taking current pose and goal pose. For objects that cannot be specified as pose, such as towels, keypoints or video goals could serve as alternatives, making the policy an inverse dynamics model of current state and final goal state.
Continuous arm twisting was identified as a failure mode. The question of reset dynamics between actions remains open. Real-world applications typically avoid extreme arm contortion.
Iterating on policies involves expensive real-world evaluation that takes substantial time and often fails to produce statistically significant numbers without running truly 100 trials.
It's hard to tell if you're at 90% or 87% success rate. The team uses automated evaluation methods across novel tools and tasks in simulation. They train on random objects and random goals, then evaluate performance on real-world objects and trajectories placed in simulation. This provides metrics to determine if performance is significantly worse or better. However, human insight remains necessary because the system can occasionally perform magical tossing and catching behaviors in simulation that may work but would likely fail with the post tracker.
The team uses LSTMs (long short-term memory) in their RL packages. RL is finicky, and changing too much code risks breaking working systems. The codebase already had LSTMs implemented, so they tried it and it performed better. They started from a proven foundation and iterated from there rather than rewriting from scratch.
Transformers are probably better data sponges when unlimited data is available. However, the team is constantly updating policies rather than working with enormous datasets. Some testing has shown no improvement with transformers. The diffusion policy paper's tables show that convolution actually outperforms transformers on half the policies unless the transformer is tuned better, making transformers more sensitive.
Random goals are created by sampling a position and rotation for the first goal, then generating subsequent goals as delta poses up to 10 centimeters away and 90 degrees different. This works well for training policies to move from point A to point B, but generating meaningful goals requires human labels. An alternative approach could involve a high-level planner that understands full scene context and generates goals automatically.
Roughly 60% of failures come purely from pose tracking, making it one of the biggest system bottlenecks. This occurs most frequently with smaller objects where markers are easily occluded. Larger objects with more features experience fewer pose tracking issues. The team uses only third-person views, selecting a good angle that minimizes occlusion. They note potential improvements through first-person camera pose tracking or distilling to image-based policies.
Niko, co-founder and CEO of Rerun, previously worked on machine learning computer vision for physical world applications in shipping products for a decade. Rerun builds unified data layers for physical AI, providing tools and infrastructure for working with physical data from collection through training. Their open source SDK handles logging, visualization, generic querying, and data loading for training. Rerun Hub serves as a data catalog and large-scale data backend for cloud-based operations.
A new category of robotics companies is emerging that takes ownership of full business problems end-to-end. These companies focus on data center management, construction, warehouses, and manufacturing. They emphasize operational excellence, deploy minimal custom hardware, and often start with teleoperation before fine-tuning models. This approach avoids starting with foundation models to solve general problems.
The speaker believes robotics application companies will become the new SaaS, transforming the physical world's economy. The strategy involves starting with single customer problems that someone will pay to solve, beginning with teleoperation and off-the-shelf hardware. This allows understanding end-to-end real business requirements quickly, since the physical world is brutal and everything breaks in unexpected ways.
A hypothetical paper plane factory automation business would start with teleoperation and off-the-shelf hardware. Key learnings from real deployment might include: needing 1,000 perfect planes per day for viability, determining acceptable failure rates, optimizing for new design onboarding speed, adding paper trays to reduce failures by 50%, understanding that humans require 20 hours of practice to meet customer requirements, and discovering that adding paper pickup and box packing capabilities increases value 10x.
The basics for learning include fine-tuning open models like pi on a few hours of teleoperation demonstration data. Companies need replicas of customer environments in their offices for testing. Finding repeatable success evaluation methods forms the backbone of learning, encoding business-specific importance factors learned from customer interactions. Teams must track metadata including rollout information and failure classifications.
Good data makes models better and requires constantly training, evaluating, and debugging data. Researchers at large robotics companies frequently discover that the solution involves sending different instructions to data collectors rather than changing training approaches. Collecting all data upfront before training is a major mistake. The most important factor is testing and iterating rapidly on real problems rather than specific collection methods.
Physical data differs fundamentally from web data—it is multimodal, multi-rate, episodic, with 3D semantics and deep nested structures. Traditional table-based databases struggle with querying and storage efficiency for this data type. Teams without proper storage layers at the bottom face significant workarounds and friction throughout their operations.
Once companies have deployable robotic services, performance evaluation capabilities, and data collection processes, they can iterate and scale. This includes improving intelligence through larger data scaling, algorithm iteration, and adding sensors like tactile, depth, or audio. Companies must excel at sales, robot assembly, shipping, unboxing experience, operations, and support. These non-modeling areas often provide the moat that pure model companies cannot replicate.
Successful robotics application startups focus on pragmatic, simple, flexible systems with minimal moving parts. They achieve fast data turnaround, full stack debugging, and complete system understanding. Frontier labs invest heavily in compute for researchers including both GPUs for model experiments and CPUs for data exploration latency reduction.
The market is enormous, base models continue improving, and physical world friction creates real business moats. Companies can succeed with relatively small capital requirements without billion-dollar seed rounds, though great AI and engineering remain essential for winning.
Early success appears in data centers, warehouse robotics, small-scale manufacturing, and food industries. Companies building reliable robots are 100% supply constrained with easy demand fulfillment.
Robotics application companies haven't reached billions in revenue yet because robotics involves death by a thousand cuts. The ability to try applications cheaply is relatively new, with more arms available and better base models than two years ago. Previous companies had to raise funds for more general approaches.
Estimating required data scale for problem solvability before deployment involves teleoperating first. Many businesses don't require full autonomy, and companies can learn by training models and plotting scaling curves rather than guessing upfront. Some tasks initially thought necessary may prove unimportant.
General Instinct builds infrastructure for running physical AI models quickly. Their founders have backgrounds in VLMs and robotics RL. For physical AI models like world action models and VAS, they provide alternatives to existing frameworks.
World action models use a central diffusion transformer to predict future appearance and future kinematics simultaneously. Current robot camera observations in video streams condition the prediction of future frames, which then inform future kinematics prediction. Nvidia's Dream Zero exemplifies this approach with flow matching for acting on action chunks in predicted frames.
Dream Zero benchmarks show strong performance compared to state-of-the-art models including pi and other Nvidia models. However, action models using diffusion for frame prediction remain computationally heavy, requiring two GB200s to run even after optimizations, with each unit costing around $70,000, making this approach economically unscalable for robotics.
The training pipeline treats current observation as a condition for flow matching, adds noise to future latent, and trains the model to predict future velocity field before sending it to the ODE to drift back to clean future latent. The inference pipeline operates autoregressively for 50-100 steps to ensure model accuracy. For each chunk of 16 frames, the DiT (diffusion transformer) must run 32 times due to classifier-free guidance (CFG), requiring both conditioned and unconditioned flow matching runs to compute gradient derivatives.
Instead of running diffusion models to predict future frames, some approaches skip video prediction entirely. Image Run predicts only the future end state (t+n) as a single frame rather than generating entire video chunks. Fast One takes an extreme approach by using hidden states already learned in the DiT, eliminating the need for a decoder entirely and training action heads directly on these hidden states as conditions.
Generative world action models decode and predict future frames, analogous to viewing a Google Maps VR experience. Latent world action models use hidden states as navigation signals to determine policy direction and future actions without explicit frame decoding. The core challenge remains maintaining rich world representations across both approaches.
Lekin proposes using two encoders: one for current observation and another for future observation, applying MSU loss between current and future latents to teach future prediction. Other approaches treat future states as action distributions, using flow matching to estimate these distributions.
Dissolution was applied to the VAE encoder-decoder and DiT components to reduce model size. The architecture was split into two separate DiTs instead of using a single transformer, with cross-attention from video transformer to action transformer. This allows the action transformer to learn world representations from video transformer hidden states without decoding future frames. Auto-regressive flow matching sampling was distilled from 50-100 steps down to 1-2 steps, achieving 50x speedup.
Beyond pixel-level and latent-level learning, mask and flow modalities were tested for retaining world representations. Heatmap visualizations demonstrate how the model guides action selection for the next 0.5 seconds. The resulting world action model runs at 500 milliseconds per chunk (16 actions) on JSON.
Training begins with clean future latent and gradually adds noise until reaching pure noise, teaching the model to reverse this process. During inference, the model starts from pure noise and gradually reverses to future clean latent. This curriculum learning approach explains the computational cost of 50-step processes, which distillation reduces to 2-3 steps without performance degradation.
VA models predict current actions from current observations without explicit future kinematics learning. World action models introduce future videos to teach models how kinematics change at the pixel level, creating a correlation between physics understanding and action generation. This physics-based approach enables better action prediction through learned future dynamics.
Dividing processing into two transformer stages enables more efficient world representation maintenance. Image Run and Fast One demonstrate cross-attention from backbone models to action experts, allowing hidden state utilization without explicit frame decoding. Mixture of transformers offers another efficient approach for this separation.
Flow matching inherently performs test-time planning by treating actions as distributions and sampling optimal trajectories. Teacher forcing trains the model to sample from action distributions, achieving the equivalent of generating multiple samples and selecting the best trajectory.
The focus is on building infrastructure that enables world models to perform efficiently on robots in real-time, rather than becoming an end-to-end robotics application company. The priority is creating generalizable research infrastructure before edge deployment optimization.
Chunk size determines prediction scope, with 16 frames being standard but configurable to 32+ frames. Larger chunk sizes increase learning difficulty due to extended future state prediction requirements.
Asymmetrical denoising measures KV cache energy to determine which frames require highest detail preservation versus latent-based processing. This approach selectively predicts different resolution frames based on action-relevant energy measurements rather than uniform frame generation.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
