AI Can't Learn The Way Humans Do - This Could Fix That
In a Nutshell
The core problem in AI is poor sample efficiency—models need massive data while humans learn from a handful of examples—because current systems lack explicit world models that predict how actions change future states. The solution path is model-based RL using video diffusion models as world models: pretrain on abundant unlabeled video, add cheap action conditioning from limited teleop data, then train policies entirely inside the neural simulator via imagined rollouts. This approach scales to robotics and self-driving because it mirrors human neocortical world modeling, enables test-time planning, and bypasses the need for enormous real-world interaction data.
These notes were generated by AI and may contain inaccuracies.
One of the biggest open problems in AI is solving sample efficiency—how to get models to quickly learn new tasks or skills from relatively small amounts of training data. Humans learn new games, concepts, and skills after just a handful of tries, while current models often need tens of thousands of data points.
The two major remaining problems are intelligence per watt and intelligence per sample. Intelligence per sample refers to how much more intelligent a model becomes with each additional data point. The rate of skill acquisition differs from total skill acquisition. Current models are poor at getting smarter with fewer samples.
The ARC-AGI test sets demonstrate cases where humans intuitively solve puzzles with thinking and effort, but frontier AI systems cannot. Humans bring inductive bias from K-12 education and math training, whereas models compress the entire internet yet still struggle to generalize to new tasks.
In the extreme case of perfect sample efficiency, zero samples would be required. A perfect world model would eliminate the need to collect samples from the environment. Newton's second law of motion serves as an example—NASA can plan asteroid intercepts years in advance using Newtonian mechanics, allowing the system to act without collecting new environmental samples.
The real world is differentiable, enabling model predictive control. Humans build world models over years of experience, such as entrepreneurs learning what customers want or product taste—predicting what others will like. A 1967 COGSAI study by Richardson showed that participants who physically practiced basketball layups for one hour improved by approximately 24%, while blindfolded participants who only imagined the layups improved by 23%.
Neuroscientist Shaw Duckman at Stanford argues that the growing neocortex during the great cortical expansion 10 million years ago evolved to improve world modeling. An implicit world model in the brain, encoded by genetics and learning, enables intuitive thinking. Current models may have some implicit understanding in natural language but lack explicit world model representations, which becomes problematic in robotics and self-driving domains.
In a drone control problem, the drone has mass m with gravity g acting on it. The state includes position (px, py, pz) and velocity (vx, vy, vz), forming a 6-dimensional state vector. The controllable thrust vector is u. The goal is reaching target position p-star and velocity v-star (typically zero for landing).
The state transition function s(t+1) is given by s(t) and action u(t). This transition function, dynamics function, or world model is fundamental to reinforcement learning. A policy determines which action u(t) to emit given state s(t).
With a perfect world model from Newtonian physics (F=MA), the position update is p(t+1) = p(t) + Δt·v(t) + ½Δt²·a, and velocity updates accordingly. Acceleration is the sum of forces: thrust divided by mass plus gravity.
Model predictive control solves for the policy by minimizing the sum over time of (p-star - p(t)) + (v-star - v(t)), plus a lambda penalty on u(t) for energy expenditure. Constraints include maximum thrust u-max. The objective is convex, allowing solution via convex optimization tools like CVXPY using log barrier or interior point methods. This yields the optimal u(t+1) to infinity—the optimal path minimizing energy or achieving fastest arrival.
Deterministic differentiable control works because the system is differentiable, allowing Lagrangian formation and gradient computation. Non-differentiable systems prevent convex optimization and SGD.
An adversarial drone scenario introduces stochasticity and non-differentiability because the opponent's actions cannot be differentiated through. The state transition now depends on both own thrust and unknown opponent actions, requiring reinforcement learning methods such as value iteration, policy iteration, DQN, actor-critic, and Q-learning to estimate the non-differentiable stochastic process.
A value function estimates the value of states. In rocket landing, paths that burn houses have low or negative value, while safe cones have high value. Value represents the expected sum of future rewards. Rewards are defined as +1 for winning, -1 for losing, and 0 for draws in games like Go or chess. Heuristics assign point values to pieces (pawn = 1, rook = 5).
The value V^π is the discounted sum of rewards following policy π. The objective is finding a new policy π while using machine learning to estimate the value function from received rewards. World models incorporate state and action distributions for more intelligent policies.
The joint distribution factors as policy π(a|s) and world model p(s(t+1)|s(t), a(t)) with parameters θ. These can be learned separately or jointly. Video generation models predict s(t+1) from frame s(t). Action conditioning injects actions into the model (similar to CLIP) to enable embodiment—learning that actions influence the world.
Joint training creates world-action models. Separate training causes issues with disparate datasets and expensive test-time planning requiring sequential sampling. A joint model outputting both action and s(t+1) is cheaper and faster for planning.
Chess has a state space cardinality of 32^64, which is large but tractable in practice. The transition function is stochastic and non-differentiable due to the opponent's unknown moves. The action space is small—approximately 8 legal moves per state on average. Rewards are +1 for win, -1 for loss, or 0 for draw, or use piece-value heuristics.
Naive lookahead requires expanding all actions then all possible resulting states, making even one-move lookahead intractable due to state space size. The actual number of reachable boards is much smaller than the theoretical maximum, and all games start from the same position.
Go uses a 19×19 grid where each intersection can be black, white, or empty, yielding approximately 3^361 states. The action space size is 361. Terminal reward is +1 or -1 based on game outcome. Rollouts are trajectories of states and actions under a policy until termination, with all moves by the winner receiving positive reward and loser moves receiving negative reward.
Rollout refers to taking n sequential steps of play by all players under a specific policy instantiation.
The policy pi theta t is frozen after training. The model plays approximately 70 games, subsampling state-action results to train and update both the policy and the world model transition function. The transition model takes in state ST and outputs the probability of the next state ST+1 along with the value of the current state.
The loss function L theta consists of the squared difference between v theta and reward R, minus the policy term pi log P theta, plus a weight decay term. After training, the system performs an expensive test-time planning step.
Monte Carlo Tree Search (MCTS) is used for test-time planning. The policy outputs 361 probabilities summing to one, representing action probabilities. For each of the 361 possible actions, the model is invoked to estimate the value of the resulting state ST+1.
Actions are selected using the Upper Confidence Bound (UCB), balancing exploitation via the Q-value (value of ST+1) and exploration via a term involving the square root of visit counts. The formula includes the policy probability of the move and a ratio of visit counts NSB over NSA.
Visit counts track how many times each state-action pair has been visited during MCTS. The tree expands by biasing selection toward higher probability moves while maintaining exploration. After each rollout, counts are updated and Q-values are adjusted based on expected value along each path.
800 MCTS simulations are performed per action. Each simulation requires approximately 30 model invocations to traverse depth-30 rollouts, totaling roughly 24,000 model invocations per action. The resulting MCTS tree is discarded after selecting the move, though the traversal probabilities are used as the target pi for training the loss function.
In a hypothetical 1000x1000 Go board with roughly one million actions, the required simulations would scale dramatically. Maintaining the same two samples per action at depth 30 would require approximately 60 million model invocations per action.
AlphaGo performed 800 MCTS simulations to cover 361 actions, yielding roughly two samples per action on average. Scaling this approach to larger action spaces multiplies test-time compute requirements astronomically.
The approach requires an extremely small action space cardinality. Real-world domains like stock markets and venture capital have constantly changing rules, violating the assumption of a fixed, deterministic environment.
Real-time constraints make the approach impractical. AlphaGo thinking for 60 seconds to 5 minutes before moving would be catastrophic in driving scenarios where decisions must occur in milliseconds.
The state space for self-driving cars includes surroundings, vehicle state, camera inputs, weather, and road conditions. The state space is effectively infinite for practical purposes. Similar to how AlphaGo compresses board states with a convolutional network into a latent representation, self-driving systems compress pixel inputs using deep convolutional networks.
Ten years of deep learning research has produced effective compression techniques that make handling these massive state spaces tractable. Latent space optimization, as demonstrated by Stable Diffusion, dramatically reduces effective state space dimensionality.
Physics-based modeling fails because other agents adapt to your actions. A strict Newtonian model would predict collision and suggest not moving at all, yet this is not how traffic functions. The environment must model how your actions influence other agents' behaviors.
Even a simplified action space with steering wheel (0-365 degrees), brake, and gas, discretized into 10 severity levels, yields 365,000 possible actions—approximately 100 times larger than AlphaGo's action space.
Real-world action spaces are substantially larger. Tesla possesses both video trajectories and corresponding actions, providing a significant competitive advantage. Other approaches only have access to state transitions ST to ST+1 without action labels.
Model-free RL trains a policy pi of action AT given state ST directly without any world model. This is equivalent to behavior cloning and is analogous to next-token prediction in language models, where an action space replaces the tokenizer vocabulary.
Model-based RL includes both a policy and a transition function sigma predicting ST+1 given ST and action AT. This enables stronger policies through test-time planning but increases inference cost substantially.
Model-based approaches are considered necessary for AGI because they mirror how the human brain operates. The neocortex expansion approximately 10 million years ago enabled world modeling, allowing simulation of future states before taking actions. During sleep, hippocampal sharp-wave ripples compress experiences into improved policies.
A six-axis robotic arm has 16 degrees of freedom. Discretizing each degree similarly to the steering wheel example produces action spaces on the order of 10 to the 16—vastly larger than self-driving car action spaces.
Teleoperation data collection is extremely expensive. Cross-embodiment gaps prevent policies trained on one robot from transferring to another with different dynamics, weight distributions, or aerodynamics. Tesla shards FSD data per vehicle model to account for these differences.
Jürgen Schmidhuber published the paper "World Models" demonstrating the approach on OpenAI Gym environments including car racing and Doom. An RNN-based model was trained on environment trajectories, then used to generate synthetic imagined rollouts for policy training. The policy trained entirely on synthetic data performed well when deployed in the actual environment.
The core insight is that a predictive model of state transitions enables generation of synthetic training data for policy optimization, with subsequent fine-tuning on real trajectories.
The Dreamer series began with Dreamer One published by Danijar Hafner in November 2018, building on a powerful idea from May 2018. Hafner has published papers continuously over seven years, with Dreamer V4 as the capstone. The approach trains a world model on state transition data, then injects action conditioning using a very small amount of data to create a world model with action conditioning. The system samples extensively from this model and trains a policy on synthetic imagined rollouts. Dreamer V4 became the first paper to successfully mine diamonds in Minecraft, a task representing next-level difficulty, using only synthetic data.
Society possesses abundant state transition data across YouTube, Flickr, and other online datasets showing people performing tasks. Previous attempts to utilize this data failed, but video generation models now enable creating world models from this data, adding action conditioning, and post-training for specific tasks with only a few examples. Tasks like chopping wood, making sushi, or folding beds become feasible. A policy can then be trained within this neural simulation.
State-of-the-art video diffusion and flow matching models represent the current best approach for handling effectively unlimited available data. Models like those from C-Dance or Sora are pre-trained and high-performing. A small amount of action conditioning converts them into world models from which extensive samples can be drawn for policy training. Wave applied this approach with Gaia, raising $1.5 billion to pursue self-driving car applications. Nvidia and the Dream Zero paper for robotics follow the identical methodology.
Dream Zero instantiates a joint model of state transitions and actions using an open-source video diffusion model. Only approximately 500 hours of teleop data are required to achieve strong performance. The approach incorporates clever techniques enabling cross-embodiment operation and scene task handling with relatively small data quantities. This represents direct application of the Dreamer paper concepts to robot embodiments and performs better than anticipated.
World models represent the path humans followed to achieve current capabilities through 10 to 20 million years of genetic evolution. Larger world models improve both training and test-time planning. The same principle applies to robotics. The Gencast weather planning paper applies a similar concept to model global weather using diffusion steps. The model predicts the next state of the world based on two time steps of lag (AR2 order) using latent diffusion rollouts.
A University of Washington paper states in its abstract that each cortical area estimates both latent sensory states and actions, while the cortex as a whole predicts the consequences of those actions. This description matches the joint model of sensory latent states and actions. The equations described align precisely with the joint embedding approach. If this mechanism works in human brains, it should function for robotics.
Classic reinforcement learning maintains a Q matrix of dimensions states by actions, requiring sufficient counts for each state-action pair to compute average values. This approach scales poorly as state and action space cardinality increases, resulting in insufficient sample efficiency for robotics applications. The classic solution applies a neural network to compress the state space into lower dimensions. This technique predates deep learning.
The Joint Embedding Predictive Architecture (JEPA) processes image sequences through a convnet encoder to obtain latent representations z_t, z_t+1, z_t+2. Instead of predicting full images in pixel space, the model predicts z_t+1 hat from z_t, minimizing the difference between predicted and actual latents. Direct implementation collapses because the model learns to output zeros. Regularization techniques such as SigL or VICReg prevent collapse by enforcing that the distribution of z_t+1 follows a Gaussian across large batch sizes. The LE-WM (Latent Embedding World Model) paper represents the most recent work in this area.
JEPA self-supervised learning does not perform as well as alternatives. In language models, token embeddings E1, E2, E3 can be used such that the model outputs hidden states H_t+1 close to E_t+1 without cross-entropy loss. This approach serves as a cheaper proxy for cross-entropy while eliminating the expensive cross-entropy head. Researchers are exploring applying the JEPA concept beyond pixels to language models. The idea recurs across domains including stable diffusion and graph convolutional neural networks for drug design, with various practical tricks required for effective implementation.
Physics-informed neural networks (PINNs) do not function effectively. Self-driving car data distributions concentrate almost entirely on normal driving scenarios. When attempting to train models to avoid driving into houses by placing the vehicle in off-distribution states, the model transforms the scene into a highway due to data distribution imbalance. Training requires extremely careful data mixing to maintain even batch composition across the state space. Even simple sine wave interpolation tasks fail to reach machine precision with SGD, leaving residual error. Architecture limitations contribute to this problem. Transformers lack compression in the time domain. Sam Altman has indicated that architectures more performant than transformers will emerge.
Test-time planning and adaptation represent critical challenges. Human brains adapt rapidly to new environments without requiring sleep and retraining. Out-of-distribution prediction remains difficult, though humans adapt quickly from single data points. Friction coefficient estimation exemplifies required rapid adaptation, as surface properties change in human environments. Current models lack mechanisms for quick estimation and adaptation. Expensive planning steps limit practical speed. AlphaGo-style MCTS rollouts deliver maximum performance but operate too slowly for real-time applications.
The brain runs multiple heterogeneous models autonomously and simultaneously. Type 1 and Type 2 thinking occur concurrently, with some models overriding others to control the motor cortex. Shortwave ripples during sleep involve the hippocampus emitting reversed spike trains through both hemispheres approximately seven times. Sleep proves essential for forming long-term memory. No intelligent species exists that does not sleep, suggesting evolutionary pressure for this mechanism despite predation risk during sleep.
The squint test evaluates whether a system resembles biological intelligence at a high level. World models with action policies performing test-time planning appear substantially closer to brain function than autoregressive LLMs. Neuroscience evidence supports the brain performing implicit future state and action prediction. However, the brain functions as both model and optimizer, invoking models while performing optimization itself. Current architectures lack the awake-sleep distinction. During sleep, the brain collects experience rollouts and updates policy functions, a process not replicated in existing systems.
World models will increasingly appear in robotic policies. Full self-driving represents a near-term application if real-time constraints can be addressed through parallel computation. Rosie the robot capabilities for household tasks appear achievable through data and compute investment despite collection costs. Companies are working across the full pipeline from egocentric and teleop data collection through world model training, embodiment development, and adaptation methods. Demonstrations in the current year indicate meaningful progress.
Practical challenges remain in real-time performance, PINNs, and tactile sensing. Human epidermis detects shear force, temperature, and friction coefficients across the entire body surface. Robotic systems typically possess single tactile sensors lacking temperature, friction estimation, and comprehensive coverage. Without tactile feedback, tasks like tying shoelaces become impossible. Training on sufficient human demonstration data might enable performance without feedback, but incorporating actual touch sensing would substantially improve efficiency.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
