What does the next training paradigm look like?
In a Nutshell
The core bet of major AI labs is that massive RLVR training across verifiable, replayable environments will produce general agents capable of long-horizon problem-solving. This approach faces fundamental limits in domains lacking deterministic simulators, where models cannot efficiently learn from sparse, ambiguous real-world data without continual learning mechanisms. The proposed path forward combines expanded context windows for in-session learning with techniques like on-policy self-distillation (OPSD) to transfer deployment insights back into model weights, potentially enabling AIs to improve through accumulated real-world experience rather than pre-release training alone.
These notes were generated by AI and may contain inaccuracies.
All major AI labs are betting that training AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments will produce AGI. This approach aims to create problem-solving agents capable of making progress on open-ended tasks for weeks at a time while handling errors, mistakes, and ambiguity.
Optimists argue that current deficits in the training paradigm—such as data inefficiency and lack of continual learning—can be overcome through scale, similar to how NLP challenges collapsed when sufficient compute was applied to LLMs.
Models are currently one one-millionth as sample-efficient as humans. Proponents of the current paradigm argue this inefficiency only matters during training, which is a one-time cost amortized across billions of inference sessions.
What matters is sample efficiency during a session, which has been improving with more RL training. AI agents are solving more ambitious problems over longer time spans, as evidenced by coding model improvements.
The argument is that continual learning—updating model weights based on deployment experience—may not be necessary if in-context learning improves sufficiently across longer time horizons. The idea is that fitting six months of on-the-job learning into a context window could substitute for weight updates.
Architectural innovations are dramatically increasing the amount of context transformers can store, leading some to speculate about effectively infinite context windows within a couple more years of progress.
Computer use is clearly verifiable through outcomes like whether an Etsy item was delivered, whether a venue was booked, or whether taxes were submitted. Despite this verifiability, progress has been slower than in coding and math.
One reason is limited high-quality multimodal data during pretraining. A more underrated reason is that domains must be both verifiable and grindable—meaning you need to run many parallel rollouts against a deterministic, replayable simulator from identical starting points.
For coding tasks, you can create a container with a software repo missing a feature and have a thousand parallel agents work on identical copies. Computer use doesn't allow this trivially—you can't have a thousand agents attempt the same Amazon checkout flow because platforms detect and block bots.
Creating clones of Slack, Gmail, and other applications is labor-intensive and unscalable. Once AIs can code these clones with high fidelity, computer use progress should accelerate. Rebuilding applications from scratch also serves as an RL objective for coding.
Computer use lethargy reveals that without replayable training targets, models struggle to progress. Models are incredibly sample-inefficient during training. For many skills, deterministic simulators cannot be built.
Examples of skills without farmable simulators include building a business from scratch, winning court cases, having a profitable trading day, or helping a candidate win an election. These require real-world interaction where outer-loop verification takes months or years.
Dealing with reset-free, non-stationary environments is a known open RL problem. Due to idiosyncratic and sparse data in most domains, sample efficiency is required for proficiency. AIs need to learn from unstructured, unverifiable, and ambiguous information from scarce real-world interactions.
The labs are betting that RLVR will generalize—that training on enough containerized, reproducible environments will produce a general agent that can make and execute plans, learn rapidly from new information, and pick up new skills within a single session.
The claim is that an endlessly RLVR-trained AI dropped into Texas politics in 1948 could outperform Lyndon Johnson at winning a Senate seat, or given $100 million in 2002 would build SpaceX. Whether RLVR generalizes this well is an empirical question.
If labs increased RL environment spending from billions to a trillion dollars, would that produce fully human-like general intelligence within the context window?
Dario Amodei stated during a podcast: "There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations."
This suggests short-horizon RL training may not generalize to long-horizon performance. If generalization from short to long horizon fails, it raises questions about generalizing from white-collar task training to building a business from scratch.
Around 30 to 50 percent of lab compute goes to inference, currently contributing nothing to model improvement. This is especially problematic because the most valuable learning information is revealed only in deployment—how organizations actually use the model, what tasks they assign, and what real-world mistakes occur.
AIs are broadly deployed and privy to domain- and organization-specific tacit knowledge but cannot use it for improvement. Continual learning requires updating weights, not just building larger KV caches, which is unscalable and unlike human learning.
Human learning involves compression that aids generalization and grokking. Humans with autistic-savant abilities to recall random information with model-like fidelity have impaired abstraction and metaphor understanding. Human continual learning is about chiseling intuitions and big-picture knowledge into weights, not retaining all observations.
Moving into weights means giving up in-context learning's sample efficiency. Gradient updates are super sample-inefficient, so successfully shipped online-learning models learn the same thing across millions of users.
The Cursor Tab model online-learns by predicting the same objective—which edits got accepted by users—across over 400 million requests daily. So far, models have not online-learned different things for different users because a single session generates insufficient data to train a more capable AI.
Current online learning works for limited use cases, but continual learning's purpose is handling the complexity where each job, company, and problem differs, requiring intelligence to learn deployment-specific information that cannot be shared across a training run.
As the podcast grew, operational overhead increased. Previously, contractor invoices were emailed, manually organized into folders, and paid individually every few weeks. Now, contractors email invoices to a Mercury address that automatically downloads, scans, extracts information (contractor name, address, payment amount, invoice number, due date), creates draft payments, and stores them for review.
Mercury consolidates business financial operations. Banking services provided through Choice Financial Group and Column N.A., Members FDIC. Mercury is a fintech company, not an FDIC-insured bank.
Sample efficiency and continual learning are deeply connected. Relatively little data is available during deployment. Learning from this data requires sample efficiency, which in-context learning achieves through fast weights built by attention, but this scales poorly in memory.
Architectural innovations are needed for intermediate representations. Ideas like sparse attention and KV cache compaction exist, with new papers released weekly. Architecture does not appear to be the fundamental bottleneck for continual learning.
The loss function may be the bottleneck. On-policy self-distillation encourages the base model to make the same predictions when solving real-world problems as the model with accumulated session context would make. This distills session learning back into weights.
OPSD is superior to RLVR because it doesn't require outer-loop verifiable rewards—only a model that learns correctly within context—and provides denser supervision through per-token probability discrepancy rather than a single projected reward.
OPSD is superior to supervised fine-tuning for continual learning. The naive SFT approach of training the base model to predict all session tokens makes no sense because job improvement comes from consolidating relevant insights, not recalling transcripts with perfect fidelity.
RL training concentrates updates on outcome-relevant information, producing sparse updates that avoid overwriting other model knowledge. OPSD preserves this property by extracting only knowledge necessary to match teacher performance on real-world tasks.
A more speculative idea involves AIs building simulations of reality to rehearse skills and test strategies. This could enable orders of magnitude more simulated samples in the same wall-clock time.
EfficientZero, trained after AlphaZero, could beat novice humans at unseen Atari games given two hours against a simulator, by playing dozens of simulated games per real step. Future LLMs might consume less real-world data while practicing against self-built environments.
Building world simulations is much harder than emulating Go. If successful, dreaming would become a fourth scaling axis alongside pretraining, RL, and inference-time compute. The model would spend compute writing RL environments and training against them, rehearsing production skills for specific users.
Instead of hitting /compact for summary-based simulacrum of continual learning, hitting /dream would incinerate compute to build and train against a video-game version of witnessed reality.
RLVR training produces agents that can orient to unfamiliar problems, try strategies, and iterate through roadblocks—competent enough to start gaining real-world experience. Once deployed on projects off the training distribution, expanded context lengths enable week-long co-working sessions.
At session end, thumbs up or down reviews trigger distillation of session learning into base model weights through OPSD, dreaming, or other techniques. This enables improvement in domains adjacent to explicit RLVR training, then further adjacent domains through successive online learning rounds.
AI skills expand beyond originally trained verifiable domains. Pretraining created base intelligence for competent agency through RLVR; RLVR creates agents competent for broad deployment and on-the-job learning once continual learning recipes arrive.
The main improvement mechanism shifts from pre-release training to accumulated deployment experience across the economy. Each user interaction makes AIs smarter through both individual session learning and collective learning from all users worldwide.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
