What does the next training paradigm look like?
In a Nutshell
The core bet of major AI labs is that massive RLVR training across verifiable, replayable environments will produce general agents capable of long-horizon problem-solving. This approach faces fundamental limits in domains lacking deterministic simulators, where models cannot efficiently learn from sparse, ambiguous real-world data without continual learning mechanisms. The proposed path forward combines expanded context windows for in-session learning with techniques like on-policy self-distillation (OPSD) to transfer deployment insights back into model weights, potentially enabling AIs to improve through accumulated real-world experience rather than pre-release training alone.
These notes were generated by AI and may contain inaccuracies.
All major AI labs are betting that training AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments will produce AGI. This approach aims to create problem-solving agents capable of making progress on open-ended tasks for weeks at a time while handling errors, mistakes, and ambiguity.
Optimists argue that current deficits in the training paradigm—such as data inefficiency and lack of continual learning—can be overcome through scale, similar to how NLP challenges collapsed when sufficient compute was applied to LLMs.
Models are currently one one-millionth as sample-efficient as humans. Proponents of the current paradigm argue this inefficiency only matters during training, which is a one-time cost amortized across billions of inference sessions.
What matters is sample efficiency during a session, which has been improving with more RL training. AI agents are solving more ambitious problems over longer time spans, as evidenced by coding model improvements.
The argument is that continual learning—updating model weights based on deployment experience—may not be necessary if in-context learning improves sufficiently across longer time horizons. The idea is that fitting six months of on-the-job learning into a context window could substitute for weight updates.
Architectural innovations are dramatically increasing the amount of context transformers can store, leading some to speculate about effectively infinite context windows within a couple more years of progress.
Sign in to read the full notes
Get access to AI-generated notes, topic timestamps, and more.