8 Predictions for the Era of Continual Learning
In a Nutshell
Continual learning makes AI safety, regulation, and business models obsolete overnight: instead of static weights checked before deployment, models will improve daily from millions of real work sessions, requiring ongoing risk inspections and new techniques to preserve safety across weight updates. Leading labs gain accelerating moats through switching costs and economies of scale, as personalized weights become too expensive for small users and enterprises face lock-in or fall behind. The result is faster deployment pressure, greater AI diversity across domains, and a shift from pre-deployment alignment research to maintaining values during continuous learning.
These notes were generated by AI and may contain inaccuracies.
The speaker explains why actual continual learning is needed for AIs to perform whole jobs competently. The current approach of AIs writing markdown files from session to session is insufficient. An illustrative example compares this to students learning saxophone: each student enters the music hall, fails on their first attempt, writes notes about what went wrong, and leaves these notes for the next student waiting outside. With an infinite number of students outside, no sequence of text could allow subsequent students to nail the saxophone from the first try. The relevant experience must accumulate in the brain. The same principle applies to AIs needing to accumulate experience from different workplaces.
Current AI regulation proposals assume a model is trained and then deployed, allowing pre-deployment checks to prevent issues like aiding cyber attacks. This assumption won't hold when models improve daily based on millions of work sessions. Locking in safety regulatory regimes now risks creating archaic, counterproductive approaches. Government safety evaluations should involve monthly or quarterly risk inspections rather than focusing on a special moment after training but before deployment, as this distinction won't exist in the future.
Current technical alignment research focuses on ensuring frozen weights behave well during deployment. There is little research on maintaining safety with constant weight updates—preventing jailbreaks or shifts to deceptive personas. When AIs consolidate learnings between users, preventing malicious backdoors or inclinations injected into the base model becomes critical. This mirrors the human alignment problem where children learn new things but ideally retain common sense and basic values to avoid adopting weird beliefs or misanthropic ideas.
Currently, fewer than five prominent AI minds (base models served to millions or billions of users) exist, all trained on roughly the same data and thus quite similar. With continual learning from different experiences across companies and instances, significant diversity will emerge. This is a net positive, avoiding the monolithic singleton outcome and the mode collapse seen in current models.
When deployment becomes part of training, returns to leading in the AI race accelerate. The best model attracts more users doing complex work, generating feedback that improves the model further beyond session windows.
If models learn mainly from deployment, labs will face pressure to deploy smartest models earlier. Anthropic reportedly used Mythos internally since February but only shipped it publicly in June—a four-month gap that would become impossible under continual learning, as competitors shipping worse models would gain advantages from real-world experience.
Continual learning creates clear moats for leading labs. Cloud providers earn high margins despite undifferentiated services because switching is time-consuming and expensive, as seen in Amazon and Google's quarterly earnings. Currently, users can switch between Codex, Cursor, and Claude Code freely. With continual learning, switching requires replacing an AI employee with months of organizational context with an inexperienced intern needing retraining from scratch, enabling model providers to demand hefty margins.
Enterprises will try avoiding lock-in, but may face a choice between lock-in or losing valuable session-to-session improvements. If real usage drives model improvement, labs may subsidize users allowing training on sessions—similar to deals for new coding product users and Google's free search. Conversely, enterprises refusing training access may lose access to best models. Labs can use both carrots and sticks to encourage learning from experience. The technical challenge of pooling weight forks back into main models will eventually be solved.
AI training already has large economies of scale, with lab revenues increasing faster than compute. Continual learning may create inference economies of scale through batching. When per-company information requires full weight updates rather than low-rank adapters, batching advantages are significant. Back-of-the-envelope math shows optimal inference batch size for sparse models like DeepSeek v3 exceeds 2400 concurrent sequences. Without this, compute is underutilized. A given set of weights serves efficiently only when thousands of sequences decode against it simultaneously. Large companies with many employees and agents efficiently serve weight forks, while individual users with batch size of one suffer over two orders of magnitude worse efficiency. Serving personalized weights economically favors big organizations.
The speaker notes that more changes will occur by the time continual learning works, with the most important changes being hardest to anticipate. The eight predictions above seem clear even now. This narration comes from a blog published at dwarkesh.com.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
