Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again
In a Nutshell
Rich Sutton argues that AI systems fundamentally need to keep learning after deployment through ongoing weight updates rather than relying on static models like current LLMs. The core problem is that existing approaches depend on finite human-generated data and human-designed simulations, which creates an insurmountable bottleneck because the world is infinitely complex and requires continual learning from direct experience. The solution involves developing new algorithms like continual backprop that enable systems to learn from single streams of experience without catastrophic forgetting, allowing them to form abstractions and plan with self-discovered knowledge.
These notes were generated by AI and may contain inaccuracies.
Rich Sutton argues that the concept of "continual learning" is only necessary because the field has become weird. In normal times, learning would always be understood as continual. The ordinary way of thinking is that we always act and learn simultaneously.
Rich Sutton invented reinforcement learning, wrote the seminal textbook, and mentored key figures like Dave Silver. He authored "The Bitter Lesson," considered the Bible of the field. He co-founded Oak Lab with Khurram Javed, his former student from University of Alberta.
Sutton chose reinforcement learning because learning and having goals are central parts of the mind and intelligence. He was doubling down on what he had always been thinking about.
In 2003, Sutton was dying of cancer when he decided to take a job at University of Alberta. He continued working on research during this period because of habit. He references Benjamin Franklin's observation that people do things either out of habit or vanity.
Sutton maintains he is not radical or weird. The field is weird for needing to specify "continual learning" when all learning is naturally continual. He notes that a decade ago, learning was understood as important, goals were central, and perception mattered. Before AI craziness, no one would need to specify continual learning.
Sutton describes himself as lucky rather than skilled at selecting students. He points out that David Silver picked him rather than the reverse. Khurram Javed's collaboration with Sutton began organically through a project, converted to a thesis proposal without formal application, and proceeded naturally.
Sutton wrote "The Bitter Lesson" after a long time coming, drawing from decades of experience including the symbolic AI era. The essay advocates against getting distracted by human knowledge and instead focusing on scaling with computation.
Sutton distilled the bitter lesson into 26 words: don't be distracted by human knowledge as AI traditionally has been. Instead focus on learning methods that scale with computation like search and learning. The emphasis is on algorithms and improvements that scale with computation rather than human input.
Sutton views LLMs as both positive and negative examples of the bitter lesson. They enabled enormous scaling by drinking in the internet. However, they become limited by finite internet data while the world is massively bigger than stored information. This represents over-reliance on human knowledge that eventually holds systems back.
Sutton calls synthetic data generation a big mistake and the next bitter lesson. The big world hypothesis states the world is infinitely big with infinitely many things to learn. Synthetic data is bottlenecked by humans deciding what constitutes good versus bad data. Human expertise is required to determine appropriate synthetic data, creating a fundamental limitation.
Sutton uses the example of a drone learning echolocation like a bat. No amount of synthetic data from 50 different universes would enable this task without humans first figuring out the right approach. The world is infinitely complex, and any simulation is microscopic by comparison.
The big world hypothesis states the world is massively more complex than any agent's mind because it contains many other agents. This complexity means agents can never achieve anything optimal or perfect. Approximations will always be severe, requiring continual learning tuned to specific encountered parts of the world.
Sutton questions how many engineers are involved in building simulations and whether the only solvable problems are those where large engineering teams can first create simulations. Current approaches involve human-in-the-loop fixing of simulations based on real-world feedback.
Sutton advocates for agents learning models from their own experience rather than human-created simulators. When agents learn models themselves, incorrect models can be fixed through continuous learning. Human-created simulators only get updated when humans identify problems.
Sutton's essay states: "Seeking the improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain. But the only thing that matters in the long run is the leveraging of computation." AlphaGo represented a triumph of removing human priors, which made Sutton happy and felt vindicated.
Sutton clarifies there is no necessary conflict between prior knowledge and learning. Prior knowledge can help, and in principle these should work together. Life involves gaining knowledge and having knowledge. However, in practice they have been enemies, with people favoring existing human knowledge wanting to minimize learning.
Sutton argues that large language models represent the weird position that systems can achieve PhD-level expertise without learning anymore. The weights never change during interaction. The fundamental disagreement is that current systems stop learning after deployment.
Sutton envisions intelligent robots that can be copied digitally rather than retrained from scratch. Instead of spending enormous resources retraining from the internet, systems can copy agents and continue learning from wherever they are. Prior knowledge becomes dismissible through copying.
Sutton describes systems that learn from experience as potentially being robots, internet-based systems like package routing that improve over time, or user interface assistants that become better through interaction. Intelligent assistants must know what users want and improve over time.
Current LLM-based assistants have memories and do in-context learning, but their weights never change. Sutton emphasizes that all the structuring and generation of new concepts in creating large language models represents weight learning that should continue rather than stop after initial training.
Sutton distinguishes between changing model state through context versus actual learning through weight changes. Giving more context changes the state, but the model itself is not learning. Examples like Cursor's tab completion model involve actual weight changes and represent continual learning.
Current approaches collect data from millions of users and perform batch policy updates. This creates inefficiency when individuals want to teach models something specific without competing with thousands of other users' preferences. The shared knowledge approach doesn't allow personalized learning on individual model versions.
Context can live in the state as well as the weights, but systems still need the ability to update weights. Human disabilities provide good examples of this capability. People who lose proprioception completely cannot walk because that capability is ingrained in the brain, but over 2-3 years they can learn to walk again using visual feedback from looking at their feet. The brain is plastic enough that something true for 20 years can be updated when it stops being true.
Biological systems serve as a source of inspiration but not a constraint for AI development. No animals learn by supervised learning because they do not receive examples of how their muscles should twitch. Even in school, no one tells students how to produce specific muscle motions to pronounce words like "Paris" - each individual must figure out their own muscle movements. Supervised learning from others is important and language transmission is extremely important, but the fundamental learning of sensory-motor capabilities occurs without explicit targets.
School represents a very recent development in human history, only existing for a few hundred years, and is not part of the essence of intelligence. Animals like squirrels demonstrate sophisticated capabilities such as getting into bird feeders despite obstacles, jumping, climbing, calculating trajectories, and breaking falls in real time without any formal education. The Moravec paradox shows that advanced tasks like proving math theorems or playing chess are easy for computers, while basic sensory-motor tasks like moving and seeing with attention remain difficult.
World models should span from sensory-motor learning through higher levels of abstraction. The difference between human and animal capabilities is largely a question of degree, with society, culture, and language providing significant advantages. Humans 1,000 years ago had fundamentally the same brains but lacked 10,000 years of accumulated knowledge that modern humans access through school and supervised learning.
Paradigm shifts occur when humans accumulate knowledge through experience, build new abstractions, and discover new knowledge through planning. Current systems lack the ability to form new abstractions and then plan with those models. While games like AlphaGo work because the model is known, and mathematics works because operators are predefined, there are no instances of systems learning models and then planning with self-discovered abstractions.
The Alberta Plan consists of 12 steps to crystallize particular research directions. Step two focuses on continual deep learning as almost the most important capability because it unlocks everything else. The second half of the steps address getting abstractions right, where agents must learn correct abstractions for whatever world they operate in rather than having predefined correct abstractions.
Elite athletes develop niche terminologies for specific movements, sometimes without names for actions they perform alone. The question of how these abstractions emerge represents a crucial missing element that the later half of the Alberta Plan addresses.
Continual deep learning faces an algorithmic gap rather than just infrastructure or privacy issues. Naive single-sample weight updates cause catastrophic forgetting where previous knowledge gets destroyed. Current workarounds like using large batches from many users work only in specific use cases. Most applications involve single data streams where naive approaches fail.
The cure for catastrophic forgetting requires step size optimization where every weight has a separate step size that gets meta-learned. Most weights should have tiny step sizes so new examples don't destroy prior knowledge. Additionally, generate-and-test mechanisms in feature space create new units with random weights without following gradients, since gradients are slow and don't provide paths for sustained complexity growth.
Continual backprop extends traditional backprop by planting new seeds of units initialized with random weights throughout training rather than only at initialization. Backprop's randomness gets used up over time, while continual backprop continuously injects randomness through generate-and-test, with backprop serving as the tester to evaluate new units.
These algorithms cannot be applied to existing models because they meta-learn how to learn. New foundation models must be trained from scratch with these algorithms, simultaneously learning knowledge and learning how to learn future things. This enables learning new things without catastrophic forgetting.
Previous exploration of these ideas occurred in limited settings with specific problem distributions, whereas the goal is general applicability from a single stream of experience. The most ambitious objective is achieving the full spectrum of knowledge from tiny details to large abstractions like airplane route planning, treating all scales uniformly with self-maintaining consistency.
A trillion-parameter model operating at 20 watts represents an ambitious target. Current memory technologies would exceed 20 watts just for storage, but 5-10 years of computational efficiency improvements could make this feasible. Two orders of magnitude improvement over 10 years aligns with Moore's law projections, suggesting current systems could achieve this with 2,000 watts using the right algorithms.
Current AI development appears stuck in a local minimum where moving toward new algorithms will initially produce worse performance before improvement. Large labs locked into product paradigms cannot pursue paths where performance degrades first. The new paradigm requires solving difficult technical challenges that demand long-term thinking about problems that make sense when discussed with researchers who have considered them for many years.
If successful, the architecture would enable genuine continual learning, abstraction formation for planning and reasoning, and true intelligence. Large language models might face competitive risk from such systems. The vision emphasizes that humans would not become irrelevant but rather the world would become more exciting and interesting for humans.
Large language models represent an amazing scientific breakthrough in the skillful use of language by neural networks, a development that was wholly unanticipated. The field had always held out for symbolic methods in language, but neural networks have completely changed how language processing is thought about. Despite this breakthrough, the concern is that large language models might be at risk when certain developments eventually occur, though they have already had a successful run.
All of intelligence is not fluid, capable use of language. While language represents an important part of intelligence, estimated at around 20% or a quarter, there is significantly more to intelligence than language capabilities. The frustration lies in celebrating progress in a subset of the AI problem while pretending it represents all of AI. There is much more to intelligence than what large language models currently demonstrate.
Future AI systems are envisioned as operating under a single design that can adapt to different environments. However, there will be many different minds rather than a single mind capable of everything from learning to swing from tree branches to building spaceships. Different versions of that mind would learn different things because they have different experiences.
The big world hypothesis suggests there are infinitely many things to learn, making it impossible for one system to learn everything. If there are two of the largest systems in the world, it is trivial that they cannot model each other because they are equally complex. A single system would never be able to reach a point where it can learn everything, as multiple systems will always be learning from their own experiences.
The team is hiring, with most of the initial team already identified in mind. These are people who have thought about these ideas for a long time. The approach will be different because this represents a different paradigm. It doesn't make sense to become large very quickly because everyone hired has to come to see what the team sees, and not everyone sees that perspective. The plan is to start small, slowly grow to maybe a handful or two or three people, and then expand from there.
The goal is to be super aligned to enable very productive collaboration and scaling of progress. This alignment is considered essential for effective teamwork and advancing the development of these new AI approaches.
The conversation explored deep thinking about where reinforcement learning and algorithmic design will go, representing a true pleasure in exploring these ideas together.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
