Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
In a Nutshell
Engram's core claim is that the next leap in usefulness comes from internalizing private context directly into model weights via continual adapter fine-tuning, rather than relying on external memory or ever-larger generic models. By compressing company documents, workflows, and interactions into compact LoRAs or similar adapters, inference cost drops up to 100× while the model gains the implicit knowledge of a long-tenured employee. The approach treats training as an always-on process that decides what to bake in versus leave in retrieval, solving the scaling limits of token-heavy context engineering.
These notes were generated by AI and may contain inaccuracies.
The fundamental question is what about pre-training or post-training enables models to generalize in emergent ways, and how to control that process so a company can make models learn private data just as deeply as they know the capital of France or how to write Python.
Engram is a new lab focused on memory and continual learning. The website states they do not see the world through pre-training or post-training. Their models are always training. The bottleneck for making models more useful is not raw intelligence but understanding new and evolving context, whether a new task or a particular job context. The goal is to bake that context into the model weights the same way pre-training and post-training do.
Memory as a separate database or context window is not true memory or true continual learning. Context engineering, tool use, and large prompts help but are insufficient. The underleveraged approach is applying the same training pipelines frontier labs use for math or code to every domain and private company context. As individuals, taking notes and using sticky notes is valuable, but people also carry internalized traces of memory and new intuitions into the next day. Current solutions are more externalized memory.
The amount of tokens generated collectively will reach tens of millions per day soon. Keeping, searching, and rereading this volume will be expensive and confusing for models without major breakthroughs. The vision is combining externalized memory with internalized traces.
Sign in to read the full notes
Get access to AI-generated notes, topic timestamps, and more.