Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil
In a Nutshell
Core Automation was founded because current transformer-based systems cannot learn or adapt at test time on new real-world tasks, and the founders believe the architecture itself—not just scaling—is now the bottleneck to AGI. Their mission is to automate the entire AI research lab, particularly kernel optimization, to enable rapid architecture experimentation at scale. The key insight is that models need new learning algorithms capable of meta-learning and continual adaptation, rather than relying on pre-training plus RL alone.
These notes were generated by AI and may contain inaccuracies.
Football learning resembles reinforcement learning through repeated attempts with incremental adjustments and self-reinforcement based on outcomes. Mathematical learning operates differently, involving reading complex concepts and deep internal thinking until connections form. Both represent learning from experience but through fundamentally different mechanisms.
Current AI systems likely allocate more compute to learning from experience than ever before, though reinforcement learning represents only one approach. Better methods for utilizing experiential data will emerge from research in coming years.
Jerry Tworek previously served as VP at OpenAI, leading the strawberry and reasoning teams. Rohan Anil held one of four pre-training lead positions at Gemini and conducted fundamental AI research at Google Brain, serving as the fix-it person across Google and later at Anthropic.
Jerry's statement that "the first step to replacing transformers is appreciating deeply how far they were able to carry us" emphasizes understanding both capabilities and limitations. Appreciating transformers requires recognizing what they solve effectively before addressing their weaknesses.
Current architectural work focuses predominantly on making transformers cheaper and more efficient rather than more powerful or expressive. Understanding the shape of transformers—both strengths and weaknesses—reveals the same underlying analysis.
Two algorithms have been mastered at large scale: pre-training and reinforcement learning. The architecture itself now represents the bottleneck to better models and smarter systems, requiring reconsideration of the six-year approach of adding more parameters to MoE and attention operations.
Codex, Claude Code, coding workflows, automation systems, and current products represent six years of scaling achievements. However, successful Codex usage prompts questions about why tasks weren't pushed harder, why certain work still requires human intervention, and why Codex doesn't handle everything.
Core Automation's mission centers on automating tasks that remain unautomated despite current capabilities.
Models trained in laboratories face deployment in real-world conditions, creating fundamental tension. Jerry's disappointment stems from his RL maximalist belief that scaling reinforcement learning would solve all problems and achieve AGI by 2025.
Despite training successive models with improving benchmark scores, real-world task performance did not match expectations. Benchmarks essentially replicate training distributions, while real-world tasks present messier, murkier distributions that training data fails to replicate.
Models require learning at test time with user data on real-world tasks and distributions. Current options include in-context learning, which avoids catastrophic forgetting and offers data efficiency but remains limited in scale—approximately 20 minutes of Codex usage before compaction becomes necessary.
Fine-tuning presents catastrophic forgetting and low data efficiency problems that remain unsolved despite extensive research efforts.
New algorithms must be meta-learnable and expressible at the architectural layer to represent learning processes operating over much longer horizons.
Transformers and reinforcement learning scaling occurred simultaneously, though economics primarily drove transformer adoption. LSTMs could technically scale but demonstrated inferior scaling laws and would have been more expensive to train with less impressive product outcomes.
Transformers generate training costs lower than revenue produced, creating economic viability that LSTMs likely would not have achieved.
OpenAI's bet involved prioritizing smaller, more scalable algorithms over increasingly efficient ones, pouring more compute to achieve better results despite community criticism. This approach enabled current model capabilities.
Many architectures likely require baseline compute levels to demonstrate interesting or useful capabilities, similar to reinforcement learning's need for sufficient ability baselines.
Major labs face intense competitive pressure that discourages exploration of alternative paths when transformer scaling remains profitable for near-term wins. Smaller labs replicate what successful labs do, creating ecosystem uniformity.
Different approaches and paths are needed, which Core Automation aims to provide.
When transformers were invented by Ashish, Noam and others, the work was presented alongside online distillation research at an internal Google conference. Few people recognized its significance initially.
Original transformer work focused on solving real problems—beating LSTMs on translation. Internal Google interest in scaling language models existed, but GPT-2 and GPT-3 demonstrated transformers' broader benefits.
Architecture determines how computation gets spent. Current industry computation focuses heavily on inference time and token generation. Pre-training built transformers with certain context lengths, while RL adds computation through one-token-at-a-time chain-of-thought reasoning—an inefficient approach from inference perspectives.
Most trained transformers remain quite shallow, with at most 100 layers. Deep learning's goal involves deeper representations, yet experiments haven't demonstrated extremely deep representation learning.
Chain-of-thought reasoning and RL increase computational depth by adding pathways per token, but this creates inference-time scaling that requires more tokens for better results through one-token-at-a-time autoregressive decoding.
Transformers excel at distilling internet knowledge and relationships, plus baking desired workflows through RL. Any task with training data can be incorporated into gigantic models trained with substantial compute.
Transformers lose usefulness when new events, relationships, tasks, codebases, or tools emerge that weren't present in training data. Adaptation ability exists but remains limited and inflexible.
Transformers function as tools requiring human scheduling and prompting, with failed tasks added to training data through lab-based retraining loops.
Reinforcement learning represents an established approach. The Age of Experience paper's concepts have always existed in some form. Pre-training criticism often contrasts it with other model-building approaches.
Pre-training today largely involves distilling other models into new models, as most internet tokens originate from AI systems. Pre-training functions as behavioral cloning, mimicry, and internet data compression.
Reinforcement learning has been used historically to solve backgammon, Go, StarCraft, Dota, and programming problems. The core mechanism involves models writing their own experience and learning from that experience. However, reinforcement learning is not the only way to learn from experience, and there will be more algorithmic innovation in how we learn from experience. The mathematical formulation of reinforcement learning currently favors parallel rollouts for variance reduction and comparing model performance across parallel versions of the world, which differs from how humans learn from experience.
Jerry Tworek explains that human learning involves multiple algorithms working together. Learning football resembles reinforcement learning through repeated practice and adjustment, while learning mathematics involves reading hard concepts and thinking deeply until connections form. Both are learning from experience but operate very differently.
Rohan Anil emphasizes that pre-training as currently defined focuses on compression measured through perplexity reduction, where scaling parameters and compute yields epsilon improvements in metrics. This approach works for building priors but misses the end-to-end perspective. When Jerry takes a pre-trained model and applies RL, it destroys the perplexity metrics that were optimized for. The key insight is that pre-training and RL should be combined together, which could yield one order of magnitude improvement.
Rohan traces his optimization work from logistic regression solvers at Google in 2016 through the Sybil large-scale linear solver, to developing the Shampoo algorithm with Vinit Gupta. The community was initially uninterested in optimization improvements, but the work found application in Gemini 1.5 Flash. Shampoo provides roughly 2x improvement over existing methods, but remains limited in utilizing available training information. The relationship between optimization and architecture is bidirectional - stronger optimizers can train harder-to-optimize models, while weaker optimizers work on easier-to-optimize architectures.
Jerry states that surpassing biological learning efficiency is unlikely with current hardware approaches. Biological systems build their own circuits and learning algorithms during growth, then disappear, while neural networks maintain fixed hardware that learns inefficiently through massive parallelism. Hardware would need to become more analog with better error correction to approach biological efficiency.
Jerry defines AGI as a model that can improve itself without humans in the loop, arguing this is a meaningful sub-definition of OpenAI's definition of outperforming humans in economically valuable work. Current paths won't achieve this, requiring serious research into making models learn and adapt at test time at a deeper level.
Combining pre-training and RL presents non-obvious optimization challenges due to different characteristics - pre-training uses large batches with more parallelism, while RL involves rollouts with higher variance. Even leading researchers acknowledge the logic but face pressure from release cycles that make long-term 6-month research difficult.
Core Automation aims to be the most automated lab in the world, focusing on giving humans maximum agency rather than removing humans from loops. The goal is to empower researchers to iterate on ideas much quicker through automation. The team is rebuilding the deep learning stack to enable one experiment per day, potentially scaling to 10 or 200 experiments daily.
Jerry emphasizes that current coding agents represent progress toward the AI scientist vision he worked on at OpenAI. The speed of iteration and data gathering on ideas fundamentally changes what's possible, requiring native processes rather than retrofitting old structures.
The immediate focus is kernel generation automation. Core Automation has a set of GPUs that require training and inference, and the bottleneck is moving from architectural ideas to highest TFLOPS implementations. Current coding agents plus humans can achieve 7x improvements on operations like QR factorization, but the highest performance requires exceptional human taste and significant resources.
A competition with GPU mode demonstrated that achieving 60x speedup on QR kernels for B200 nodes required the top human expertise plus $100,000 spent on coding agents over 4 weeks. Current models cannot solve these kernel optimization problems, representing a real bottleneck for efficient architecture development.
Rohan describes three projects where kernels became bottlenecks. At Google, he couldn't spend 10x compute for 2x improvement, requiring 20% budget for 2x gains. Novel operations had limited expert availability - only two humans at Google could write deep XLA code for the work. The N-gram memory project required hardware changes for sparse gathers and scatters that never materialized.
Kernels enable entirely new algorithmic approaches once performance reaches roofline levels. QR factorization is fundamental for eigen decomposition analysis and optimization methods like Shampoo. The scarcity of kernel expertise across the industry makes automation critical.
Core Automation's quest is finding a transformer replacement capable of continual learning, which current transformers cannot support. The approach involves experimenting at scale with strong priors to efficiently search architectural space.
Success indicators include plots showing performance curves that bend differently from existing approaches. Jerry notes that successful research typically involves knowing what you're looking for before the definitive plot appears - the best plots often materialize in dreams before reality. Deep learning systems require getting five things right in sequence for meaningful results.
The ultimate evaluation is whether systems get better at doing Core Automation scientists' work each day, with the team testing this by going on vacation to see if the lab produces better work during their absence.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
