Jeff Dean: The 1% Rule for Building in AI
In a Nutshell
Jeff Dean predicts AI will automate its own development by 2027 through recursive self-improvement—automated experiment loops that decompose problems, run trials, and produce better systems across science and engineering. The scarce skill will shift from coding to "taste"—knowing what problems are worth solving—while startups win by building domain-specific systems around data general models lack or problems where frontier models currently fail 99% of the time. Hardware specialization for inference and data-efficient algorithms represent the biggest opportunities, with energy as the fundamental constraint shaping system design.
These notes were generated by AI and may contain inaccuracies.
Jeff Dean stated that AI models are now at the level of a junior engineer, which aligns with his May 2025 prediction at AI Ascent. The models have improved significantly at agent-based, longer-running coding tasks, and depending on the definition of junior engineer, the prediction appears accurate.
Dean underestimated how quickly models could handle increasingly complex tasks and how agent-based systems are expanding beyond coding into other domains. For 2027, he predicts significant automation of ML systems themselves, where systems will run experiments, break problems into subproblems, execute automated experimentation loops, and produce improved systems through fully automated problem decomposition.
This automation extends beyond ML to other fields of science and engineering, particularly where measurable objectives exist.
In 2001, Google search ran on hard drives. Dean and Sanjay Ghemawat calculated that the search index would eventually fit in RAM across all computers, leading them to ship a new production search version that worked in RAM rather than hard drives within days. This made Google searches significantly faster.
The equivalent moment today involves high-performance, low-energy inference hardware systems. Inference is becoming key for making agent-based systems available to more people, with latency being critical. Hardware specialization provides energy efficiency and lower latency compared to general-purpose devices like GPUs or TPUs.
Dean suggests imagining capabilities with 50x better latency.
A key false assumption is that agent-based systems can only run for an hour or two on problems. With highly capable models, agents can run for days or weeks on complicated tasks. This capability is emerging but not fully internalized by everyone.
Agents can implement entirely new versions of software in different programming languages with better safety or performance properties, executing these tasks in a serious, sustained manner.
In 2013, when speech recognition began working at Google, Dean calculated that if every Google user used speech recognition for just three minutes daily, the system would require doubling Google's entire server fleet. This calculation led to building custom chips, which became the origin of the TPU.
Speech recognition models using deep learning halved error rates compared to previous systems, representing approximately 20 years of advances in just months. The concern was that better speech recognition would increase usage for dictating emails and talking to phones.
TPUs are specialized for low-precision dense linear algebra, which forms the core of modern machine learning algorithms. This specialization made them 30 to 80 times more energy efficient than CPUs and GPUs of the era, with 20 to 30 times lower latency. The TPU was built as a general-purpose linear algebra system because ML algorithms were still evolving, avoiding over-specialization while achieving dramatic performance benefits.
Founders should identify bottlenecks in current approaches and consider fundamentally different solutions that could achieve order-of-magnitude or two-order-of-magnitude improvements in performance or capability. Approaching problems from first principles rather than current solution methods can yield novel ideas.
Key measurements for AI systems include:
- Bandwidth between main memory on accelerators and on-chip memory
- Energy cost of single multiplier operations
- Interconnect bandwidth between chips and connection capacity
- Network bandwidth degradation when scaling from 500 to 10,000 chips
Energy has become the primary measurement unit. A calculation costs about one picojoule, but moving data costs approximately 1,000 times more. This thousand-fold difference between data movement and computation shapes machine learning decisions, requiring batching of many examples or tokens to amortize data movement costs. Low-latency applications cannot effectively use batching.
Training models with batching and epochs is fundamentally a systems and data IO problem rather than a model problem. Ideally, batch size one training would be preferred, but efficiency demands larger batches.
Dean has been focusing on inference rather than training, as inference requires very low latency. There is significant room for hardware specialization for inference, including minimizing data movement and using extremely low-precision operations without supporting multiple precision types.
AI can be understood as a compression problem: fully lossy compression and restoration of data requires understanding it. The transformer architecture has proven effective at this compression task.
AI progress now extends beyond better models with more data and larger parameters. It encompasses retrieval tools, memory, agent tools, and context engineering. The model represents only one piece of building systems that solve interesting problems.
Context information is clearer to models than training data, which exists as trillions of tokens mixed into hundreds of billions or trillions of parameters. Context provides direct, clear information for specific problems.
Effective context engineering involves understanding available tools, selecting appropriate ones for problem phases, decomposing problems into tool call sequences, trying multiple solution approaches, and evaluating results. This orchestration of complex agent and multi-agent systems is becoming increasingly important.
Context engineering is accessible to individuals with API access to models like Gemini, without requiring massive GPU resources or training data.
The most effective approach involves using models, harnesses, and tools to solve problems, then identifying where models fail. Rather than adjusting model parameters, creators can write better guidelines and skills for models to use different tools effectively. This creates self-improving setups.
Dean and Sanjay Ghemawat wrote skills teaching models to perform microbenchmark measurements, code changes, performance measurements, and iteration cycles. This enabled self-improving benchmark optimization that worked well for certain problems.
Dean and Ghemawat published a 30-page document called "Performance Hints" containing various performance tricks. Some people have used summarized versions of this document with models to improve their reasoning about performance issues in code.
Current agents often fail after approximately 10 tool interactions. This occurs because models encounter tasks outside their training distribution, causing performance degradation as they move further from familiar patterns.
Strategies include providing skills and hints that keep models on familiar paths, implementing multi-agent systems where multiple agents try different approaches, and using evaluation models to identify promising solutions. Inference-time compute for searching plausible solution paths can achieve higher performance and reliability in long-running agent flows.
Google uses harnesses and skill sets in their internal development environment, enabling agents to use internal tooling for coding, code reviews, performance measurement, and log file fetching. These skills make base models more capable without specific training on internal systems.
Google focuses on building general models that can do almost anything, which means less attention on particular domains. Well-designed surfaces with models and skill sets, or specialized models, can have significant advantages in domains where founders are passionate, achieving delightful, high-accuracy, high-quality results.
Founders must assess whether their focus area will remain durable or if frontier models will improve at that capability within 6-12 months versus 2-3 years.
The most important selection criteria when picking a problem is to choose something you're super excited about and want to build that would be useful in the world. If you're not excited or the problem isn't actually useful to many people, you're already at a disadvantage.
The second criteria is to examine what current general models can do in that domain. Test whether they can handle the task very well. If they're completely failing, that's a good sign. If they're able to do some of it but not very well, that may not be a great sign because the capability is already starting to be present in those models and will likely improve with more training data or larger scale.
Look for problems where the model succeeds 0% or 1% of the time, not 20%.
One approach is building a product that has access to particular kinds of data that the underlying general model doesn't have. For example, organizing all of a user's personal information gives you a big advantage because the product has visibility into important data the general model lacks.
Another approach is tackling incredibly hard problems where you can get the right training data and train a more specific model than a general purpose one in an affordable way. The niche model might not require much compute but can be highly accurate, serving as a good building block for solving important problems not handled well by general models.
AlphaFold is a very specific model for protein folding that was highly successful and handles that domain quite well, providing effective answers about proteins and their structure. It's not a general model but a very specific one.
Other domains where this approach works well include material science and chip design. These enable leveraging capabilities of a very accurate but niche model to solve problems that are hard today.
Managing a fleet of agents is about writing really good, crisp design docs or specs. You'll have more success working with virtual agents when you can clearly specify what you want. The clearer you are on what you want, the more the agent has guidelines, rules, and an outline of what it's trying to accomplish.
If you don't specify very much, the agent has to infer what you meant, and it might infer things different from what you imagined. Specifying what the software is trying to accomplish before writing it has always been important, and now with agent-based systems that can do the writing, the importance of clear specifications has increased.
Translating software from one computer language to another works extremely well because you have an incredibly detailed specification—the whole software that says what the system is supposed to do. A Python implementation can be translated to Go effectively because the model can take all the tests in Python, make sure they pass in the Go version, translate the tests to Go, and compare behavioral differences until there aren't any.
Once everyone gets good at running hundreds of agents and all code is written by agents, the scarce skill becomes having incredibly good taste in what you ask your agents to work on. This is the crux of research problems—researchers can have all the tools and techniques, but most of the battle is what problem to spend time on. Picking the problem well and succeeding is better than delightfully executing a research investigation into a boring problem.
High-level wisdom of what to work on is incredibly important, and models aren't necessarily going to be good at it. People will steer AI-assisted computation to accomplish great things, but the essence of what you want your models to do is the key thing to focus on.
Taste is difficult because there's no measurable objective in many cases. Some of it comes from experience—working on many different problems teaches you what kinds of problems might be interesting in the future, what might be barely possible by combining previous approaches, and what open problems need work to reach something magical or highly useful.
Another way to build experience is to write down things you think might be important in the next 12 months. Pick one to work on, but go back in 12 months and evaluate which ones actually seemed important, which ones other people created, and which ones weren't pursued. This gives more samples for developing taste.
It's good to question assumptions that most people take as given. A thought experiment involves questioning the 60-year assumption in the silicon chip design industry that every chip manufactured of the same design should be identical with very low error rates—no bits should flip. Memories have ECC, and there are error margins built in.
At the macro scale for large distributed systems, we build reliable systems out of unreliable parts—individual disks can fail but data should be safe through mechanisms like three copies on different machines and racks, or Reed-Solomon encoding. But we don't seem to do this at the extreme transistor level.
The thought experiment asks what would happen if you built a system out of transistors with 20 errors per day rather than one every million years. This would be a very different design point enabling different fabrication approaches and design methodologies, such as sending signals along multiple redundant paths to ensure delivery.
These thought experiments don't always work out because there may be good reasons for doing things the current way, but it's good to revisit assumptions periodically.
TPUs are an example of specializing hardware for a very niche problem domain before that domain seemed as important as it is today.
MapReduce originated from working on crawling and indexing systems at Google. The team had written hand-parallelized code with checkpointing for robustness, but the relatively simple logic got obscured by parallelization and reliability code. Remembering training in functional languages, they developed the MapReduce abstraction where the implementation below could handle checkpointing and reliability, becoming a successful way of dealing with large-scale computations robustly.
The scientific method foundation involves proposing experiments, implementing what you need to run them, evaluating them, and getting results. More problems are now possible to implement where the entire loop can be automated with extremely low latency. This enables tackling many problem domains in science, engineering, machine learning model design, and engineering tasks like chip design.
An orchestration framework can take high-level objectives, break them into subproblems, have each subproblem run automated loops exploring the best solution, then put together subproblem solutions into the overall solution. This accelerates machine learning progress, science, and engineering.
Domains ripe for self-improving AI systems are those with good evaluators and things that can be formally verified. Sometimes evaluators need to be made much faster.
Work in quantum chemistry involved generating molecule configurations and understanding their properties using density functional theory simulators that might take a night of computation per molecule. Researchers took outputs from simulation runs and trained a neural approximation to the simulator—300,000 times faster and nearly as accurate. This changes how science is done because screening 10 million things becomes feasible during lunch rather than a six-month endeavor.
There's room in many domains for much faster validation models, possibly learned validation models that provide approximations to the true answer rapidly, changing how experimental loops are conceived and how quickly you can iterate.
Machine learning itself is a domain where models could recursively self-improve by running lots of experiments. Currently, large research teams think of ideas, run small-scale experiments, evaluate promising ones at larger scale, and integrate results into new model recipes. There's no impediment to making this a much more automated loop where the model itself decides to explore, possibly with human nudges at the highest level.
The goal is optimizing discoveries per unit of compute input.
A paper on distillation written with Geoff Hinton and Oriol Vinyals in 2014 was rejected at NeurIPS. The paper addressed taking a big teacher model to train a much smaller, more efficient model with fewer parameters that's cheaper to compute. One reviewer said it was unlikely to have significant impact.
The authors saw this as a super important problem because making cheaper, highly capable models was desperately needed to serve models to more people in domains like speech and vision. The paper was put on arXiv, people read it and used it. It's used in making Flash models from larger Gemini Pro models, which is partly why Flash models are so capable relative to their size and speed—they're among the best in benchmarks for their model size class.
The lesson is that even if you get rejected, keep going.
Important questions to ask are whether you're working on something you really care about, whether you can make progress with colleagues you like working with, and whether solving it will make a positive difference in the world—helping biochemists, programmers, consumers, or others.
The goal should be having positive impact, working with people you enjoy, working hard, and doing your best.
Joining a frontier lab versus starting a company with one or two or three close friends are different experiences. Large organizations provide structure, amazing colleagues with knowledge you don't have, interesting problems, and a platform for impact. Small startups require passion and involve risk but can be incredibly rewarding.
Regardless of path, ask yourself if the best possible outcome would make the world a lot better or if the world would go "eh, that's kind of cool but whatever."
Find people with really good skills in areas needed for the team, whether inside a company or starting one. Also find people you delight being around because you'll spend a lot of time working on hard problems together.
Look for people who are low ego, team players, with complementary skills. Working in small teams where people know things you don't and you have skills others don't have is super fun because you're collectively building something none of you could do individually, while gaining new knowledge and skills.
View your engineering or research career as having an amazing tool belt of techniques. Always add new tools because you never know when you'll encounter a problem where you need them.
You never know when you might come across a problem where you need these four specialized tools rather than these three. Adding more tools makes it more likely that the problems you encounter in the future will be solvable by you.
Someone in this room or multiple people will eventually build something as consequential as map reduce, TPU, distillation, and similar innovations.
There are a lot of interesting problems in the world, and this list is not exhaustive because the world is a very big place and full of problems.
New Hardware Approaches
Particularly exciting are new approaches to hardware, including much more efficient inference hardware.
Data-Efficient Algorithms
There are radically different kinds of algorithms for machine learning that might be much more data efficient than the approaches we're using today. Large scale models today probably see a thousand times as much data as a human does by the age of 18. Yet the human by the age of 18 is better in a lot of things and on par with those frontier models that have seen way more data.
Continual Learning Systems
Could you come up with much more data efficient systems that can learn continuously from their own actions? Continual learning is a really interesting thing.
Multi-Agent Interactions
Multi-agent interactions is an interesting thing.
Better Human Discourse
Creating ways of having better discourse among people in the world could be interesting. Are there ways to have much more civil conversations and helping people meet other people all over the world that they should know based on their interests? These are kind of interesting things.
There are lots of cool things in the world and we should all go and strive to make even cooler things occur.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
