Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Work
In a Nutshell
Waymo's co-CEO warns that a working demo represents only 1% of the effort needed to build a real physical AI product—getting the remaining "nines" of reliability requires fundamentally different approaches than achieving the first 90%. The critical lesson is that architecture choices, technology bets, and evaluation frameworks must be dictated by the exact number of nines required for safety-critical deployment, not by what's easiest to prototype. The company's success stems from building an integrated AI ecosystem (agent, simulator, critic) around a foundation model, using structure-augmented end-to-end learning, and earning trust through rigorous safety validation at scale—now achieving 17x better safety than humans across 220+ million autonomous miles.
These notes were generated by AI and may contain inaccuracies.
Good afternoon everyone. It's great to be here. We talk a lot about AI that lives on your screen, lives in the digital world. Today, let's talk about a different kind of AI being built at Waymo: AI that lives in the real physical world.
Many people in the audience have ridden in a Waymo. For those visiting who have not had a chance to check out Waymo, the invitation is to give it a try while in the Bay Area. This presentation is structured as a sequence of seven lessons learned over the years at Waymo around what it takes to build and safely ship today's most mature application of AI in the physical world: the Waymo driver.
A clip from a ride taken in a Waymo with kids shows moving forward through an intersection when a couple of human drivers cut in right in front. The Waymo driver reacted safely and smoothly. The kids in the back seat were preoccupied and didn't even notice that anything happened. This was a powerful moment after working on this technology for close to two decades. It acted safely, kept the kids safe, kept everybody safe, and nobody noticed. This will be a theme: for physical AI, the best AI moments will look like nothing happened. The task got done safely and smoothly. These moments where the Waymo driver kept everyone safe are happening daily across the fleet.
Today, the Waymo driver is serving around 500 trips per week and driving over 4 million fully autonomous miles every week in 15 cities across the United States. That's over 300 years every week of an average American driver per year. The Waymo driver is accomplishing that with a superhuman safety record.
In Silicon Valley, there's a common mantra to move fast and break things. However, when dealing with atoms instead of bits, breaking things is not really okay. The approach must be to move fast and ship safely. That's a much more difficult thing to do. Systems must be built that are robust from day one. AI models and training recipes must be built where safety is the foundation and not an afterthought, not an add-on.
The problem of physical AI is different from digital AI. There are four main gaps to contend with when building AI for the physical world versus the digital world.
Cost of Error Gap
If a language model or a chatbot or a co-pilot makes a mistake, it usually costs a retry. In the physical world, the cost of a mistake can be measured in human lives, not tokens. There's simply not an undo and a retry button.
Latency Gap
When running a VLM or a digital assistant, it can take many seconds, sometimes minutes to come back with an answer. A car traveling at freeway speeds moves about 100 feet in 1 second. Milliseconds really matter. All inference must be run and all decisions made on board a compute that fits in a trunk of a car.
Data Gap
Digital AI had the internet, this wonderful immense cache of pre-labeled human knowledge and human thought that has been assembled. There's no digitized version of the internet for the physical world.
Validation Gap
In digital AI, often something good enough can be shipped and users can find the edge cases. That allows deployment on day one practically at unlimited scale, then iteration and hill climbing on quality from there. In physical AI, the situation is different. Given the high cost of errors, there needs to be a very high level of safety and a very high level of confidence on day one before deploying the first robot, before driving the first autonomous mile.
At the same time, when dealing with physical AI, the actual experience of having the agent in the real world is invaluable and irreplaceable. These systems are not just something that can be built in the lab, get perfect, and then deploy at full scale overnight. Given those two factors, operating conditions and deployment parameters of the agent need to be super clearly and super crisply defined, then a rigorous framework built to guide deployment so that scaling can happen in a responsible manner. This is absolutely critical. This is how trust is earned from customers, from communities, from regulators, and from oneself.
At Waymo, these gaps are seen in the context of autonomous vehicles, but these gaps will show up in practically any sort of non-trivial physical agent deployed in some shape or form. Driving is just simply the first domain where AI has crossed these four gaps at scale with the public interacting with the product. Seven lessons have been learned over the years at Waymo from working on this problem. They're all technical. A lot more goes into building a company and building a product, but today the focus is on the technical aspects of building AI for the physical world. Each lesson by itself will not be exactly earthshattering. A lot will overlap with things heard elsewhere. The grounding of these lessons in the experience and some of the nuance about how they showed up in the experience of deploying a physical agent and scaling it safely will hopefully be interesting and useful for those in the space building products and startups.
The first lesson has to do with this massive, frustrating, sometimes soul-crushing difference between a demo and a real product. A working demo is 1% at best of the work that has to be done. The many nines of performance, the many nines of reliability that follow, that's where the real work happens. If a founder is in the room, chances are the focus is on getting that first prototype, that first demo, off the ground. When that first version of a system that works, that first 90%, when the demo actually works, it feels incredible. It feels like it's solved, the sky's the limit, extrapolating forward.
In this world, that first milestone, that first 90%, was hit back around 2010. When this project started in 2009, before building the system, a couple of pretty ambitious goals were set. One was to drive 100 autonomous 100,000 miles in autonomous mode. The second goal was to drive 10 routes, each one 100 miles long, chosen to cover a variety of conditions across the Bay Area. Each one had to be done from beginning to end without a human intervention. At the time, there was a team of about a dozen engineers. Both of these goals were accomplished in about a year and a half. This was well before any of the AI breakthroughs, before convnets, before transformers, before VLMs, before any of the stuff talked about today. Yet it got done. By demo standards, autonomous driving was solved in 2010. Everything was handled: day, night, traffic, pedestrians, cyclist traffic, traffic lights, construction zones, freeways, surface streets. So capability complete was achieved. At the time, it felt like being on top of the world. But then, as building towards a product began, a brutal reality was quickly run into: there's a massive difference between doing something once or driving 10 routes once and building a scalable service with nobody behind the wheel.
It took about 10 more years to begin providing a service and then five more years to scale to half a million trips per week. The demo took 18 months. The product took about 15 years. Now scaling is exponential. To date, well over 20 million fully autonomous trips have been served, and well over 200 million fully autonomous miles have been driven. Rider-only vehicles are operating in 15 cities across the United States. Scaling is exponential. It took 15 years to get to that first 100 million miles and about 7 months to drive the next 100 million. It took about 8 years to go from the time when the initial rider-only operation started to the time when riders were being served in four cities. Earlier this year, four cities were launched in just one day.
Bridging that gap from demo to product takes so long because there's this harsh engineering reality that you can't really cheat. Reliability and performance live on this exponential ladder of nines. Getting to that first 90% or 99% is the easy part. But then every next nine that is wanted takes about 10 times more effort. Exactly how many nines the product actually needs must be known upfront. A demo might need one nine. An assist product or co-pilot might need a few. But a fully autonomous AI agent being put out in the physical world that engages with the public, with kids running around, needs a whole stack of them. At scale, the long tail is the problem space. It's the entire problem statement. When driving millions of miles per week, a rare event that might happen once in a million miles just becomes daily reality. Getting those next nines means doing something different every time. Six nines of performance or reliability cannot be achieved by doing the same thing that was done to achieve the first two, but longer. Fundamentally different things must be done. A fundamentally different approach is required.
For example, with reliability, the first couple of nines can be achieved by just doing proper engineering and some bug fixes. But to get to the next few, fundamentally different approaches must be invested in. Fully redundant systems must be built, tiered fallback architectures must be had, and so forth. The same thing holds for the performance of AI models. In this space, it's incredibly easy to get started, but it can be excruciatingly difficult to get to the real product. That effect is only amplified with every wave of technological breakthroughs. That naturally leads to hype cycles. Every AI breakthrough from deep learning to convnets, VLMs, you name it, makes it that much easier to get started. Demos and prototypes get 100 times easier. But the tail, that's where the hard problems are, that moves much less. It moves, but the effect is muted. That's why every hype cycle produces a wave of absolutely spectacular demos and very few real products. The recurring mistake of every cycle is spending on the demo what should be saved for the nines.
This being a startup school, the last thing is to throw too much cold water on the magic and the excitement of those early days. This time is absolutely magical. It's amazing. Cherish it. Leverage it. But the key is to remain honest about the product being built, the number of nines in performance and reliability that that product demands, and not cutting corners to get there. Otherwise, there might be a pretty rude awakening later. Count the nines before counting the demo views.
This brings us to the second lesson. Once it's known how many nines the product actually needs, it fundamentally dictates the architecture and the core technical approach that must be pursued. Every technology has a performance versus effort curve. They all tend to start fairly steep and go up, and then they flatten out. As just mentioned, every other nine gets an order of magnitude more difficult. A common failure mode is picking the tech that gives the fastest early ramp, riding that steep curve, feeling like winning, projecting that steep slope into the future, feeling like the sky is the limit, and then hitting the plateau and discovering that the technology path that was picked actually flattens out way before the performance that is required by the product. The choice might still be made to be on that steep curve for a while for a variety of practical reasons. Maybe prototyping something or demoing something or building something in service of learning is wanted. But be honest with yourself: are you building for the purpose of a demo, for the purpose of learning, or towards an actual product.
Let's take an example from the autonomous vehicle domain: sensing. There's been a long-standing debate about what kind of sensors are actually needed for autonomous driving. Naturally, more sensors means higher performance, but also means higher complexity. Humans can drive with just eyes. There's that proof of existence. If the goal were to just approximately match human performance or to build an assist product, that's a very reasonable way to go. However, if targeting full autonomy and targeting superhuman, strongly superhuman performance, weak sensing just leads to a safety curve that flattens out way too early. At Waymo, an approach has been taken where multiple sensing modalities are used: cameras, lidars, and radars. They all complement each other. Cameras give high resolution and color, but they're passive, and they degrade in darkness and glare. Lidar gives a direct measurement of the 3D structure of the world around. Radar is very good at punching through environmental conditions and weather like fog or rain or snow, and it directly can measure velocity using Doppler. Lidar and radar are active sensors. That means they see just as well in pitch darkness or, for example, when driving into a blinding sunset. These different sensing modalities are not backups to each other. In the stack, each modality has an encoder, and the information from all of those sensors gets fused into a single view of the world around that is much more precise and generally vastly superior to what is gotten with any one sensor.
Here's a scene where a Waymo is driving in a dust storm in Phoenix. What is seen is what the scene looks like to a fairly advanced high-resolution and high-dynamic-range camera. It's very close to what a human would see in the same conditions, which is not much. On the right is what the lidar sees for the exact same frame. Much more clearly seen is that there's a pedestrian standing on the side of the road. If they were to step onto the road, that early detection can make a really big difference in how the situation plays out and the safety of everyone involved.
Another example: at night, driving along, and there are a couple of pedestrians who are about to jump onto the road over a concrete construction barrier. At the bottom is the camera view. Really can't see much. The lidar view at the top. Again, lidar versus camera.
Another example: a couple of dogs chasing a bull and a couple of kids chasing the dogs. Big difference. Here's what it looks like to the camera. Here's the lidar. The early detection of the kids is off to the side, and there are no headlights. There are no lamps there. It's complete darkness. It makes a big difference.
Or think about what happens when something physically obstructs the view of sensors. If redundancy in sensing is not had, a single leaf can land on the sensors and bring the robot to a full stop. Redundancy is needed. Redundancy does not necessarily mean multiple sensing modalities, but if redundancy is needed anyway, benefit might as well be taken from the complementary physics of the different sensing modalities in the nominal case. Here's a video of one of the cars that picked up a leaf, or actually a full branch of a tree, that the wipers were unable to shake. The car detected that, and because sensing redundancy is had, it was safely able to get back to the depot for proper cleaning.
Specifically when it comes to hardware, do not anchor to today's component prices. We're on the sixth generation of the Waymo driver, the Waymo hardware suite today. With every generation, the hardware not only delivered amazing capability, but we're able to drastically simplify and radically reduce the cost of the hardware as well. Betting the company, betting the approach on today's hardware prices is just betting the company on a number that has a fairly short shelf life and is going to expire. Hardware will change. Many components will get commoditized and drop in price. Design for that future and be ready to upgrade.
That brings us to the next lesson. Lesson number three. Technology moves incredibly fast, especially nowadays. Readiness is needed to ride those tech waves and do that repeatedly. When doing so, not only must wins in performance and wins in capability be thought about, but very mindful attention must be paid to unification and simplification.
Over the years, a number of major breakthroughs in technology have been seen. A lot of them around AI. With every wave of innovation, the Waymo driver has pretty much been rebuilt around that major wave of AI breakthroughs, and the state-of-the-art in those areas has often been pushed forward ourselves. Convnets were leveraged around 2013 for computer vision and perception. Then, when transformers came about around 2017, a big bet was made on them both for perception and for the task of behavior prediction and decision-making and planning. Turns out the task of driving is not that dissimilar from the task of modeling language. Because of the social aspects of driving, it's kind of having a conversation with other dynamic actors in the world, but doing that in the space, kind of body language of the agent, the car, as opposed to just the language of words. Sequences are operated in. Local continuity matters, but so does global context. Today, the latest in VLMs and frontier world models are being leveraged.
Using the latest tech for capability and performance wins is not easy, but it can be reasonably straightforward. Doing applied research in isolation or starting a tiger team to prototype some new technology is not the most difficult part. There are many companies, many teams that are excellent in this. The much harder muscle to build is to carry that bleeding-edge research into production and deploy it in a safety-critical environment without regressions and do it without breaking stride on the scaling of the product. Adding capability again is not the hardest part, but adding capability while at the same time reducing fragmentation and reducing complexity is really important. And finally, the hard muscle to build as a company is to be able to do that repeatedly through multiple waves of technical innovation and technical breakthroughs.
On this front, two bits of advice. The first one: when new technology shows up, it can be very exciting, very tempting to kick off a new effort, a tiger team to pursue it. That's great. That should absolutely be done. However, when doing so, it's very important that what would be done after under a success scenario is considered. Let's say that effort succeeds. There should be very clear vision on what the path of that new innovation is for the company, for the entire product, for the entire system. Often times, a failure mode is seen where a project, a very difficult technical project, succeeds, and then there's a dead end. That can be very wasteful. That can be completely deflating.
The second bit of advice: when pursuing new tech, don't just ask what this new tech gives in terms of capability and performance. Also ask: has it simplified the stack and has it led to fragmentation or unification. Set the launch bar to demand both breakthrough performance and at the same time radical simplification and unification.
This exact philosophy and this muscle that has been built at Waymo over the years is what produced the latest core technology. The heart of it is the Waymo foundation model. The Waymo foundation model is a multimodal world action language model. That's kind of a mouthful. Let's unpack the ingredients. It's a multimodal model because it is able to process these multimodal sensor inputs: cameras, lidars, and radar. It's a world model because it inherently understands how the world works, the physics, the dynamics, as well as the social and semantic aspect of it. It's an action model because we're not just passively observing how the world evolves. We're an active participant. So the model needs to understand the effects of the actions of the agent on the world and be able to tell the good ones from the bad ones. And finally, it's aligned with language, and that allows unlocking general world knowledge from visual language models. That's incredibly useful in the long tail of rare semantic situations.
More specifically, this is what the architecture looks like. It's kind of the typical encoder-decoder architecture. The encoder part takes in the multimodal sensing and compresses it or encodes it into an efficient representation that retains all of the relevant data, all of the relevant information for the generative part or the decoder. It's an end-to-end model, which has a couple of nice properties. It allows effective back-propagation of the gradient from the task that is actually cared about all the way to the early layers of the model, and it allows the encoder to reach and learn the right rich representations for what the generative part needs to solve the task. It uses a system one, system two, think fast, think slow architecture, and it leverages the general world knowledge of VLMs for efficient learning of semantic tasks.
Let's dive deeper. First, the think-fast path. That part fuses the raw data from the cameras, the lidars, the radars, and that allows for split-second, safety-critical decisions. Think of it as driving instincts. This is what allows the car to brake instantly if, let's say, a pedestrian runs into the road or a cyclist that's nearby swerves into the path. This is like the lizard brain of the agent that deals with a lot of geometric tasks and can react in milliseconds.
Second is the slow path. That's the part that's responsible for the more complex semantic and scene-level understanding type tasks. These sorts of tasks don't typically change in milliseconds. So there, a bit more latency can be afforded, and that can be traded off for higher capability and higher levels of reasoning. For example, if the Waymo driver encounters a situation when there's a vehicle, let's say it's on fire on the side of the road, the fast path might just see it as a generic obstacle and reason that the path ahead is clear. This is where the slow path comes in. That path can use deep semantic reasoning to understand the semantics of that object, the car being on fire, and the broader scene context. That allows the driver to decide to take a very different action or a different route entirely, even if geometrically the path ahead is very clear.
And finally, there's the generate component. That's the decoder. That's the component that understands and can produce behavior. It understands how other actors behave, and it allows
The generate component functions as the decoder that understands and produces behavior by analyzing how other actors behave, enabling predictions and planning of driving decisions. The Waymo foundation model powers the Waymo driver across different hardware generations and vehicle platforms, including fifth and sixth generation systems, JLR IPA, Zeekr, and Hyundai Ioniq vehicles. This foundation will power future products and commercial applications including trucking and personally owned vehicles.
By focusing on the high-capacity foundation of the onboard model, complexity moves upstream to the large shared foundation. This approach makes the specialization layer running on vehicles lightweight, which speeds up the development process. The critical capability is not just leveraging current technology but building the muscle to repeatedly ride technology waves, incorporating innovation into production without regression, maintaining deployment and scaling momentum, and avoiding drowning in complexity.
The AI community recognizes that general methods leveraging massive compute and massive data consistently outperform methods relying on handcrafted engineered human knowledge. This principle, known as the bitter lesson formulated by Richard Sutton in 2019, has proven true across every wave of technical breakthroughs. Methods that scale best with compute and data consistently win out.
Betting on high-capacity models with data and compute produces better scaling laws, then distilling into smaller, more efficient models running on agents in real time outperforms focusing directly on smaller models.
The use of structure in models presents a critical nuance. Structure that fights scale always loses, while structure that channels scale always wins. This principle directly applies to end-to-end models, which have valuable properties including proper gradient backpropagation from final tasks through the model, allowing the API between encoder and decoder to learn rich representations. These models are easiest to build and train, enabling rapid progress through imitation learning on the initial steep part of the curve.
However, achieving superhuman performance levels in fully autonomous agents in safety-critical environments requires more than basic vanilla end-to-end approaches. The key question becomes whether structure boosts scale or fights it, whether it limits and constrains solution space or helps scale without loss of generality.
Consider building a robot to play Go in the physical world with a camera observing the board and an actuator moving pieces. An end-to-end system going directly from pixels to actuation could be trained on videos of humans playing, representing an interesting research exercise. However, building the world's best playing Go robot requires recognizing the simple intermediate representation that captures the complete game state: the 19 by 19 board, providing fully observable and complete state of the world for the game-playing portion. This structure doesn't limit the model or constrain solution space but provides a helpful scaling mechanism.
Non-trivial physical world deployments lack such simple, clean engineered representations, which is precisely why end-to-end systems and learned representations are necessary. However, physical world structure does exist through laws of physics, rules of the road, and objects behaving in reasonably predictable ways. This structure can be used alongside learned representations to boost performance, simplify validation, and achieve better scaling laws.
Waymo pursues structure augmented end-to-end, going beyond basic vanilla end-to-end by augmenting learned embeddings with materialized structure representations. This approach provides three key advantages: validation at inference time through powerful correctness and safety validation layers running in real time on deployed vehicles; efficiency gains in large-scale training and evaluation of the generative decoder component; and strong verifiable feedback signals for evaluation and training recipes supporting reinforcement learning.
Building and safely deploying agents in the physical world requires good large-scale realistic high-fidelity simulators. Training and evaluation can occur through open loop approaches, passively observing input to output pairs for evaluation and imitation learning training, or closed loop approaches where actions are taken, their effects on the world observed, sensor views updated, and sequences of actions and world evolutions evaluated and trained upon.
The ability to take actions and evaluate counterfactuals is vital for building and deploying safety-critical agents in the physical world. Real simulators represent big AI models themselves, with the problem of building realistic simulators being as hard as building the agent itself. The AI behind simulators must understand how the world works, including physics, semantics, traffic, and weather. Simulator quality must be high enough to train and evaluate with high confidence agents operating in safety-critical environments, requiring highly accurate generative world models.
Waymo has built behavioral world models for years, predating the popularization of the term world models. In the era of end-to-end models, sensing realism is also required alongside behavioral realism. Building end-to-end models has been relatively easy, but closed-loop evaluation remains the hard part. Structure augmented representations enable leveraging structure in simulation, with behavior world models operating in structured intermediate representation space and tightly coupled sensor world models producing realistic sensor simulations.
Leveraging Google DeepMind's Gen3 work enables production of controllable and highly realistic scenarios in both behavioral and sensing aspects. This allows training and evaluation not just in previously encountered situations but in purely synthetic rare scenarios never seen in the real world. Examples include simulating stopped cars in freeway lanes, planes landing on freeways, elephants walking through intersections, snow on the Golden Gate Bridge, and dinosaurs walking around.
Complex problems require building entire ecosystems with flywheels powering them. Three AI systems must be built: the agent (the driver), the simulator (virtual playground for learning), and the critic (rigorously evaluating and judging agent performance with improvement guidance). All three share fundamental reasoning and generative capabilities based on the same foundation world model.
These three pillars create a powerful flywheel: real-world agent deployment generates data, which grounds the simulator making it more realistic. The simulator generates harder edge cases for the critic to score and the agent to learn from. The agent gets smarter, deploys in the physical world, generates more data, and accelerates progress. Flywheels require guidance through metrics to move in desired directions.
Models represent table stakes, but evaluation and metrics constitute the most important strategic moat. Building evaluation before technology and evaluation and metrics before products is essential. Without quantitatively defining what good enough means, products aren't being built but rather demos are being iterated upon. Best model architectures are well-known with new ideas proliferating quickly. Data is important, but without good metrics, development proceeds blindly without leveraging best data or evaluating ROI on changes.
Model-level evaluation is insufficient for physical AI agents. Evaluation and validation must extend much deeper and broader, covering every system component from physical to behavioral layers running onboard in the physical world, plus offboard components and operational processes. Waymo's safety and readiness framework, built and refined over years, guides development, deployment, and scaling as one of their most important assets.
In the physical world, trust is everything, and evaluation and metrics represent how trust is earned. Trust isn't won through clever technical solutions, state-of-the-art architectures, or flashy demos, but gradually day by day through relentless field proof that systems are safe and functional. Publishing safety data and ongoing safety research openly demonstrates this commitment, as earned trust becomes the ultimate business advantage. Models can be leaked and algorithms replicated, but hundreds of millions of miles of fully autonomous operations backed by evidence-grade evaluation and publicly audited proof are much more difficult to replicate.
The complete playbook compounds multiple effects: setting appropriate bars and selecting correct technology and technical approaches to avoid local minima; intentional structure use to boost scaling and ability to ride technical innovation waves to reach appropriate performance levels; and AI ecosystems with agents, simulators, and critics guided by evaluation and metrics to build powerful flywheels. This refined playbook enables strongly superhuman safety performance.
Latest safety data based on over 220 million fully autonomous miles shows the Waymo driver is approximately 17 times better than human drivers for crashes causing serious injury in operating areas. This performance matters because somewhere in the world every 26 seconds someone loses their life in a crash. At current scale, Waymo prevents a serious injury every eight days, meaning someone's loved one walks through their front door safe and unharmed at day's end.
These represent early safety benefits of AI in the physical world that will continue growing. Physical AI currently stands where digital AI was a few years ago, with all necessary ingredients present: generative world models, architectures, affordable compute and sensing, proven scaling laws, and real products operating at scale. The last decade of AI occurred in the digital world, while the next decade will happen in the physical world.
For those building in this space, success requires remembering who is being built for, the mission, and customers. Otherwise, technology remains just a science project. At the end of the day, as exciting and exhilarating as the technology is, nothing beats the joy of making a difference in people's lives.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
