Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon
In a Nutshell
Diffusion models will win AI inference because they generate tokens in parallel rather than sequentially, mapping far better to GPU workloads than autoregressive models. This yields 10x faster inference at equivalent quality, which becomes critical for scaling test-time compute and RL post-training. Inception has already deployed production diffusion-based LLMs that match speed-optimized frontier models like Haiku while running on standard Nvidia GPUs.
These notes were generated by AI and may contain inaccuracies.
Stefano Ermon is a longtime Stanford professor and co-founder and CEO of Inception. He has an extraordinarily broad body of work around generative modeling, and is especially well known as one of the fathers of diffusion. The company is challenging the large labs, with speed and efficiency being the name of the game in AI over the next few years.
Ermon has been doing research in generative models for his entire career. He started at Stanford in 2014 as an assistant professor working on building generative models. Back then the research area was not particularly hot. The models were not working well. Researchers were building little generative models over MNIST, and it was a big success if you could generate grainy images of digits. It was even hard to publish papers on that topic, and you had to justify training a generative model as a way to learn features from unlabeled data that could help with supervised learning.
Things took off and Ermon was at the right place at the right time working on the right thing. He has been doing research in that space since the beginning.
Ermon always felt that building a generative model was the right way to make sure you understand the structure in the data. He was thinking from a world models perspective, working a lot on images and thinking about having a world model where you can imagine what's going to happen if you stand up and walk out the door. Having this kind of model of the world requires some generative capabilities.
He felt it was the right direction to work on, a very hard problem that would keep him busy for his whole career. He was very wrong about the timeline as things evolved much faster than expected.
Ermon was working on generative models of images initially, working on autoregressive models which were very slow and blurry, then VAEs, and then GANs took over. Back then researchers were unhappy with the state of generative models for images. GANs worked but they were very unstable to train and very hard to reproduce results.
They were trying to see if there was a way to build something that is as good as GANs but more principled. They started working on score-based generative models, which eventually became diffusion models, back in 2019 with his PhD student.
They came up with the idea of training a neural network to denoise images. If you can denoise an image, you are understanding enough about the structure of the image that it should be possible to build a generative procedure based on these denoisers.
This became the underlying technology of modern diffusion models where instead of generating images left to right one pixel at a time, you start from pure noise and then gradually refine the object until you get a clean picture at the end.
This started back in 2019 in his lab and took over the space. Even today the best models for image generation, video generation, music to some extent, and a lot of the protein work are based on diffusion. His group has worked a lot on various kinds of diffusion models, techniques for accelerating them to generate samples very quickly, and improving the quality of these models.
Since they were able to get diffusion models to work on images, Ermon started thinking about how to get diffusion models to work on text or code generation. The question was whether discrete objects could move beyond autoregressive models to something more parallel with built-in error correction.
He has been doing research at Stanford on getting diffusion models to work on text and code generation. They had a breakthrough in 2024, publishing a paper showing that for the first time it was possible to match the quality of an autoregressive model at the GPT-2 scale.
They trained a transformer model as a diffusion model on the same data and matched the quality with the same perplexity. The diffusion model was significantly faster because it outputs many tokens at the same time. They were able to generate text 10x faster compared to the autoregressive model.
This felt very exciting, and Ermon wanted to see what happens if you scale up and train bigger models. He started Inception to scale up the technology and build commercial scale diffusion-based language models.
Diffusion models are increasingly a dominant form of generated short form video and a dominant form of entertainment in other parts of the world. The quality possible today is unbelievable.
It is not intuitive that diffusion would work for other fields or that this is an interesting competitive direction to the current full transformer focused AGI labs.
There are two main paradigms for building generative models. Autoregressive models predict the next token or next pixel and generate left to right one token at a time. Diffusion is a coarse-to-fine generation with iterative denoising.
Diffusion dominates continuous modalities. Discrete modalities like text and code are where primarily all the big labs are betting on autoregressive models. As we move towards more multimodal models, nobody knows what architecture will win.
Inception is betting on diffusion models because what matters eventually will be inference time scaling. There are fundamental reasons why diffusion models are better than autoregressive models at inference time.
There was an inflection point in 2017 when people switched from RNNs to transformers. RNNs had to process tokens sequentially one at a time and training was very slow. Transformers allow processing many tokens at the same time in parallel, scaling better for training.
Autoregressive models are still sequential at inference. The computation is one left to right one token at a time. You cannot generate the 10th token until you've generated everything that comes before it.
That kind of workload does not map well to GPUs. It is extremely memory bound. You spend most of your time moving around weights across the memory hierarchy and doing very little arithmetic. This is a fundamental problem of autoregressive models.
A diffusion model is built to have at inference time a workload where you process many tokens at the same time. The workload at inference time in a diffusion model is very similar to the workload for training where you process many tokens at the same time in parallel.
It is built to have an inference workload that maps really well to what GPUs do well. The bet is on building the architecture and models that will scale best at inference time because economics are dominated by intelligence per watt and intelligence per dollar.
A lot of advances with reasoning models come from scaling test time compute. Being able to scale better along that axis will matter. For RL post-training, a lot of the bottleneck is generating rollouts, letting the model explore, scoring the trajectories, and improving the model.
Inference is a key bottleneck for post-training. If you have a model that scales better at inference time, you automatically get better scaling during post-training.
The diffusion-based LLM was chosen because it is inherently more parallel. The bitter lesson is that the more parallel solution is the one that is eventually going to win.
There was a lot of research figuring out how to apply technology that was inherently tied to continuous structure in the data. Diffusion models learn how to denoise images, which makes sense for continuous data because you can interpolate between two pixel colors.
For discrete data like words, there is not necessarily something in between them. It required a lot of R&D and new science to extend those ideas to discrete spaces.
Diffusion-based LLMs are comparable in quality with speed-optimized models from Frontier Labs. The Mercury models are on par with the Haiku models, Flash models, Mini and Nano models from OpenAI on benchmarks while being significantly faster.
They have crossed from pure research prototypes to things used in production today. They are serving these models in production and built their own serving engine since you cannot run diffusion-based LLMs on vLLM or SGLang.
Inception is about two years old with around 50 people. They spend a lot of time on R&D figuring out the right way to train these models and how to accelerate inference.
It is not obvious how to accelerate inference for diffusion models. There are a lot of different possibilities for trading compute for quality at inference time. There are techniques like distillation or fancy differential equation solving techniques that allow sampling very quickly from these models.
There is research on training and inference, and engineering work on data mixes, RL post-training infrastructure, and building recipes that work for this new model architecture.
They leverage existing things as much as possible. It is still a transformer-based model so they do not have to throw away work on good architectures. They still use attention and public datasets, evals, and benchmarks.
As a startup they try to be scrappy and use existing things as much as possible, focusing on aspects where they can have the highest impact and be most differentiated. Right now that is speed.
In the future, it is possible that a diffusion-based language model will be significantly more intelligent than an autoregressive one. Nobody knows, which makes this exciting because they are developing powerful AI systems in a very fresh and new space.
Efficiency will be very important. Nobody really knows how the AI factory is going to work. Being able to play in that space and think about alternative ways of creating intelligence is exciting.
In an increasingly compute and supply constrained environment, efficiency focus is going to go dramatically up. Many people argue that one of the most important decisions for a company in AI space is how to use power between training and the most valuable use cases.
Speed wins in applications where latency matters. Everyone cares about speed in the sense that if you can give the same quality but faster, people will always pick the faster solution.
People are willing to pay more to get access to faster models. Once you get used to a fast model, it is hard to go back. It is kind of like broadband where people cannot go back once they get the fast model.
OpenCall is building voice agents in the voice space. Speed matters a lot because the pipeline includes an ASR model, an LLM doing tool calls and figuring out what to say next, typically a reasoning LLM, and then a text-to-speech component.
They were previously using Cerebras to serve their LLMs to get the speed they need. They switched to diffusion-based LLMs because they can get the same speed as running an autoregressive model on custom hardware.
With a diffusion-based LLM built to be parallel and accelerated at the software level, they can get the same speed on Nvidia GPUs, which means more availability since GPUs are more available than custom chips, plus lower cost and higher quality.
Hardware is one way to accelerate things. Software might be better if existing hardware can be used. They are complementary. The gains from software are multiplicative with gains from hardware.
Someday people might develop hardware that fits even better with the models being built.
It is hard to invest in new architectures because advances will be absorbed by large players with resources to scale compute. Initially the mode is IP and trade secrets and the ideas researchers have to build and improve these models.
As the company matures, having real customers and getting feedback means developing components like a serving engine. If you do not have the serving engine, you cannot serve these models in production.
By deploying something end to end from the beginning, they are learning about how to serve these models and building software needed to run them, which becomes IP.
Working with real customers and getting feedback on the models helps figure out what works and what does not. They collect data from customers and create evals based on what customers are seeing. That becomes part of the technical moat because those things are harder to replicate.
When training a generative model, whether autoregressive or diffusion, you are trying to identify structure in the data by building a compression scheme. The more you can compress the data, the more structure and patterns you identify.
By predicting the next word, you are learning something about the structure of the data. This is the same whether using a diffusion model or autoregressive model. Both methods try to learn a compression scheme.
In the 2024 paper showing parity with autoregressive models, the metric used was perplexity, which measures how much structure has been identified in the data. They achieved the same amount of structure identification at the GPT-2 scale as an autoregressive model.
There is noise in everything. The numbers do not lie. To the extent that perplexity can be driven down, it means a compression scheme can be built and that level of compression achieved. The structure must be there and the model must have been able to identify it.
The structure must be present and the model must have uncovered it. The question becomes one of inductive bias, such as whether a transformer is a better way of identifying patterns compared to alternative approaches. Whether next token prediction is the right modeling framework or whether denoising is more effective remains an empirical question where current tools are insufficient to fully understand the dynamics.
In voice customer applications, some AI product builders have identified benefits of having an LLM in the middle of the voice pipeline because alignment and controllability are better understood. For diffusion-based models, this alignment process requires a different approach.
The key value proposition centers on how much value comes from the harness and ensuring models do the right thing. Diffusion-based LLMs maintain backward compatibility with existing systems. The API remains identical, staying OpenAI compatible with text input and text output. The trained models excel at following instructions, outputting structured formats like JSONs, and handling these requirements effectively.
Performance proved good enough and actually exceeded the models previously used. Customers can maintain their existing level of service using Mercury. The interfaces remain the same, allowing use of existing stacks without modification.
Diffusion models offer easier control compared to autoregressive models. Autoregressive models require waiting until full object generation to evaluate constraint satisfaction, alignment, or other objective functions. For example, generating a molecule and evaluating solubility requires the complete object before scoring with a reward function.
Diffusion models enable coarse-to-fine generation, allowing progressive evaluation from the beginning. Assessment of whether an object matches desired characteristics can occur throughout the process, enabling steering of generation according to external reward functions or constraint sets. Academic literature provides evidence that diffusion models are easier to control with steering methods unavailable to autoregressive models.
Different interfaces may become available that aren't possible with autoregressive models. The right product experience around new capabilities not provided by autoregressive models requires ongoing consideration and development.
Beyond speed advantages, which were easier to test and measure initially, emergent capabilities remain unknown. As training methods improve, new capabilities will likely surface. Academic literature indicates diffusion-based models demonstrate greater data efficiency than autoregressive models.
Training diffusion models involves learning by denoising, where noise is added to images followed by learning noise removal. This process effectively performs data augmentation since each image generates multiple noisy views. This increased data efficiency could prove valuable in data-constrained tasks where data becomes the bottleneck.
Current frontier-level intelligence requirements limit diffusion model applicability. Estimates suggest 20-30% of workloads prioritize latency, making them addressable by models meeting specific latency budgets while delivering highest possible quality. OpenRouter provides taxonomy of tasks including research, conversational, coding, software engineering, and log processing.
Different technology stacks present challenges, requiring in-house development of serving engines, kernels, and related components. The ecosystem lacks mature open source alternatives, with existing open source models performing inadequately. This creates deployment difficulties and customer adoption barriers due to unfamiliarity.
New architectures face increased challenges with post-training processes including SFT, RLHF, and RL implementations. Building proprietary stacks creates intellectual property value, influencing decisions against full open sourcing. Trade-offs include reduced community contribution opportunities, harder adoption, and more difficult on-premise deployments.
Training details including model sizes and compute requirements remain trade secrets. The organization continues pushing frontier capabilities. At 50 people, the company attracts researchers and engineers seeking ownership opportunities in a relatively small field with significant invention potential. The green field nature and open-ended problems appeal to those preferring less constrained environments.
Access to frontier models accelerates iteration speed and idea testing. However, human ingenuity remains essential for developing correct ideas, pruning possibility spaces, and identifying promising directions. Current limitations suggest the field has not reached full recursive self-improvement capability.
With a small but highly talented team working intensively alongside AI agents for increased productivity, compute rather than personnel represents the primary bottleneck. Organization includes a product team handling platform development and customer success, a serving team managing current model versions, and a research team building next versions encompassing training, RL, and inference.
Confidence in research directions developed through collection of results including early diffusion work that beat GANs at academic scales while offering greater stability. Flash Attention emerged from academic work with eventual industry impact. DPO originated as a rotation project, providing alignment algorithms used across LLMs and diffusion models through clever mathematical insights about problem structure.
Academia enables contrarian bets despite resource constraints. Access to talented students willing to pursue novel approaches contributes to impactful idea generation. Many important AI concepts originated or have roots in academic environments where risk-taking faces fewer barriers.
Keep No Priors: AI, Machine Learning, Tech, & Startups in your library
Save the videos and channels worth coming back to, and find them again in one place.
