Inference, Diffusion, World Models, and More | YC Paper Club
In a Nutshell
Three papers showed concrete algorithmic wins for scaling intelligence under real constraints. Speculative speculative decoding parallelizes drafting and verification on separate hardware to reach 300 tokens/sec on Llama-70B, turning extra FLOPs into lower latency and higher throughput. Diffusion MPC factorizes action proposals from dynamics models, enabling runtime adaptation to new rewards and dynamics where joint models fail, while a JEPA-style world model delivers fast latent-space planning and built-in uncertainty estimates using only 15M parameters. In the data-constrained regime, aggressive regularization plus ensembling yields a 5× data-efficiency gain over standard pre-training; distillation recovers most of the gain at inference time.
These notes were generated by AI and may contain inaccuracies.
YC Paper Club is the first of its kind, created as a community for founders and researchers. Over 1,000 people applied and roughly 100 were selected. The event is held at Pioneer, a location chosen to bring together AI talent from the Bay Area, including people at Anthropic, OpenAI, Cursor, Google DeepMind, Tesla, xAI, and Thinking Machines who may not regularly travel to San Francisco for YC events.
The speaker references their Winter 2016 YC batch experience, noting that 140 companies went through that batch and 10-15 became unicorns, including WPY, Astronis, and Deep Graham. During that period, early OpenAI discussions occurred at the same location.
Tanishk, a Stanford grad student, presents work done with Triau and Aar May. The talk focuses on inference, arguing that inference should be viewed as a capability rather than merely a cost or convenience factor. The core claim is that if a method's performance scales with thinking time, then tokens-per-second directly determines peak intelligence deliverable.
A live comparison shows three approaches on a code prompt from VLM: standard autoregressive decoding, speculative decoding, and a custom inference engine implementing a new algorithm. The custom engine achieves significantly higher speed, demonstrating that algorithmic improvements matter more than systems optimizations alone.
The setup uses a small draft model (Tiny Llama) and a large target model (Big Llama). The draft model generates tokens autoregressively through multiple forward passes. The target model verifies these guesses with a single forward pass by checking token probabilities. Verification is easier than generation because transformers compute probabilities for many tokens in parallel.
Accepted tokens are those the target model would plausibly generate. Rejected tokens trigger a rejection point, after which a bonus token is sampled for free using the already-computed probabilities at the rejection position.
Speculative decoding exchanges FLOPs for latency by precomputing predictions. Some predictions may be incorrect, but correct ones enable fast-forwarding. A key limitation is the sequential dependency: drafting in round t must complete before verification, and drafting in round t+1 requires the verification outcome as prefix.
SSD parallelizes drafting and verification so they occur simultaneously on separate hardware. The draft model sends tokens for verification while immediately anticipating likely verification outcomes and drafting subsequent rounds on those predicted prefixes. This hides drafting latency entirely when predictions are correct.
Verification outcomes consist of an accepted token count plus a bonus token. The bonus token is difficult to predict due to large vocabulary size, yet the method achieves 80-90% prediction accuracy using token distributions from the draft model. Alternative tokens that were not sampled during drafting serve as plausible bonus token candidates.
Cache miss handling requires careful strategies beyond simple fallback to vanilla speculation. As batch size increases, prediction failures become more common. Compute allocation across plausible prefix lengths is not uniform; clever allocation improves cache hit rate. Trade-offs exist between cache hit rate and drafting quality.
On Llama 70B using 4 H100s, SSD reaches 300 tokens per second. The method improves both latency and throughput, unlike standard speculative decoding which primarily benefits latency. The blue bars represent the custom inference engine, light blue shows baseline speculative decoding, red shows SGLang with speculative decoding, and dark blue shows SSD.
The next presentation covers diffusion model predictive control. The speaker mentions being scooped on a similar idea involving using video models for robotic control by running them at test time.
Stannis, a staff research scientist at Google DeepMind, presents early work on world modeling for robotics. The project predates his current focus on general-purpose policies built on video and world models.
Model predictive control (MPC), also called receding horizon control, uses a dynamics model (world model) and an action selector (planner) to solve tasks by maximizing an objective. Advantages include test-time adaptation to new reward functions, easier learning of dynamics models compared to policies, better performance than direct policies, and easy adaptation to new dynamics via the action-dynamics factorization.
An action proposal generates a sequence of actions. A dynamics model evolves those actions into future states. An objective function is optimized by a planner to select actions for execution.
Two practical challenges for MPC are accurate dynamics models to avoid compounding errors and sufficiently powerful planning algorithms. Diffusion MPC addresses both by using diffusion models to learn multi-step action proposals and multi-step dynamics models. This reduces compounding errors and allows a simple sampling-based planner to outperform prior approaches.
Approaches differ in how they construct joint state-action distributions. Factorized approaches separate policy (action prediction) from dynamics model (state prediction). The Dyna paradigm learns a model, generates data via imagination, and trains a policy. MPC uses a planner to select actions. Some methods build joint state-action models and perform MPC. Model-free methods learn policies directly.
Key trade-offs include runtime planning capability, adaptation to new rewards, adaptation to new dynamics, ability to leverage non-expert data, runtime speed, and single-step versus multi-step modeling.
Diffusion Policy conditions on observations to generate future actions and remains dominant for complex control but requires expert demonstrations, staying within behavior cloning. Diffuser jointly models states and actions, providing implicit world modeling and model-based planning. Decision Diffuser conditions on observation history to generate future observations and uses a separate inverse dynamics model for actions, enabling video-only data learning. Diffusion MPC proposes actions, evolves them with a dynamics model, and uses a planner for selection, supporting runtime adaptation to new rewards and dynamics.
Models are learned entirely from offline datasets. The approach learns a policy that, given current observation
The approach learns a policy that predicts actions from current observations and a dynamics model that evolves observations forward given actions to predict future states. At inference time, action proposals are sampled, scored, ranked, and the best action selected. The key difference from prior methods is the adoption of multi-step action proposals, which function similarly to a diffusion policy. Training on more diverse data provides greater coverage of the action space. A multi-step dynamics model enables long-horizon evolution with reduced compounding error. Diffusion models are leveraged for their strength in modeling multimodal data, which empirically allows simplification of the planning algorithm while still solving tasks effectively.
DMPC is contrasted with model-based offline control, offline planning, and the Diffuser approach, which learns a joint model and applies classifier-free guidance for planning.
DMPC achieves competitive results in fixed-reward, single-task setups, performing on par with prior state-of-the-art methods. More notably, it adapts to novel rewards at runtime. Models trained only on simple local motion tasks (run forward, jump) can exhibit new behaviors such as jumping when the reward function is changed at inference. DMPC also adapts to novel dynamics where joint-modeling approaches fail. This stems from the factorization of action proposal and dynamics model: the action proposal remains fixed while the dynamics model is adapted on play data collected in the changed environment, recovering substantial performance when dynamics shift (for example, a walker with a broken left ankle).
Ablation studies confirm that each DMPC component contributes to performance gains: diffusion action proposals improve results and simplify planning; multi-step diffusion action proposals further boost performance; and multi-step dynamics modeling also improves outcomes.
The following presentation covers the Lay World Model paper from Yann LeCun's group, presented by Isaac Ward.
World models learn the dynamics of the world by using large neural networks to predict how a system evolves over time given its inputs. Given a current state or scenario (denoted S) and an action (such as a robot movement or language command), the model predicts the resulting next scenario. This enables generating imagined outcomes, model-based control, and surprise quantification. The concept dates back at least to a 1990 paper by Richard S. Sutton presented at EURASOFT, which described a black-box model taking situation and action as input and outputting a prediction of the next situation.
In real-world systems, only observations (not true states) are available. For a quadrotor example, observations include kinematic state (position, velocity) plus forward-facing camera images. Actions are control inputs (for example, yaw). The model must predict the resulting high-dimensional sensor outputs such as images or LiDAR. Challenges include long action sequences and the risk that minima in the optimization landscape may not align with desired behavior.
Model-free policies map observations directly to optimal actions via a neural network without an explicit future representation; evidence suggests these networks may contain implicit, highly obfuscated world models in their weights. Model-based policies train an explicit world model and use it to predict action outcomes. Model-free approaches can be brittle out-of-distribution, while model-based methods allow quantification of modeling error, which is valuable for real-world deployment. Model-based methods require an additional mechanism to propose candidate actions for evaluation.
In the Push-T environment, an image of a blue ball agent must be pushed into a green slot. The observation comprises the image plus the 2D end-effector position; the 2D action specifies where to move the end-effector. A small model (~200k parameters) can be trained to predict future observations given action sequences, producing visually plausible rollouts despite minor wobble.
Training requires jointly learning a compact representation of high-dimensional inputs (images, LiDAR) and the dynamics that govern how actions change that representation. The optimization landscape contains many trivial solutions, such as collapse where every state is represented identically. Existing approaches avoid collapse through explicit heuristics that enforce desirable properties in latent space, foundational models (autoencoders, diffusion, or video models) with added action conditioning, or privileged data available only at training time.
Lay World Model is a Joint Embedding Predictive Architecture (JEPA) model. An image encoder maps observations to latent vectors. An action-conditioned predictor forecasts the next latent embedding (not the next raw image). A decoder can reconstruct images if needed, but most computation occurs in latent space. To prevent collapse, a SIG regularizer is added: across a batch, latent embeddings are required to follow a healthy Gaussian distribution. SIG stands for Sketching (one-dimensional passes over high-dimensional data), Isotropic (distribution looks the same in any direction), and Gaussian. One-dimensional slices through the latent space are checked for Gaussianity; satisfying this condition indicates a healthy, non-collapsed distribution. This adds a single regularization term rather than multiple heuristics or privileged data.
Open-loop prediction quality is strong: on Push-T and a 3D Push-Cube analog, imagined trajectories closely match real ones. For model predictive control, an initial observation and goal observation are encoded; actions are searched in latent space to move from start to goal latent state. The world model outperforms competitors on small 2D tasks; Dino World Model wins on 3D tasks due to its large foundational backbone. On the simple Two-Room environment, performance is lower because the method prioritizes high-dimensional healthy embeddings. The approach runs approximately 50 times faster than alternatives because all work occurs in latent space, requires no extra forward passes or duplicate model copies, fits on a single GPU card under 24 GB VRAM, and uses only 15 million parameters.
World models can quantify their own error. Perturbations such as changing object color or teleporting an object produce detectable spikes in model error, allowing agents to estimate prediction uncertainty. Model-free methods do not natively provide this capability.
Open questions include whether model-based or model-free policies will ultimately prove superior for intelligent agents, whether representation and dynamics learning should remain coupled or be separated, the value of bio-inspired or foundation-model-based approaches, and the most elegant methods for preventing representational collapse.
Andrew Gordon Wilson's paper argues that deep learning generalization is not mysterious and can be explained via classical theories, particularly PAC-Bayes. PAC-Bayes bounds test loss (generalization) using training loss plus a compression term. Prior applications produced loose, vacuous bounds when models were overparameterized because the compression term dominated; an alternative computation of the compression term yields tighter bounds.
Classical bias-variance tradeoff predicts that increasing parameters should cause overfitting, yet scaling laws show improved generalization. PAC-Bayes accounts for this: larger models reduce empirical risk (training loss) by fitting data better, and also find more compressible solutions. Work by Lotfi et al. shows a negative correlation between bits required to encode the training set and number of parameters. Additionally, increasing parameters exponentially expands the volume of flat minima in parameter space (which are more compressible) while the volume of sharp minima grows more slowly.
Andrew's work demonstrates that flat minima provide useful bounds on generalization even for models at billion-parameter scale. The compressibility view shows that the volume of sharp minima increases much less comparatively, making this property both interesting and practically useful for understanding why certain solutions generalize better.
Benign overfitting refers to the phenomenon where deep neural networks can fit totally random noise while simultaneously generalizing well on structured data. The core mystery is how an inductive bias can enable good generalization if the model can also fit completely random data.
Andrew's paper provides intuition through a regularized polynomial model. On random data, the model has enough parameters to fit the data completely. On structured data, regularization pushes the model toward using lower-order terms, enabling both flexibility and an inductive bias that supports generalization.
A flexible hypothesis space allows fitting observed data but leads to overfitting without bias toward one solution over another. An inductive bias solves the overfitting problem but prevents modeling all details of reality.
The middle ground involves an expressive hypothesis space combined with bias toward solutions that generalize. In the PAC-Bayes framework, this includes biasing toward more compressible models when possible.
Deep learning mysteries are consistent with and partially explained by existing theories including soft inductive biases and PAC-Bayes. The key insight is that neural networks are expressive models with soft inductive bias.
By the no free lunch theorem, the only way to achieve improvements in learning efficiency is through inductive biases. Finding the right inductive biases and optimizing for them represents a strong research direction given the massive sample efficiency gap between AI and humans.
Two major remaining problems in AI are intelligence per watt and intelligence per sample. Current AI systems are roughly an order of magnitude off on intelligence per watt and orders of magnitude off on intelligence per sample compared to humans.
In Chris Re's lab, research focuses on the question: given a fixed amount of data and infinite compute, how much generalization can actually be achieved? This paper addresses that question directly.
Over the past six to seven years, pre-training has improved model capabilities in surprising ways:
- 2020 with GPT-3: emergence of in-context learning
- 2022 with Anthropic's RLHF: advent of alignment
- 2024 with OpenAI o1 and DeepSeek R1: emergence of reasoning
Newer pre-training runs like Mythos and 5.5 continue to improve models. Pre-training is expensive, so research has focused on compute efficiency through scaling both parameters and data points, quantified by Chinchilla scaling laws.
Public projections indicate human-generated text on the internet grows by roughly 3% per year, while compute spent on pre-training grows by roughly 4-5x per year. This means compute per data point will increase by roughly 4x year-over-year.
The core question becomes how to approach pre-training when data-constrained but compute-unconstrained. This differs from the compute-efficient pre-training regime that has dominated modern AI development. It resembles classical statistics and older benchmarks like MNIST and Penn Treebank where data was implicitly constrained.
The paper brings modern scaling law methodology to the data-constrained problem. It proposes scaling recipes that monotonically decrease IID validation loss, following clean power law functional forms.
When power laws can be fit, the best possible loss of a recipe can be estimated from the asymptote, quantifying best possible performance under infinite compute. The goal is identifying algorithms that lower this compute asymptote.
The canonical setting uses 200 million tokens from DCLM (general web data) to simulate a data-constrained world. Models of increasing size are pre-trained using different recipes, with the y-axis measuring IID validation loss on DCLM. The objective is finding recipes that allow spending more compute on larger models while monotonically decreasing loss.
The standard approach epochs data while scaling model size, performing an exhaustive grid search over parameters until overfitting begins, then applying early stopping. The standard recipe shows that even with more compute, overparameterized models overfit more quickly and loss increases after a certain point.
The first baseline applies aggressive regularization by increasing weight decay. Weight decay, learning rate, and epoch count are optimally tuned for each parameter count. The resulting loss follows a clean power law with exponent of 1 on model parameters n, matching data constraint theory predictions.
Weight decays used are roughly 30 times larger than those for compute-optimal pre-training. The scaling law has an asymptote of 3.43, characterizing best possible performance of the regularized model under infinite compute. Baseline approaches that overfit more quickly lack measurable asymptotes.
Ensembling proves highly data-efficient in modern pre-training. Experiments with 300 million parameter models show that ensembling more members (for example, five members totaling 1.5 billion parameters) yields clean scaling laws with exponent 1 on ensemble members and an asymptote.
The ensemble asymptote is substantially lower than the regularized recipe asymptote, representing a true data efficiency win under infinite compute. In compute-matched comparisons, ensembles outperform the regularized recipe. Training an ensemble of small models is better than training one large model when data-constrained.
Benefits of regularization and ensembling can be composed. Regularization enables scaling model size while ensembling introduces a new compute axis through training more models.
The joint scaling recipe quantifies hypothetical performance from training an infinitely large ensemble of infinitely large models. This is computed via a double limit: first fitting scaling laws to ensembles of 150 million, 300 million parameter models, and so on, then fitting a second scaling law to those ensemble asymptotes. The first limit is over K (ensemble members) and the second is over n (model size).
Training infinitely large models and infinitely many ensembles yields substantial loss improvement. All experiments use the toy 200 million token data-constrained setup.
Data scaling laws repeat the same experiments at four different pre-training token counts up to 1.7 billion tokens. For each token count, the best possible performance of each recipe under infinite compute is quantified.
Red points represent actual models that overfit quickly. Purple and gold points represent single or double limits. These laws enable quantifying data efficiency numbers by projecting new recipe loss onto the standard recipe data scaling law, measuring effective extra tokens gained.
The joint scaling recipe provides roughly a 5x data efficiency win over the standard recipe. A five-ensemble of 1 billion parameter models yields roughly a 3.7x data efficiency win. Data scaling laws show similar exponents and asymptotes across recipes, suggesting the 5x win remains constant even at 10 trillion token scales if the laws extrapolate.
Distillation reduces inference compute requirements while preserving data efficiency gains. An eight-ensemble (roughly 2.4 billion total parameters) can be distilled into a single dense 300 million parameter model while retaining roughly 83% of the loss improvement.
Data efficiency does not require large inference compute if test-time compute is amortized during training.
Self-distillation starts with a 300 million parameter model and distills it into a fresh 300 million parameter model. This yields substantial loss improvement, even beating the regularized recipe asymptote. The result connects to ensembling, with prior work viewing self-distillation as implicitly training a two-ensemble.
All trends hold on downstream benchmarks despite optimizing only for IID validation loss. The standard recipe overfits, model scaling improves performance, ensembling performs better, and distillation retains substantial benefits.
The approach extends beyond pre-training to continued pre-training. A 3B model is continued-pre-trained on 4 billion math-related tokens from a 73 billion token corpus. Aggressive epoching and ensembling match full-corpus performance using only 4 billion tokens, representing roughly a 17x data efficiency win.
When data-constrained and compute-unconstrained, algorithmic choices matter substantially. Classical machine learning techniques including regularization, ensembling, and distillation should be revisited. The asymptote evaluation tool helps identify algorithms with lower compute asymptotes.
The ultimate goal is developing new ideas under infinite compute that improve data efficiency beyond existing methods. Follow-up work examines how synthetic data interacts with data efficiency.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
