Inference, Diffusion, World Models, and More | YC Paper Club

Name: Inference, Diffusion, World Models, and More | YC Paper Club
Uploaded: 2026-05-28T20:37:13.000Z
Duration: 1 h 7 min 18 s
Channel: Y Combinator

Y CombinatorMay 28, 20261h 7m

Topics50

YC Paper Club Launch0:07 First Paper: Speculative Speculative Decoding3:30 Fast Inference Demo6:31 Vanilla Speculative Decoding7:32 Speculation as Currency Exchange10:33 Speculative Speculative Decoding (SSD)11:32 Implementation Details and Trade-offs15:00 Performance Results16:32 Second Paper Introduction17:32 Diffusion Model Predictive Control18:33 Core MPC Components19:32 Diffusion MPC Motivation20:01 Taxonomy of Related Approaches20:33 Diffusion-Based Agents Landscape22:31 Algorithm Overview24:30 Diffusion-Based Multi-Step Planning in DMPC25:17 Comparison to Prior Work26:32 Performance and Adaptability Results27:02 Component Ablations29:02 Transition to World Models Presentation29:50 World Models: Definition and Motivation30:30 Observations, Actions, and Prediction in Real Systems32:31 Model-Free vs. Model-Based Policies33:30 Toy Environment Demonstration35:31 Challenges in Training World Models36:02 JEPA and the Lay World Model Approach37:31 Capabilities: Open-Loop Prediction and Model Predictive Control40:00 Surprise Quantification and Uncertainty42:01 Discussion Points42:31 Generalization in Deep Learning43:31 Explaining Overparameterization44:31 Flat Minima and Generalization Bounds47:38 Benign Overfitting48:00 Flexible vs. Biased Hypothesis Spaces49:00 Deep Learning Mysteries Explained by Existing Theory49:30 Intelligence per Watt and Intelligence per Sample50:30 Pre-training Progress and Scaling Laws51:30 The Data-Constrained Regime53:01 Core Contribution: Scaling Laws for Data-Constrained Pre-training54:30 Experimental Setup55:00 Standard Recipe and Overfitting56:01 Aggressive Regularization Baseline56:30 Ensembling Results58:01 Joint Scaling Recipe59:00 Infinite Compute Performance1:00:01 Data Scaling Laws1:00:30 Distillation for Practical Inference1:02:31 Self-Distillation Results1:03:30 Downstream Benchmarks and Other Settings1:04:31 Key Takeaways1:05:30

In a Nutshell

Three papers showed concrete algorithmic wins for scaling intelligence under real constraints. Speculative speculative decoding parallelizes drafting and verification on separate hardware to reach 300 tokens/sec on Llama-70B, turning extra FLOPs into lower latency and higher throughput. Diffusion MPC factorizes action proposals from dynamics models, enabling runtime adaptation to new rewards and dynamics where joint models fail, while a JEPA-style world model delivers fast latent-space planning and built-in uncertainty estimates using only 15M parameters. In the data-constrained regime, aggressive regularization plus ensembling yields a 5× data-efficiency gain over standard pre-training; distillation recovers most of the gain at inference time.

AI-Generated Notes

These notes were generated by AI and may contain inaccuracies.

YC Paper Club is the first of its kind, created as a community for founders and researchers. Over 1,000 people applied and roughly 100 were selected. The event is held at Pioneer, a location chosen to bring together AI talent from the Bay Area, including people at Anthropic, OpenAI, Cursor, Google DeepMind, Tesla, xAI, and Thinking Machines who may not regularly travel to San Francisco for YC events.

The speaker references their Winter 2016 YC batch experience, noting that 140 companies went through that batch and 10-15 became unicorns, including WPY, Astronis, and Deep Graham. During that period, early OpenAI discussions occurred at the same location.

Tanishk, a Stanford grad student, presents work done with Triau and Aar May. The talk focuses on inference, arguing that inference should be viewed as a capability rather than merely a cost or convenience factor. The core claim is that if a method's performance scales with thinking time, then tokens-per-second directly determines peak intelligence deliverable.

A live comparison shows three approaches on a code prompt from VLM: standard autoregressive decoding, speculative decoding, and a custom inference engine implementing a new algorithm. The custom engine achieves significantly higher speed, demonstrating that algorithmic improvements matter more than systems optimizations alone.

The setup uses a small draft model (Tiny Llama) and a large target model (Big Llama). The draft model generates tokens autoregressively through multiple forward passes. The target model verifies these guesses with a single forward pass by checking token probabilities. Verification is easier than generation because transformers compute probabilities for many tokens in parallel.

Accepted tokens are those the target model would plausibly generate. Rejected tokens trigger a rejection point, after which a bonus token is sampled for free using the already-computed probabilities at the rejection position.

Speculative decoding exchanges FLOPs for latency by precomputing predictions. Some predictions may be incorrect, but correct ones enable fast-forwarding. A key limitation is the sequential dependency: drafting in round t must complete before verification, and drafting in round t+1 requires the verification outcome as prefix.

Sign in to read the full notes

Get access to AI-generated notes, topic timestamps, and more.

Inference, Diffusion, World Models, and More | YC Paper Club

YC Paper Club Launch00:07

First Paper: Speculative Speculative Decoding03:30

Fast Inference Demo06:31

Vanilla Speculative Decoding07:32

Sign in to read the full notes