The ML Technique Every Founder Should Know
These notes were generated by AI and may contain inaccuracies.
Francois Shaard, YC visiting partner, discusses diffusion, a fundamental machine learning framework. Background: computer vision since 2012 in Fei-Fei Li's lab, ran Focal Systems for a decade, now at Stanford finishing PhD on diffusion-based world models for AGI. Covers what diffusion is, its evolution over the past decade, and current uses.
Diffusion learns any p(data) for any domain given data. All ML models learn data distributions, but diffusion excels at mapping high dimensions to high dimensions in low-data regimes. Example: 30 images of Gary (1000x10000x3 dimensional space) to map to 3 million dimensional space with only 30 training samples.
Take data sample (image of Anka or Gary), add noise repeatedly to create noised-up images. Easy to noise; hard to reverse to original data. Train model to reverse: noiser and denoiser, where denoiser is the trained model. Teach model with noised images to learn intermediate representations back to p(data).
Roots in 2015 Joshua Susskind paper on CIFAR-10 images. Sprawls beyond: DeepMind Nobel Prize for protein folding using this procedure. Diffusion policy for driving cars. Weather prediction. Image/video generation (Stable Diffusion and newer versions). Life sciences: DiffDock for small molecule binding to proteins; newest AlphaFold uses diffusion heavily.
2015 Joshua paper laid key components of modern diffusion. Innovations: noise schedules (Euler), loss functions (predict x_{t-1}, error added, velocity = error/time, flow matching = global velocity from start to end). All denoise but differ in objective (delta between steps or first step).
Hill-climbed on FID (Fréchet Inception Distance) metric. Predicting data hard; error easier; velocity easier; global error easiest. Ease for model to learn, plus simpler math/code (opposite most ML where complexity increases). Architectures evolved: U-Nets to diffusion transformers with cross-attention.
Downsampled Gary images to 64x64x3, augmented for more data. Noise schedule hardest part: linear interpolation unstable (small error early, large late). Need constant relative error per step; cumulative looks like pink curve (beta schedule). Beta = diffusion rate (beta_min to beta_max); alpha = 1-beta (signal retained); alpha_bar key weighting term (1-minus-sigmoid shape).
Training objective: minimize KL divergence between real and learned distributions. Results: generated images after 100 steps, FID=222 (high today; modern ~8-10). Code hairier than later versions.
From Meta (Yann LeCun? Litman): forget circuitous noising path; use straight-line global velocity from noise to data. Independent of time position. Code: ~10-15 lines.
- Sample data (Gary image), isotropic Gaussian noise, time t.
- x_t = t * data + (1-t) * noise.
- Velocity = noise - data (global, time-independent).
- Train: model predicts velocity on x_t; MSE loss (5-line loop).
Model-agnostic: RNN, U-Net, diffusion transformer, protein/LLM/text/image/weather/stock/robotics/trajectories/proteins/DNA. At inference: Euler step, iteratively denoise from noise using predicted velocity (many model calls, like Midjourney wait times). Limitation: can't exceed trained steps (e.g., 100 steps train/test match; doubling turns to white). Distillation tricks for fewer steps (train 10-step to mimic 100-step).
Yann LeCun lecture: flight didn't need flapping wings (Wright brothers needed two wings; squint at plane vs. bird). Many paths to flight/intelligence. LLMs: monolithic transformer stack, 3 training stages (pretrain, SFT, post-train), no further learning, one token at a time, no backward steps.
Brain: massive recursion, one learning procedure, two lobes with corpus callosum (back/forth), thinks in concepts (not one token), recursive improvement, dynamic emission of concepts to text/thought/revision.
Diffusion advantages mimicking brain:
- Leverages randomness (biology does; neurons log-normal, random spikes; diffusion noises data to learn).
- Emits concepts then decodes big chunks (vs. one token).
Common image generation model widely available last few years. Lesser-known: other ways diffusion used in
Diffusion is most widely accessible in mapping from very high-dimensional perceptual data to very high-dimensional action spaces or perceptual data. Examples include:
- Generating images: Midjourney, Sora, Veo, Flux, SD3.
- Generating videos: images stabilized together, VideoGen, image-to-video.
New applications include:
- Sentences: diffusion LLMs (continuous or discrete), prominent topic in Europe.
- Writing code.
- Creating proteins: DeepMind won the Nobel Prize.
- Robotic policies: Diffusion Policy, key for robotics working, e.g., Rose the robot.
- Weather forecasting: GenCast, most accurate system in the world.
- Failure sampling: Harrison's work on diffusion for sampling failures and bad events.
Products using AI beyond text-based chat often use diffusion, especially for images, videos, code, and life sciences.
Diffusion has dominated AI except two holdouts where state-of-the-art is not diffusion:
- AR LLMs, still outperforming.
- Gameplay like AlphaGo, where MCTS remains state-of-the-art. More research needed for diffusion in these areas.
For those training models: seriously look at diffusion regardless of application. Use it to get a latent space for further training. No machine learning application should ignore diffusion procedures as a fundamental piece of the training loop.
For those not training models: update your prior on how good these things are getting. Image generation improved thousand times in last 5 years from early Midjourney to Veo, Sora, Flux, SD3 through scaling (time, money, data). Apply this to proteins, DNA, robotics policies, self-driving cars.
Skate to where the puck's going to go.
Core diffusion procedure is getting simpler and working better. Bet on Rose the robot working in homes, protein folding improving, applications to DNA, metabolomics.
Founders are developing new models for robotics, text generation, video using diffusion, or building companies on top of these methods, creating a new wave of companies on either end.
It's going to redefine the entire economy.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
