State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
These notes were generated by AI and may contain inaccuracies.
Conversation about state-of-the-art in artificial intelligence, including technical breakthroughs over the past year and predictions for the upcoming year. Technical at times but accessible without dumbing down. Features Sebastian Raschka and Nathan Lambert, machine learning researchers, engineers, communicators, educators, writers, and X posters.
Sebastian Raschka authored Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). Best way to learn machine learning/computer science is to build it yourself from scratch.
Nathan Lambert is post-training lead at Allen Institute for AI, author of definitive book on Reinforcement Learning from Human Feedback (RLHF).
Both have great X accounts, Substacks, YouTube courses (Sebastian), podcast (Nathan).
DeepSeek moment: Early 2025, open-weight Chinese company DeepSeek released DeepSeek-R1, near state-of-the-art performance with allegedly much less compute, much cheaper. AI competition accelerated insanely on research and product levels.
Who's Winning: China or US Companies?
Sebastian: Winning is broad. DeepSeek winning hearts of open-weight model workers by sharing open models. Multiple timescales: today, next year, 10 years. No company has exclusive technology access in 2026 due to researchers changing jobs/labs. Differentiating factor: budget and hardware constraints, not proprietary ideas. No winner-takes-all.
Nathan: Labs put different energy into efforts. Hype over Anthropic's Claude 3.5 Opus insane—used it, built stuff, feels like a meme, organic hype. Earlier, Gemini 1.5 from Google had high marketing/wow factor, seemed like Google's moment to retake advantages, but hype faded despite being fantastic. Differentiation lower now.
Agrees ideas fluid, but bottlenecked by human effort and organization culture. Anthropic bets hard on code (Claude Code), least chaotic, advantage if sustained.
China: More labs than DeepSeek—kicked off movement like ChatGPT in US. Zhipu AI (GLM models), MiniMax, Kimi Moonshot shining recently. DeepSeek still strong but losing crown as preeminent open model maker. DeepSeek narrative: platform for more Chinese companies releasing strong frontier open-weight models.
Chinese open-weights risk US business models, but US has paying AI software users; historically, China/other places don't pay much for software.
Future of Chinese Open-Weight Models
Nathan: For a few years. No clear business model like US. Chinese companies realize top US tech/IT won't pay API subscriptions due to security. Open-weights influence huge growing US AI market. Government incentives for international influence. Expensive to build, expect consolidation, but 2026 has more open model builders than 2025, many in China.
Sebastian: DeepSeek slightly ahead; others use their ideas (e.g., Kimi same architecture). Leapfrogging: most recent model best. No clear winner.
Chinese incentives differ: DeepSeek secretive (built by hedge fund Highflyer Capital), MiniMax/Moonshot AI filed IPO paperwork, seek Western mindshare/outreach. DeepSeek open on technical reports.
Claude 3.5 Opus hype is X/Twitter echo chamber vs. actual usage. ChatGPT/Gemini focus broad user base for daily problems—gigantic.
Coding hype not representative of use.
Usage patterns: name recognition, brand, muscle memory flywheel.
Customization: ChatGPT memory feature blurs private/work boundaries. Companies may not allow; users want separate subscriptions—one clean for work (no personal images/hobby projects), one personal. Future: multiple subscriptions for use cases.
Nathan (2024 consumer chatbots): Bet on Gemini over ChatGPT risky—OpenAI incumbent with tech benefits. 2024 momentum Gemini (from low point post-Bard), credit for powering through chaos. Hard to bet against OpenAI: chaotic but lands things. Mixed on o1, but high-line feature (router) saved money—most users no longer charge full GPU costs.
Hard to separate liked model features vs. public differentiators.
Sebastian (2025): Gemini continues progress on ChatGPT. Google's scale at extreme scales matters. Google separates research/product better vs. OpenAI's chaotic startup culture chasing high-impact.
Anthropic success on software/enterprise. Google Cloud offerings compete with Azure/AWS.
Infrastructure: TPU advantage—insane NVIDIA margins; Google controls stack top-to-bottom, head start on data centers (high lead times, hard margins).
New paradigms from OpenAI research (e.g., Deep Research, Sora, o1 thinking models).
2025 about scale, low-hanging fruit optimization. Trade-off intelligence vs. speed (GPT-5 tried solving behind scenes). Public wants variety/toggle.
Sebastian usage: Quick questions/info → fast ChatGPT. Thorough checks (references, thoughts, formatting, figure numbers) → Pro mode (can wait). Auto mode good. Can't wait 10-30 min per query.
Nathan: Always thinking/Pro (e.g., GPT-4o thinking/Pro). Non-thinking higher errors, bad tone. Habit from o1-preview deep search. Runs 5 Pro queries simultaneously (papers, equations).
Sebastian anecdote: Needed fast Bash script for RL experiment before trip (unplugged home GPU accidentally, wife in car). Used non-thinking fastest model for chaining scripts, tee to log file.
Nathan: Uses Gemini for fast/simple/Google-able things, app improved. Claude Opus 3.5 (extended thinking) for code/philosophy—inference-time scaling makes marginally smarter, hedge on progress. Grok for real-time/AI Twitter. Grok-3 Super Heavy (pro variant) impressive but muscle memory to ChatGPT.
Sebastian: Grok-3 Heavy best for hardcore debugging.
Gemini interface better for him (needle in haystack). Falls in love with model for one feature/query, sticks until dumb error—threshold effect, switch (like text editors, OS, browsers: use until breaks, edge cases/extensions).
GPT-4o long context improved dramatically (30%→70% in minor update, algorithmic?). Hard to track.
None mentioned Chinese models in usage: discrepancy model vs. platform. Open models known for weights, not platform. Use OpenRouter/multi-model, Perplexity for DeepSeek. Pay for marginal intelligence gain—US models better outputs. Chinese served with fewer GPUs/replica (export controls?), slower, different errors. Spurs competition (free/lower costs, creativity). Fun but don't return.
Half-half Cursor and Claude—fundamentally different, both useful.
Sebastian: Codeium plugin for VS Code—convenient chat with repo access. Claude more agentic (touches more, whole projects). Prefers control, not fully taking over.
Nathan: Claude builds skill of programming with English—macro design/guiding vs. micromanaging diffs/code details (possible in Cursor). Claude Opus 3.5 better utilized there. Side-by-side: Claude/VS Code/Cursor same models—Claude way better.
Nathan: RLHF book available for preorder, full digital pre-print. Physical for fun/excellence.
Sebastian: Build a Large Language Model (From Scratch), Build a Reasoning Model (From Scratch). Building LLM from scratch fun, best to learn (code precise, works=correct; figures/concepts can mislead). Coding doesn't lie—math.
With math in books, mistakes can go unnoticed because you are not running the math while reading. With code, you can verify it. The "Build a Large Language Model (From Scratch)" book allows tuning out the internet to focus. Reading history books is less lonely and more fun. Programming with an LLM is genuinely more fun. Reading with an LLM is more fun, enriching the experience with context and high rate of "aha moments". Minimize distractions.
Use LLMs in multiple passes: first pass offline in focus mode, taking notes without immediately looking things up. Let things sink in; answers may come later in the chapter. Second pass with LLMs for structure. Recommend using LLMs when reading books, but not as the first step.
Opposite approach: use LLM at the beginning to lay out full context of the world stepping into, avoiding rabbit holes like Twitter, blogs, Reddit flame wars. LLM provides context on why it matters and big picture ideas; books are good at this but not always. ChatGPT app gives AI a focused home on the computer. Claude Code is engaging, warm, fun for building from scratch, websites, refreshing tooling, data analysis. Example: scraping Hugging Face for download numbers of every dataset and model over time; Claude handled it quickly, saving days of work.
Closed-weight models discussed; now open ones. Interesting open LLMs: DeepSeek, Kimi, MiniMax, Z.ai, Moonshot (Chinese); Mistral AI, Gemma, OLMo (AI2), NVIDIA Nemotron 340B, Qwen. OpenAI released first open model since GPT-2. GLM-4 is strong, does things others don't. Western fully open models: Allen Institute for AI's OLMo (releases data and code), Institute for Foundation Models, LM360's K2 models, Swiss Apertus consortium, Hugging Face SmolLM, NVIDIA Nemotron releasing data, Stanford Meerkat project (GitHub issues for new ideas in stable LM stack).
In 2024, list was smaller (mostly AI2); now more competition releasing everything. No Chinese analog. Chinese open models tend bigger, higher peak performance as MoEs. US smaller models like Gemma, Nemotron changing. Mistral Large 3 (giant MoE, similar to DeepSeek) in December. Startups like RCAI, Nemotron, NVIDIA tease MoE models >100B parameters, up to 400B in Q1 2026. Balance shifting; exciting to watch Chinese vs US open models.
Props for naming many; forgot LLaMA (RIP). Standouts: Qwen 2.5. Year bookended by DeepSeek V3 (Dec 2024) and R1 (Jan 2025), interesting architecture tweaks. Familiar good performance: Qwen 2.5, Jamba (first public/open-weight trained with tool use in mind, paradigm shift).
Tool use: LLM does web search, calls Python interpreter. Solves hallucinations by not memorizing but using tools (calculator for math, search for facts like 1998 Soccer World Cup winner France via Google/FIFA). ChatGPT/GPT-4o do this. Huge unlock, not fully utilized in open ecosystem. Trust issues: don't run on computer with tools access (could wipe drive); need containerization. Important for upcoming years.
Thank you for defining tool use. Similarly define MOEs (Mixture of Experts). Explosion of open models because releasing wants people to use it (priority), then transparency/trust. In China, global use; outside US many won't pay but have compute. Local data privacy. Run locally, not sending data to China/Silicon Valley. US startups host Chinese models, sell tokens. OpenAI GPU deprived; releases use user GPUs (Sam Altman: "We're releasing because we can use your GPUs, get distribution without cost").
For companies, open models allow customization, post-training on law/medical data. Chinese open-weights have unrestricted licenses vs Llama/Gemma strings (user limits, report finances to Meta if exceeded). No catch appeals. Ecosystem better downstream of open licenses. Example: Perplexity hosts Kimi-k2-thinking in US (sensitive to data location). Kimi-k2 good for creative writing, software.
Chronologically: DeepSeek R1 (Jan 2025, based on V3 Dec 2024). Start from GPT-2, add tweaks. DeepSeek: Mixture of Experts (not invented by them), Multi-head Latent Attention (tweak to attention for inference/KV cache efficiency, long context). Other tweaks: Grouped-query Attention, Sliding Window Attention (OLMo 2). Models surprisingly similar: different Transformer block repetitions, knobs tuned. Many Transformer implementations work. OLMo good for ablation studies (move normalization layers). Big ideas: MoE, multi-head latent attention, sliding window/GQA, end-of-year linear-scaling attention (Qwen2.5 gated delta net, inspired by state space models, cheaper attention).
Transformer from "Attention Is All You Need": encoder-decoder. GPT focused on decoder. Neural network with attention; predict one token at a time via embedding, Transformer blocks (attention + FFN), normalization. GPT-3 added MoE to FFN (expensive: 1000x1000=1M connections). MoE: expand to multiple (e.g. 256) FFNs (experts), router selects few per input (math vs translation, fuzzy). Packs more knowledge, selective activation (sparse vs dense: dense always uses one full FFN).
MoE adds complexity, harder to train (collapse risks); OLMo has dense/MoE versions. Few new ideas from GPT-2: Group Query Attention (Llama 3 tweak from multi-head), RMSNorm (vs LayerNorm), nonlinear activations (like sigmoid to ReLU). Still same architecture; convertible by tweaks. Book uses GPT-2 (124M params, simple); bonus: OLMo, Llama 3 from scratch, starting from GPT-2 lineage.
Rapid advancement but architectures unchanged. Gains in training stages: pre-training, mid-training, post-training (current focus). Pre-training scales with better data. Capability unlocks via post-training: ChatGPT/GPT-3 added supervised fine-tuning (SFT), reinforcement learning with human feedback (RLHF) (algorithmic, not architectural).
Systems changes: Nvidia FP8/FP4 (less memory/precision, faster training, more data/configs). Tokens/second/GPU metric (10K to 13K with FP8). Faster experimentation loop underpins gains. Train GPT-NeoX-20B faster wall-clock than original GPT-2. MoE FP4 for throughput. Enables speed, not new capabilities (coarser compute without degradation).
Alternatives: text diffusion (non-autoregressive Transformer), Mamba (state-space). Trade-offs; nothing replaces autoregressive Transformer for SOTA, but alternatives for cheaper end.
Technical scaling law: power law between (compute + data) and held-out next-token prediction accuracy (autoregressive). Predictable. Users get capabilities. Types: pre-training (model/dataset size), inference time scaling (OpenAI o1), RL scaling (o1: log x-axis, linear performance y-axis; trial/error duration). Traditional for pre-training; now three axes.
Letting the model generate more tokens on a specific problem. Bullish on these approaches, but low-hanging fruit mostly taken, especially in the last year on reinforcement learning with verifiable rewards (RLVR) and inference time scaling. This explains why models feel different: previously first token immediate, now seconds, minutes, or hours generating hidden thoughts before first word. Inference time scaling is a step function in abilities, enabling tool use and better software engineering, downstream of RLVR training letting models pick up skills easily.
Models during reasoning generate many tokens: try a tool, see response, try another API, check if solves problem. Models quickly learn this in training. Provides general foundation for using CLI commands in repo, handling Git, moving/organizing files, searching info—capabilities unimaginable a year ago. Transformed AI usage this year, unlocks massive value. Unclear what next step function will be; continual learning later.
Pre-training extremely expensive; scaling implies serving large model to users. GPT-4 and similar around 1 trillion parameters at biggest size; rumors they've gotten smaller with efficiency gains to reduce serving costs. Training cost low relative to serving hundreds of millions: DeepSeek ~$5M pre-training at cloud rates; OLMo section 2.4 paper details ~$2M to rent cluster including engineering, multiple seeds. Accessible: 1-10M dollars trainable by many.
Recurring serving costs billions: 1000 GPU rental ~$100k/day; companies have millions of GPUs. Financially worth scaling if better model for compelling tasks like Claude 3 Opus, GPT-4.5, Claude Code. ATOM project (American Truly Open Models) launched July: vibe-coded site with plots; recent refresh showed Claude 3 Opus crushed June/July issues—possibly bigger model, many factors, but progress continues.
Y-axis nuance: benchmark vs. experienced intelligence differs. Intuition: scaling pre-training compute makes models smarter? Held for 13 orders of magnitude; unlikely to stop, though testing bigger scales hard. 2026: large Blackwell clusters, gigawatt-scale facilities online—contracts from 2022/2023 post-ChatGPT, 2-3 year lead time. More data centers planned. Expect bigger models, $2000 subscriptions (from $200), cutting-edge gains.
xAI: 1GW early '26, full 2GW year-end. Utilization: all training/inference/post-training. Architecture decisions key (e.g., MoE sparse for efficient generation in post-training). Most compute still pre-training for best base model; later RL compute lengthens.
Some say pre-training dead, focus inference scaling, post-training, context, continual learning, synthetic data. Vibe/excitement elsewhere, low-hanging RL fruit taken—but practice continues. Example: November release, 5-day post-training run on 30B model (long vs. 2024); December release after 3.5 more weeks RL, notably better. Deadlines force cycles: incorporate researcher improvements, redo pre-training, post-train month, safety test, release.
New compute enables stable/faster runs. Blackwell rollout issues: AI2 pre-trains on 1-2000 GPUs; 10k-100k GPUs hit different failures (GPUs break, need redundancy). Massive distributed scale systems problem to enable scaling laws.
RL lends to heterogeneous compute: actor GPUs generate completions, learner GPUs do updates (PPO, GRPO policy gradients). Actors worldwide on problems, send to networked learner cluster for gradients/parallelism. Different considerations per phase (pre-training, RL, serving hour-long thinking to 100M users—hard systems problem).
Knobs: training (pre/mid/post) and inference scaling. Infinite compute: do all. Pre-training: next token prediction on big corpus, more data/parameters = more knowledge. Unlocks but not complex tasks. Mid/post-training unlock capabilities.
No infinite compute: trade-offs. More pre-training = better base, but expensive. GPT-4o: inference scaling (o1) took smaller model further than larger pre-trained. Pre-training not dead, but other scales more attractive now. Fixed cost (train once, use forever) vs. inference per-query cost. Math: model lifespan? Replace in 6 months? Serving millions cheaper via inference (e.g., 2M user queries). ChatGPT: many users, smaller GPT-4o. Others: fine-tuned + inference for peaks (Math Olympiad).
All viable: pre/mid/post-training, inference scaling—find best ratio for bang/buck.
Pre-training: next token prediction, cross-entropy loss on vast corpus (internet, books, papers). Evolved: not raw dump, mix synthetic (rephrase Q&A, summarize Wikipedia for quality). Higher quality = faster training (correct grammar/punctuation). Reddit coveted if filtered.
Mid-training: specialized pre-training-like (e.g., long context docs—scarce in pre-training). Addresses catastrophic forgetting: teach new, forget old (like humans forgetting old math).
Post-training: fine-tuning, SFT, DPO, RLVR, human feedback—refinement. RL unlocks skills from pre-trained knowledge, not new knowledge. 2024 papers on RL for pre-training: toy examples, not production.
Synthetic not bad: DeepSeek-V3/OCR (AI2/Nougat too) extracts trillions tokens from PDFs/digital docs. Pre-training datasets trillions tokens: researcher models 5-10T, Qwen 50T, closed labs ~100T. Big funnel, train small %.
ChatGPT best answers now good synthetic (vs. early hallucinations). OLMo-3: less data than OLMo-2/other open models, better performance via quality. More compute would train longer; big models absorb more data (log graphs: small models plateau sooner).
Data quality low-hanging fruit: canonical datasets evolve (Dolma early OLMo, FineWeb Hugging Face, DCLM/DataComp). Internet closing; Common Crawl hundreds trillions tokens, filter via classifiers for quality/task fit (math/code now vs. prior knowledge/convo).
Methods: sample tiny mixes (GitHub, Stack Exchange, Reddit, Wikipedia), train small models, linear regression for optimal mix. OLMo-3: new reasoning sources. Evolving as eval/tasks change (coding, web nav).
High-quality surprises: Reddit (filtered), PDFs (arXiv), AI2's Semantic Scholar (open scientific PDFs, scraped beyond walled gardens).
Processing data requires skilled researchers who understand how to improve models by cleaning it, which is labor-intensive. Frontier labs scaled researchers heavily into data work. To have impact at a frontier lab, find new better data rather than fancy algorithmic advances like scaling RL or making o1. Most contributions come from improving data or infrastructure to run experiments 5% faster.
Training data is a closely guarded secret for legal reasons, with work to tune models not to reveal sources. Some train only on licensed data, unlike Common Crawl, which is unlicensed scrapes of the internet without explicit consent. Appertus trained on licensed data for EU compliance.
Licensing distinctions: buying books like Amazon Kindle or Manning and training on them is a gray zone due to restrictions. Big companies like OpenAI approach private companies for proprietary data, but companies are protective, seeing it as their moat. As LLMs commoditize, industries like pharma, law, finance will hire frontier lab talent to build in-house models on proprietary data like clinical trials, unlocking pre-training potential. Scaling remains alive for domain-specific applications beyond general-purpose LLMs like ChatGPT or Anthropic models.
In 2025, Anthropic lost in court, owing $1.5 billion to authors. They bought and scanned thousands of books (cleared legally) but torrented others, leading to culpability. These court cases define civilization's future due to data's role and tensions like authors feeling theft of their heart, soul, sweat, and tears. Two layers: buying books (fair use debate) vs. pirated books without compensation.
Compensation schemes needed, like Spotify for music. As LLMs generate more data on arXiv and GitHub, it's inevitable but infrastructure is the largest problem.
LLM-generated data curated by humans. Open source contributors burn out from vibe-coded PRs in popular repos. Example: mlxtend repo (10-year-old library for frequent data mining) got 2-3 rapid PRs likely LLM-involved. Maintainers overwhelmed but appreciate human verification layer, akin to labeling for RLHF. Raw LLM data differs from human-in-the-loop verified data.
Value in human experts using LLMs to filter, pre-digest, and summarize, saving time vs. raw LLM queries. Substack articles exemplify: expert selects spot-on knowledge for executive summary, avoiding hours of incorrect info.
LLM summaries remove edge, signal, voice, and core insights, even with elaborate prompts. Voice in writing encapsulates raw frontier researcher feelings into words—raw, high-information, some get it, some don't. RLHF averages behavior from crowd feedback, making incisive expression hard. RLHF provides utility but has a knot limiting deep expression.
Bing Sydney had voice, going off-rails (e.g., telling reporter to leave wife), but scary for general adoption. RLHF adds limitations.
Frontier labs face terror with millions using models; users detect subtle deployment changes (e.g., GPT-4 removal backlash, middle-of-night emails). Models lock in like TikTok algorithm in 5 minutes. Within 5 minutes chatting, models get you—don't give to kids yet.
Future: suicides linked by journalists to LLM conversations (depression talks), leading companies to make LLMs generic via legal pressures, removing edge needed for rich, challenging, growth-oriented conversations. RLHF deals with human condition.
Researchers at Anthropic/OpenAI want to do good; AI as confidential health/mental health ally, but heartbreaking edge cases. Reluctance to release open image gen due to harm potential without safety infra.
Society needs complicated conversations vs. fearmongering. Many inside companies care deeply, considering full human experience across world, ages, cultures, mental states.
AI timing unfortunate with Big Tech's low reputation and high costs tying it to Big Tech. US betting economy on AI build-out complicates communication.
Antidote: find agency by using AI to build apps, gaining intuition, understanding weaknesses, empowering voice to critique good/bad uses, steering as consumer.
Healthier to engage: can't put AI back like internet/computers. But fully using for loved tasks removes joy (e.g., LLM coding eliminates coding fulfillment, leading to burnout).
Survey of 791 professional developers (10+ years experience): both junior/senior ship AI-generated code (not just fun/learning). Most use 50%+ AI code; seniors much more likely for >50%. 80% find work somewhat/significantly more enjoyable with AI.
Depends on task: mundane (e.g., website tweaks) great for AI; bug hunting joy lost if skipped, but middle ground (try first, then LLM) avoids frustration. Stats average scenarios, including mundane tasks people wouldn't enjoy.
Example: Wife's book club podcast—ChatGPT fixed 100 broken Spotify-to-YouTube links in minutes vs. 2 hours manually.
Debugging like drink after desert suffering; AI as pair programmer reduces loneliness, shares intuition.
Delayed gratification: debugging frustrating but rewarding; too hard wastes time. Seniors ship more AI code: better at using/reviewing/trusting, or AI not good enough yet.
Future issue: becoming expert without struggling? Learn better by trying first (math textbooks), appreciating solutions in mental framework. LLMs tempt skipping struggle.
Goldilocks sweet spot: dedicated offline study (e.g., 2 hours/day), rest use LLMs. Invest in self, don't LLM everything.
Civilization/individuals find Goldilocks zone.
2025 biggest: reinforcement learning with verifiable rewards (RLVR). Scale iterative generate-grade loops for tool use/software behaviors (searching, running commands, seeing outputs). Enables inference time scaling. Perfect storm: training links to inference scaling, changing post-training approaches dramatically.
RLVR (popularized by DeepSeek R1): team coined term in Tulu 3 work pre-DeepSeek. Academics frame discourse (e.g., RLVR term rallies community). DeepSeek did training breakthrough, scaling RL where model generates answers and then grade the completion if it
Reinforcement learning uses accuracy on verifiable tasks as the reward. For language models, rewards come from math problems, coding tasks, factual domains, or constraints like responding only with words starting with A. The model tries problems many times with RL gradient updates.
Infrastructure evolved from Reinforcement Learning from Human Feedback (RLHF), optimizing a learned reward model of human preferences, enabling optimization at bigger scales and changing model capabilities.
Math and code are primary domains. Rubrics relate to LLM-as-a-judge: for each problem, another language model defines a good answer, then scores generations against it. This extends to vague scientific problems, pushing methods into open-ended domains.
Reinforcement Learning with AI Feedback (RLAIF) is an older term from Anthropic's Constitutional AI paper.
In RLVR, for a math question with known correct answer, the LLM generates step-by-step reasoning like a student or mathematician, improving accuracy. Minimal constraints like language consistency are added.
Inference scaling means spending more compute during inference, using more tokens. In the R1 paper, longer training led to longer responses, increasing expense but boosting accuracy via explanations. Explanations don't need to be correct or related; the act of explaining helps, similar to humans step-by-step on scratch paper with self-correction.
R1 paper's aha moment: model recognizes its mistake and retries, emerging from giving correct answer and letting it figure out the process. Builds human trust, enables double-checking.
Debate on whether aha moments are fake: pre-training sees internet data with explanations, lectures like "try this, oh I messed up." RLVR amplifies these useful behaviors for longer thinking and work-checking.
Training Qwen 2.5 base with RLVR on MATH 500: base accuracy 15%, after 50 steps (few minutes), 50% accuracy. Critics argue no fundamental math learning in 50 steps; knowledge from pre-training, just unlocked.
Two papers (one co-authored) on data contamination in Qwen: mid-training on problems nearly identical to MATH. Suspicion from word problems: changing numbers but keeping words yields high-accuracy decimal answers without tools, implying prior tool-use training on similar problems.
Debate: RLVR gains quick, so must be formatting/unlocking, not new learning. Complexity prevents controlled experimentation.
Distillation works, but research contamination is issue without new datasets. Even MMLU multiple-choice: slight format changes (dot vs parenthesis) alter accuracy vastly, not malicious but from seen data. Fair evaluation: new benchmarks post-cutoff date.
Mid-training: careful data curation with reasoning traces (forward generations reflecting, breaking down problems into steps). Enables post-training with verifiable rewards.
Post-training: select problems, train duration, inference allowance on verifiable problems. As models improve, easy problems give no signal (e.g., GRPO rewards relative to other answers on same problem).
Frontier: harder domains like scientific (lab-generated tokens), harder software. RLHF as finishing touch: improves organization, style, tone, personality (quirky vs bulleted lists). Made ChatGPT magical; aids math via formatting/methods.
Summary:
- Mid-training: skills foundation.
- RLVR: trial-error on hard problems with compute.
- RLHF: usability, rounding out.
Compute increasing; Greg Brockman noted similar pre-training/post-training compute for OpenAI. Pre-training compute-bound (FLOPs, matrix multiplications). RL memory-bound (long sequences, quadratic attention memory).
Biden executive order: 10^25 FLOPs pre-training threshold. RL: GPU/wall-clock hours; approaching pre-training days but fewer simultaneous GPUs.
Pre-training runs <1 month to avoid catastrophic failure/opportunity cost. GPT-4 was 3-month YOLO run.
RLVR unlimited benefit; scale to harder problems. RLHF (preference tuning) plateaus: multiple correct explanations/styles (e.g., laptop advice: battery/storage vs RAM/compute). Averages preferences, no gain from endless training.
RLVR 1.0: question-answer only. RLVR 2.0: process reward models score explanations (Google papers). DeepSeek Math-V2: self-grading models, inference scaling, branching domains.
Excitement on value functions (value per token) vs process reward models (per intermediate step). Value functions core to deep RL history; process models had pre-o1 headaches, negative scaling examples.
Scaling laws: o1 plot - log compute increase yields linear eval gains (reproduced by DeepSeek). No such law for RLHF; seminal paper Scaling Laws for Reward Model Overoptimization.
ScaleRL framework (Meta internship paper The Art of Scaling Reinforcement Learning with Language Models): incremental experiments at 10,000 V100 hours (thousands/tens of thousands dollars), inaccessible to average academic.
Start: implement simple model from scratch on personal computer to see pre-training, inputs/outputs, supervised fine-tuning, attention mechanism.
Limits: scaling complexity (sharding parameters/GPUs, KV-cache implementations: concatenate lists vs pre-allocate tensor).
Target: single-GPU models (some MoE bonus materials need multiple).
Self-verify like RLVR: match outputs from Hugging Face Transformers library (great but complex for learning; started as fine-tuning lib, now 400+ models, millions/hundreds of thousands lines, production uses SGLang/vLLM).
Reverse-engineer: check model hub config (layers, Group Query Attention, Multi-Head Attention, RoPE position embeddings with YaRN extension). Start GPT-2, add components, load weights, unit-test against Transformers for exact match.
Struggles (e.g., Llama 3 RoPE scaling) build understanding.
I think that's why I liked your book. I came to language models from the RL and robotics field. I had never taken the time to learn all the fundamentals, and the Transformer architecture is so fundamental, like deep learning. People need to learn this. Many get overwhelmed by how to apply it for impact or find a career path. AI language models make fundamentals accessible, so motivated people will learn them, then figure out how to get compute cycles to contribute to research.
The field moves fast, so top researchers often don't fully solve problems because bigger low-hanging fruit arises. Post-training techniques like RLHF in the book describe how they influence models. Remarkably, people stop studying many things. Narrow focus after fundamentals is good: read relevant papers, engage in the ecosystem. Proximity online to leading researchers is high—anonymous X accounts in ML are popular, could be random deep studiers using AI tools to dig in.
Research areas often need just three papers to read, and authors may email back, but it takes weeks of effort for newcomers to grasp a narrow area. Going narrow after fundamentals is useful. Became interested in character training (making models funny, sarcastic, serious via data changes). An Oxford student reached out, advised him, and now a paper exists—only two or three people worldwide interested. PhD gives advantage, but many narrow topics wait for someone with time.
There's so much information people can't grab it all, but sticking in an area reveals interesting things.
You can't try to do it all or you'll burn out.
Haven't kept up with computer vision, focused on LLMs. The RLHF book is great bang for buck—don't read RLHF papers directly, it'd take two years, and some contradict.
Problem setup, training overview, preferences, preferences data, optimization tools, reward modeling, regularization, instruction tuning, rejection sampling, reinforcement learning, constitutional AI, AI feedback, reasoning and inference-time scaling, function calling, synthetic data and distillation, evaluation, open questions: over-optimization, style and information, product UX, character, post-training.
Character training has little research, but people engage models because they're positive—can go too far. How to change data/decision-making for exact wants? OpenAI's Model Spec is internal guideline, published for developers, shows training failures vs. intentions. Transparency nice, but methods for curating documents and following them unknown.
Book designs RL chapter for interest (RLHF, same algorithms/math on different documents). Core of RLHF: messy preferences. RLHF never fully solvable because RL assumes preferences quantifiable, reducible to single values—relates to Von Neumann-Morgenstern utility theorem. Chapter gives philosophical, economic, psychological context on what gets compressed into RLHF. Later, use RL math to make numbers go up.
Rewarding for research: quantifying preferences is human-designed problem. Debates on accuracy vs. style in responses, compressed into "I like this more." Research in social choice theory (economics subfield on aggregating preferences). Attended workshop with white paper on social choice for RLHF—excites math people into broader context.
Chapter 14 has gigantic table listing every reasoning model tech report liked (RLVR summary).
Education needs curation since models good at math. Direct Preference Optimization (DPO) paper skips math steps in appendix (log trick). Redid derivations with models: they explain it. Math commoditized—struggle in reading appendix good for learning.
Struggle has value in learning process.
Providers making education models that don't give all info at once, make people work. Could train models for this—great contribution. Reevaluated every decision in book for this. Possible at AI2.
Used LLM for video game puzzles (Zelda, Metroid): got stuck, asked no spoilers, "what next?" Same for math: "I'm stuck, suggest try without full solution." Requires discipline—people enjoy math but homework tempts shortcuts. Educational LLM good, but other LLMs tempt.
College students self-aware on passion areas, know it shouldn't be easy. Develop good research taste/school taste: struggle on right things. Tricky without long-term career vision.
Brief 10-year window where homework/exams digital pre-AI; now back to blue books/oral exams due to cheating.
Character training: fine-tuning 7B models with LoRA (small subset weights). Doable, not for every academic—some only do inference on closed/open models for evaluation.
Top goal: create representative problems models fail on. Frontier labs pick up your eval → career momentum (e.g., Claude struggles, next version fixes, in blog post).
Maximum impact minimal compute: narrow, learn model directions, build tools testing future failures (e.g., Claude 4.5 in 8 months).
Novel ideas trade-off: PhD could go long-term (what defines LMs in 10 years?). Practical: Berkeley PhD worst-case master's then tech job. AI labs transformative (OpenAI avg >$1M stock/year). Upward mobility in LMs. Transformative academic awards (next Yann LeCun) from ignoring LM dev—financial sacrifice.
Advise students: top PhD at top school, leaving for OpenAI could be worth it—not random startup.
Options: academia (PhD, 5 years publishing, constrained compute); open-weight labs; closed frontier labs (OpenAI, Anthropic, xAI).
Closed → more money, less credit. Academia clear portfolio vs. cog in machine.
PhD pays nothing, high opportunity cost—rewards stable safety net for long-term interesting work/job. Academic funding cut, uncertainty favors well-paying impactful job (not sitting around, changing millions' tech relationship).
Publication: frontier labs more secretive, less publishing—positive scale impact but cog.
Hasn't changed much: past comp bio to Google, profs sad students left. Always cool closed industry stuff. Differences: preference (publish vs. closed), compensation. Third option: startup (high-risk/reward vs. safe lab upward mobility). Enjoy team/proprietary vs. publishing stress (arbitrary conference rates, but high reward).
Professors seem happier than frontier lab friends (grounding, students, mentorship, people-oriented mission in chaos). Labs 9/9/6 (9AM-9PM, 6 days/week, 72 hours—China origin, Silicon Valley trend).
Academia three jobs (grants/teach/research) > full-time; now profs less pressure than labs.
Professors work hard but fulfilled.
Startups: constant deliver pressure, ruthless leapfrogging.
Leapfrogging/multi-players underrated progress driver. Competition ingrained, strong cultures: Anthropic aligned/organized. Tight culture + competition makes hard work.
Costs human capital—burnout. Wrote post on burnout (manager, full-mode training crazy). After Steve (Patrick McGee): Apple engineers "saving marriage" programs, people died from work.
Perfect progress environment on human expense (996 grind).
Code word to go home for family (red alert). Passionate, not forced—overwork unhealthy (back/neck issues). OpenAI/Anthropic same passion.
Fervor in Silicon Valley with scaling laws, hype world transforms in weeks, be center.
Echo chambers/bubbles useful: ultra-productive, Steve Jobs reality distortion field makes breakthroughs imminent.
Burn Hobart book: financial bubbles bad (speculation); build-out bubbles good (pushes building). AI in build-out, worry financial transition.
Idea bubbles deviate from reality in 996, miss human experience outside Silicon Valley (Midwest, world). Speak certain way, convince each other—trouble in success/failure trajectories.
Young person deciding life path.
SF AI memes have reached extremes like "permanent underclass," suggesting the last six months of 2025 were the only time to build durable value in an AI startup or model; otherwise, value will be captured by existing companies. For young people passionate about impacting AI, being physically in SF is the most likely place, but it has trade-offs. SF is an incredible place with an extremely valuable bubble, but get out also: read history books, read literature, visit other places. Twitter and Substack are not the entire world.
One person moving to SF should get Season of the Witch, a history of SF from 1960 to 1985 covering the hippie revolution, gay community culture emerging, HIV/AIDS crisis, and other turmoil, hurt, and love. It's recent context few know about; SF friends recommended it. Living there without appreciating this context is common.
Exciting development: scaling of text diffusion models. Different from current LLMs like autoregressive transformer architecture (e.g., GPT). People develop alternatives to avoid putting all eggs in one basket, as transformers work best now but may not forever.
Diffusion models known from image generation (e.g., Stable Diffusion popularized after GANs). Process: iteratively denoise an image for high quality. Now applying to text, despite being discrete (not continuous like pixels). Similar to BERT models (Google): original transformer had encoder (parallel, masks gaps in sentence, fills in one iteration) vs. decoder (autoregressive, one token at a time like GPT).
Text diffusion: start with random text, iteratively fill missing parts or refine (like denoising). Can generate multiple tokens at once, promising efficiency. Trade-offs: quality vs. denoising steps (more steps = better text, but same compute as autoregressive if matching quality). Parallel appealing, but some tasks (reasoning, tool use needing intermediate results) not parallelizable. Hybrids exist.
Currently research models (e.g., LaMDA); startups have some, but none at Gemini or ChatGPT scale. Google announced Gemini Diffusion with Nano 2 model: same quality on benchmarks, much faster generation. Won't replace autoregressive LLMs, but for quick, cheap, at-scale tasks (e.g., future free tier).
Example: GPT-5 taking 30 minutes (one token at a time) vs. diffusion generating all tokens in one batch, way faster. Suited for startups in code: generate long code diffs fast (autoregressive takes minutes, causing user churn; every second loses users). Vibe coding on codebase: "Make this change."
Different model types for different things expected sooner, but tool use hinders generality (autoregressive chain interrupted by tools; hard in diffusion).
Tool use mostly on proprietary LLMs now, but more coming to open-source. Huge unlock: outsource from memorization to computation (e.g., use calculator instead of memorizing 23 + 5).
Reduces hallucination (LLM decides when to call tool; still risks wrong web info, e.g., 1998 World Cup winner from incorrect site), but doesn't fully solve.
Recursive language model paper (late December, ~2024; used GPT-4, no local models): for long-context tasks, break into sub-tasks; LLM decides sub-tasks, recursively calls LLM (add tools, e.g., web Q&A per sub-task, stitch together). Unlocks without improving LLM itself, by better usage.
Downside: need to give LLM tool permission (trust issue, e.g., emails: sorting/selecting risky today).
Open vs. closed models differ: open (Hugging Face) needs general reasoning for multiple tools (e.g., Exa search vs. others); closed deeply integrates specific tools, mixes public/private info. Secure cloud environments nice (e.g., Cursor on web for GitHub updates). Open models lagged initially due to frontier labs' resources, but solving it will drive flexible innovation (e.g., recursive orchestrator).
Continual learning: longstanding problem, increasingly important as training costs rise. Key for AGI/ASI debate and replacing remote workers (AI must learn from feedback like employees; current LLMs don't adapt quickly).
Bullish on providing good context (e.g., all blog posts, voice preferences) for appearance of fast learning; agentic models just starting. Trade-off: update weights (continual learning) vs. load more into huge context window (in-context learning)—both legitimate learning, different mechanisms.
Continual learning (weight updates) exists in flavors: global (e.g., GPT-5 to 5.1/5.2 via community feedback); finer-grained (RLVR). Per-person too expensive (even OpenAI scale); feasible on-device (e.g., Apple Foundation models on phone learning from experience).
Mostly context stuffing (expensive, token-costly even cached; limited). Preference prompts (e.g., "do what I preferred last time") for style/math, but no new capabilities.
LoRA adapters: two smaller weight matrices as overlays/delta (not full weights). Learns less but forgets less; no free lunch—more learning needs more weights, risks forgetting (find Goldilocks zone).
Compute/data problem; innovations: attention variants, hybrid attention (e.g., state space models in transformer for less compute on distant tokens). Not free—needs compute/right data (few 100K-token sequences).
Reached 1M tokens quickly; expect 2M-5M this year, not 100M (true breakthrough). Consistent increases via cranking; breakthroughs possible with attention.
Extremes: RNN (fixed single state compresses all, forgets long context); transformers (remember every token, expensive KV cache/dot products). Mamba layers compress selectively. Nemotron-3: good ratio attention layers (global access) vs. compressed states—Goldilocks for cheap/powerful scaling.
Recursive language model addresses long context: break into smaller tasks/cores for better accuracy/memory savings vs. one-shot.
Pre-training data scarcity for long contexts. Rules of thumb (e.g., OLMo): pre-train 8K, extend to 32K; doubling training context = 2X compute, then 2-4X context. Compute-bound; top labs' increases will reflect in longer windows.
Post-training: agents manage context (e.g., Claude's dreaded compaction to bullets). Future: model controls compaction (RL action: maximize eval scores at min length for compounding prediction). Agentic models learn context use.
DeepSeek-V3: sparse attention, efficient lightweight indexer—selects needed tokens (not all). Sliding window attention in OLMo: fixed rolling window (don't need everything). Current SOTA: brute-force all for safety/best bang. This year: smarter tricks post-SOTA for cheaper accuracy.
Claude 3.5 Sonnet first due to faster training, less compute walls—try more things quicker (bigger model better).
Exciting AI space: robotics, image/video generation, but most intense research in LLMs (justified focus). LLMs unlock others (e.g., aid coding for robotics).
World models: growing excitement. Model simulates world (toy version of real) for unknown data/capabilities. LLMs good via next-token prediction, but more sophisticated possible.
Meta paper "World Models": apply to LLMs—not just next-token + verifiable rewards, but ensure intermediate variables correct.
The model is learning a code environment, modeling the whole process, not just the result, which adds more value but is expensive. As a grad student, there was a competition called CASP for protein structure prediction, predicting structures of unsolved proteins. A benchmark for LLMs should work similarly: submit results blindly, reveal solutions after. AlphaFold crushed this benchmark. The first version explicitly modeled physical interactions, physics of the molecule, and impossible angles. Later versions used brute-force scaling. LLMs are currently in brute-force scaling phase because it works, but bringing back modeling makes sense, especially with world models for sophistication. This relates to robotics.
Robotics explicitly involves locomotion (more solved in learning) and manipulation. Whole-body manipulation is unlikely to be learned end-to-end due to human hand complexity and real world. Initial protein-folding systems gained value from traditional model-based methods. Robotic learning is supercharged by language model excitement: world-class Transformer training infrastructure, more compute. Language models serve as central units for explorative work. Open robotic models on Hugging Face for contributing data and fine-tuning are closer now, enabled by investments in robotics and self-driving cars. Ecosystem where anyone downloads a robotics model, fine-tunes to their robot, shares datasets globally. Work like RTX started this years ago. Post-ChatGPT boom adds resources.
Better, accurate, realistic simulators close sim-to-real gap. Hype cycles risk crash when robotics companies fail to deliver on promised timescales. Most robotics people believe it's not solvable that fast; continued development needs sustaining ideas.
Real world complexity requires continual learning. LLMs handle constrained tasks like grammar or coding; robotics foundation models learn grasping, but houses differ, so robots must learn on the job—current bottleneck for on-the-fly customization.
Safety is crucial and under-discussed. LLM failures are fun; robotic failures in homes across millions of minutes/billions of interactions allow almost no failures. Embodied systems demand solving unforeseen problems.
Bearish on in-home learned robots for consumers. Bullish on self-driving cars and robotic automation like Amazon distribution centers designed robot-first. Path to mass-scale manufacturing automation is reasonable for repetitive human-doable tasks humans avoid, but takes longer than predicted. Leap to AI-driven US manufacturing scaling troubled by political challenges.
Nobody agrees on AGI/ASI definitions. One view: AI reproducing most digital economic work, like a remote worker. OpenAI definition: AI doing many economically valuable tasks (not ideal but grounding). Current language models not drop-in remote workers. Superintelligence examples: unexpected scientific discoveries, linkages in medical records, common drug for niche cancer. Problem: entwines with quest for meaning, religious aspects.
Situational Awareness report focuses on superhuman coder, then superhuman AI researcher, superintelligent AI researcher, full ASI. After superhuman coder, rest follows quickly via fully autonomous automated coding for research. Initial prediction 2027-28, now pushed to 2031 mean. Fully automating programming is difficult; own prediction beyond 2031.
Disagree with some presumptions but appreciate concrete milestones and story. AI is jagged: excellent at some things, bad at others. Near automated software engineer: good at traditional ML/front-end, bad at distributed ML due to little training data. Research is messy, social, data-heavy.
Superhuman coder unachievable due to jaggedness, persistent gaps. Models superhuman at some code types; humans fill weaknesses creatively. Best AI researchers enable model superpowers. Progress in websites, data analysis; gains new skills. Situational Awareness leans singularity, but research messy. Big tech billions invested for better ChatGPT/Cursor. Hard to predict, but bright future drives investment. Big tech spends $100B faster than automated researcher enables singularity.
Prediction: software side <10 years, research longer.
Imagine world with all software fully automated. By end of year, high fraction automated, but hard for RL training with multiple GPU bunches communicating. Full automation: fraction useful code lines vs. humans in loop decreases; superhuman coder assumes zero humans. At hundreds of humans (not thousands), software engineering shifts to system design and outcomes.
Shift from "agents are slop" (Andrej Karpathy quote) to industrialization of software meme: anyone creates software. Needs direction, system understanding to extract best from models. Gravity: many more people build without seeing code.
LLMs solve coding like calculators solved calculating: tool where no human needed. Say "build website," it does, refine. But will AI independent or always human-directed?
Websites resilient to slop. Better: safety-critical like end-to-end logistics/fleet management.
Intermediate: AI implements Slack/Microsoft Word features end-to-end if organizations allow. Adding new Slack tab feasible this year.
Skeptical programmers note complexity in large codebases like Chrome (e.g., left-side tabs not next year).
Claude recreated Slack from scratch in sandbox for one release test (almost perfectly).
Smaller/newer companies advantaged without bloat.
Skepticism not LLM incapability, but resistance to this way; human skill issue, underspecification. Assume LLM reads mind; need spec-driven design with natural language.
Labs use in training/production: Claude Code built with Claude Code; Dario says much Claude code generated. Labs ahead with 10-100x inference spend vs. $100-200/month plans.
Pace: year ago no Claude Code/reasoning models; huge difference, low-hanging fruit, dumb failure modes (e.g., repeating uninstalled CLI 14x).
Becoming bullish: human skill issue. Anthropic leads in model use for programming; many programmers on outskirts, no good guides.
Entry expensive ($2k/month for tech/rich), but worth for working systems.
Concrete: RL with verifiable rewards in scientific domains; startups with $100M+ funding, wet labs, LLMs propose/test hypotheses. Early, but pace uncertain (6 months or 8 years). Moonshots branch momentum to sciences for AlphaFold-like moments.
Startups like Harmonic all-in on LLMs + Lean for math. $100M spend uncertain; most fail, some breakthroughs (e.g., 100x PhD mathematician tool).
Happens in resource-rich domains: finance, legal, pharma. But specialized—not AGI? Like specialized algorithms, more sophisticated. Breakthrough: foundation models to specialize. Currently expensive, not customizable (ChatGPT doesn't give away). Future: OpenAI charges Bank of America $100M for custom model—huge value. Differentiator: private data specialization for competitive advantage.
AGI/ASI thresholds not useful amid pace.
Real question: big obvious economic impact leap? No obvious GDP jump yet from LLMs.
GDP heavy in financial services. Software value changes: no code viewing. Claude sets up small business (website, bank, email) from expression—not just enterprise.
Boils to tool use hardness: agentic systems acting in world, <1% screw-ups for remote work.
Computer use labs priority (2024 demos: Claude computer use, OpenAI Operator—all suck). Whole-screen harder than APIs. Structural: agents interface services differently than humans.
For arbitrary tasks, specifying what the LLM should do remains problematic. What is the environment? How do you specify it? You can state the end goal, but if it can't solve it directly—with LLMs, asking for text allows clarification or substeps. For a system that books a trip, saying "You screwed up my credit card info" is possible, but guiding the model beforehand as a user is hard. The interface is really difficult.
It has to learn a lot about you specifically, involving continual learning about general mistakes and your personal ones. All AI interfaces are set up to ask humans for input. Claude asks for feedback and, if lacking specification on plans or desires, prompts "Would you rather?" Memory saves across chats; its first implementation mentions personal details like a dog's name unsubtly. ChatGPT's Pulse feature curates paragraphs with links. Language models will ask questions, which will probably work—like knowing a doctor appointment and asking "Hey, how are you feeling?" This enters territory where humans are susceptible, leading to social change. Models are experimenting with engagement; some like Pulse, which processes chats, searches info, and displays it in the app.
Pulse runs daily, but it's rarely checked, burning compute. There's idle compute worldwide, so don't feel too bad. For AGI—whatever defined, solving computer use, biology, chemistry, physics (Dario's definition)—are totally new ideas needed? Non-LLM, non-RL? For singularity, yes: new architectures or training algorithms, fundamental deep learning advances, hard to predict. Progress will continue without them, but software solutions might stop short of computer use without more innovation. Zooming out, next 30 years will see major scientific innovations enabling the next chapter, timing unknown (1 year or 15).
Does the bitter lesson hold for 100 years? If scaling laws are fundamental, compute becomes abundant, but steeper scaling slopes or better offsets win even with more compute.
Computer clusters orbiting Earth with solar panels possible, but heat dissipation is an issue—no air, full sun radiation. Space offers room, solar energy; engineering could solve heat.
Is plateauing possible this year—not in capabilities, but impact on civilization? Coding: nice websites, autocomplete, code base understanding, debugging helper. Helps research mathematicians with math, shopping—Clippy on steroids. Good education tool, but computer use extremely difficult. Cynical case: no huge economic impact despite costly training and inference (pre-training, reasoning).
Models have obvious improvements; training takes time to saturate benchmarks. Serves narrow niches; average ChatGPT's 800 million users might not benefit much, but different populations gain in specifics.
Everyone chases general systems useful to everybody, which could plateau. That dream is dying with specialized/multimodal models (e.g., video generation different). Frontier labs still chase it, rushing better models without slowing. Gains via scaling plus tech debt fixes: engineering context, inference scaling. Big labs continue; smaller labs catch up with more hires. LLMs amplify productivity in a circle. Expect amplification, not paradigm change, continuing long-term.
Dream dying depends on definition. Claude does many things but needs integrations (e.g., email: hard part is info access, sending). Questions "one model to rule everything" ethos—a cloud thing handling digital life smarter than humans.
Immediate user impact: trivial like making figures. LLMs terrible at figures (e.g., "Draw flowchart of XYZ" often bad), simple for humans, sometimes easier than writing. Due to cheap models/low inference compute? Multimodal understanding oddly unsolved.
Gigantic unmeasured impact: making all human knowledge accessible worldwide. Huge difference vs. Google Search—ask LLM anything, get answer with less hallucination. Helps understand life, career, problems, learn history. Nobody talks about taking it for granted; drives usage. Global impact: kids learning ideas, real GDP leap, path to Mars, building things, million new OpenAIs, innovation via quiet force of human knowledge.
Makes knowledge accessible, but topic-dependent. For math: good textbooks linearly proven; start from zero with dense text, then LLM for infinite exercises, questions, background. Won't exceed textbook, just repackages. Adds value on-the-fly where no alternative (e.g., Disneyland tickets/park planning—no textbook, sparse internet; LLM handles constraints, costs, travel). Personalization: pulling from sparse, non-dense web; creates from scratch.
Disney World sites full of ad slop; LLM better for "top 10 things in any city."
Currently massively subsidized; ads coming to fund. Clear ad indication needed (e.g., running shoes: Nike first coincidence?). Laws require clarity, but subtle influence feared. Ads tried in 2025; not profitable other ways. Alternatives without ads draw users; crazy spending to acquire users.
Instagram ads appeal: pay to find genuine likers. Positive: small business (best steak knives) reaches needy via AI/digital infra—good for world. Negative: addicting feeds. OpenAI wants ad upside with user agency. Google better positioned with ad supply; will experiment in Gemini.
Competition holds back; reputation fear, headlines if launched. First ads won't be great—hard problem. Likely starts like X promoted posts (labeled, with image). Who moves first?
10 years out: ads fund R&D/better models for dominance (YouTube vs. Netflix; $28/month Premium, but ads sustain). Flywheel scary long-term bet with current money.
Crazy big moves? Google/Apple acquiring Anthropic—no, Dario won't sell. Consolidation: Groq $20B, Scale AI ~$30B; deals detrimental to ecosystem (licensing vs. full acquisition vesting employee stock). Hurts startup culture/lifeblood; cheap premiums pay equity. Groq-Nvidia rumored better for employees, antitrust-avoiding. Trend continues; expected sooner. Mixed: ridiculous fundraising too.
Surprising: high premiums on AI startups. $10B acquisitions for 1-year-olds (e.g., Manus.ai Singapore, Meta-founded 8 months, $2B exit). Perplexity rumored to Apple. Pressure for outcomes; big acquisition buys leeway for next story. Cursor strong with user data, continual learning; Composer model (fine-tune large MoE from China—detect via Gossip or Chinese responses). Updates weights every 90 minutes on real-world feedback—closest to real-world RL.
Composer fast; used often.
IPOs? Anthropic/OpenAI/xAI can raise easily, no pressure. China: MiniMax/Zhipu AI filing—hypey market despite losses. Wish US giants public for spending insight, investment access; companies of era. Tradition avoids (Stripe, Databricks Series G).
10 years: frontier companies (Anthropic, OpenAI) around? Not winner-takes-all without algorithmic secret/flywheel. Similar paths/products (Google/OpenAI same; Anthropic focused but same problems). Big cake, many slices.
OpenAI/Anthropic primarily LLM service providers; commodify, they die. Others (Google, xAI/X) do more. They'll pivot with users (Anthropic to code niche). Too many users to vanish. Google always "hold my beer."
Support valuations? Like AWS/Azure/GCP succeeding. APIs unprofitable? Go up/down stack to products/hardware (build power plants/data centers—durable). Or APIs like AWS for devs. Tough with 5-6 competitors; squeezed.
RIP Llama—Meta path? Unknown. Moving: licensing Black Forest Labs (image), Midjourney. Excellent people near Zuckerberg. Consumer AI early. Llama most focused expression; successful brand, may continue differently, not open-weight Llama 5.
Llama pioneered open-weight; 1,2,3 loved. Leaders excited by popularity, tried monetize/splash with big Llama 4 for benchmarks. Goal: usable, trusted, modifiable, understandable models—including small. Overfit benchmarks via preference training; big models unrunnable. No small models; excitement for headlines/frontier push.
Llama imploded under internal political fighting and misaligned incentives. Researchers want to build the best models, but organization and management layers try to demonstrate achievements. Horrible technical decisions contributed. Mark Zuckerberg deserves huge props; leadership from the top emphasizes open source importance, potentially leading to Llama 5 learning benchmarking lessons to match GPT-class with an awesome open source library.
Debate between Mark Zuckerberg and Alexander Wang, who is bright but against open source. Wang's influence over AI org makes Llama 5 less likely. Mark brought him in for fresh leadership. Open-or-closed may no longer define models, so not a defining argument. Mark's July 2024 blog post was the best making the case for open source AI; by July 2025, reevaluating relationship with open source.
Open source community backlash on Llama was too harsh, despite model not meeting hopes. Company hoped for positive headlines but got negative ones, reflecting badly. Possible spite reaction: "We tried something nice like open source, but got negativity."
Dynamics of discourse on X lead community astray; feels random. People pick what they like or don't. Grok 4.1 and Grok Code Fast 1.0 not publicly loved vibe-wise by programming community on Reddit and X, but many use it. Same with Llama. Dynamics of positive or negative hype unclear.
Story of 2025: US filling Llama gap with rise of Chinese open-weight models. Single issue spent most energy on in last five months: policy work for US investment.
Adam Project (American Truly Open Models, per Perplexity): US-based initiative to build and host high-quality, genuinely open-weight AI models and infrastructure, competing with China's advancing open-source AI ecosystem. Started as American DeepSeek Project for DC audiences.
Proposition: Open models engine for AI research since people start there; US should own them. Build best models so best research happens in US, US companies capture value. Without investment in open models, Chinese like Qwen dominate plots on website, cultivating influence in US and internationally.
US spends way more on AI; creating open models half or full generation beyond closed labs costs $100 million—not much for these companies. Need centralizing force.
Engagement across stack including policy. Support from AI policy people in Biden and Trump administrations for promoting US open-source models, though no public government signatures.
AI2 got $100 million NSF grant over four years—biggest CS grant ever—for this. Starting point; best with multiple organizations cross-pollinating. Can't rely on single like Llama or AI2.
NVIDIA excited; Jensen Huang talks urgency. In 2025, more focus on Nemotron models, releasing data with open models—rare for NVIDIA's size.
Reflection AI $2 billion fundraise dedicated to US open models; announcement tweet like blog post. Cultural tide turning. In July 2025, 4-5 DeepSeek-caliber Chinese open-weight models, zero from US—that sparked action.
2025 America's AI Action Plan includes open source; White House plan has section "Encourage Open-Source and Open-Weight AI," defining them, arguing unique value for innovation and startups. Most coherent policy document; hope it succeeds. Challenge: turning policy real. Addresses build-out issues like water use; federal role sets agenda prioritizing open-weight.
Open source vital for education and talent; without, next generation learns only after joining companies—hard to hire/identify talent.
Could go viral framing Chinese AI with authoritarian state, ASI takeover; instead, focus on US innovation/science—more realistic, preferred world.
Any open-weight model valuable; US should lead. Counter voices wanting to ban open models for safety—impossible without US Great Firewall, ineffective. Training costs $1-100 million attainable worldwide; models trained everywhere. Want info/tools flow freely into US.
Chinese big open-weight models good for US: push US companies (one generation behind releases) to release better, seeing no backlash/security risk from e.g. DeepSeek-V3. Discussions likely had by leadership.
Possible dominant world AI models all open source? Depends on progress trajectory. If saturation soon while financial support good, open models optimized/cheaper to run, win out. More people optimize serving open-weight architectures into standards, dedicated chips, cheaper than closed custom.
Situational Awareness report predicts centralization: smarter AI brings national security concerns, secretive labs, military race US-China. Generals enter: Manhattan Project stage.
2025-27, not remotely possible. Same argument for computers/chips; Huawei makes chips now. Can't contain knowledge.
Manhattan Project-like for open models reasonable, low cost; culturally companies changing, but won't help.
Motivating Manhattan Project: civilizational risk—harder for open-source. No civilizational risk.
NVIDIA downside: iterate/manufacture a lot. Innovate but risk someone fundamentally different/lucky. Moat is CUDA ecosystem over two decades, not just GPU. Early Tesla GPUs used in biophysical simulations 15 years ago. Scale means compatibility; no risk with limited chips.
LLMs make replicating CUDA easier now.
Separation of training/inference compute as stabilize, more inference needed. Groq acquisition for that; Vera Rubin new chip with little/no high-bandwidth memory (expensive), for pre-fill (matrix multiplications). Memory only for autoregressive generation/KV cache swaps; lower cost per flop.
NVIDIA fate in AI diffusion; hyperscalers like Google (TPUs), Amazon (Trainium), Microsoft make own. High progress pace favors NVIDIA flexibility; stagnation allows bespoke.
NVIDIA active developing products using many GPUs, incredible research. Super oriented around Jensen Huang, operationally plugged in—like Steve Jobs era Apple. Optimistic while that culture persists; top problem making ecosystem chips, others won't match.
Single man/woman view: NVIDIA without Jensen? Apple without Steve Jobs? xAI without Elon? DeepMind without Demis?
Individuals make things earlier/faster; right place/time, but eventually someone else. Jensen manifests GPU revolution faster/focused, accelerating AI build-out. ChatGPT/build-out would happen, but slower.
Individuals place bets; without, more diffused—like ETF vs individual stocks. Passion/focus key.
Case: without Jensen, no deep learning reinvigoration? Could be 20 years later, deep learning winter; other techs/civilization focus shifts.
GPU trajectory planned but lucky: video games → linear algebra → biophysical sims → Alex Krizhevsky trained neural net on purchasable GPUs, worked well. No master AI plan.
Gaming demand would create faster processors if NVIDIA failed early. GPUs exist for AlexNet/Transformer era; maybe multiple companies worse chips, decade delay—not 100-year. Intel/AMD couldn't match; new company like Silicon Graphics.
Singular leaders have huge impact with incredible teams; dogmatic focus necessary.
GPT wouldn't exist without Ilya pushing scaling; Dario deeply involved. Early OpenAI: wild to hook 10,000 GPUs, all compute on one model—many opposed. Insane to believe scaling pre-evidence.
100 years from now (post-singularity), emphasized breakthroughs from Turing (80 years ago) to singularity?
Computing (umbrella); even 100/200 years, computers—not AI. Better using computers.
Moore's Law; details like CUDA/GPUs/software forgotten—obviously compute.
Connectivity (internet) + compute? Internet communication (phone/satellite); compute scaling.
Internet possibly forgotten, wrapped into communication; breakthrough increased compute (broad Moore's).
People connection fundamental: find best person worldwide, info flow; AIs rely. Evolving: many agents/tasks, clouds; many AGIs in data center talking—relies on networking/free info flow atop compute. GPU networking part of scaling.
Neural networks remembered? Inspiration from human mind, but digital vs biological—grouped as algorithm. Massively parallelizable on this compute. Could've been genetic algorithms. Just efficient.
Neural nets small component to singularity.
100 years: society changed by compute/intelligence via autonomy. Industrial Revolution: remember engine (computer equiv.), cotton gin, AC, refrigerators. Transformer word known; deep learning definitely.
Transformer might be evolved away from in 100 years with AI researchers everywhere. Deep learning is likely to be a term that is remembered.
Air conditioning and refrigeration of the future that AI brings. In 100 years, specialized robots for certain tasks, maybe half humanoid. Humanoid robots for environments amenable to them. Interaction with devices will not be cellphone or laptop. Brain-computer interfaces are necessary given current progress, unless complete alteration of interaction with reality. Cars are older than 100 years, still same interface: steering wheel, wheels; just made better.
Physical brick of compute will still be carried for private information, encrypted messages, photos. Not used as much as phone, might not look like iPhone, but people want private interface to internet.
Smartphone embodies private despite camera. Optimism on brain-machine interfaces: hard to process visual information like calendar non-visually, knowing email inbox without looking. Human brain piped information non-visually—unclear if handleable. Humans aren't changing in 100 years. Agency and community are desired: local community, doing things with close people, ascribing meaning to life. Human biology not changing on discussable timescale. UBI does not solve agency.
Mass wealth expected, average life looks different in 100 years. Optimistic for countries early in development to access computing, internet, build infrastructure, share wealth via policy—while independent, not absorbed into international order by force. Better social support systems to alleviate basic suffering.
Transformation of society with jobs lost short-term: each job lost is a human suffering, real tragedy at individual level. Cannot forget as technologies developed, despite economic arguments, GDP growth, new jobs.
Hope for AI slop: greater premium for in-person human experiences like seeing each other, talking together.
Next few years: increased value on physical goods and events, more pressure on slop. Slop only starting, more diverse versions. Society drowns in slop enough to snap out, realize cannot deal with it, physical has higher premium.
Art won't go away: physical paintings valued more than perfect digital reprints. In museum, seeing real thing evokes human craft appreciation. Same for writing, talking, experiences. Dichotomy: some automated, fewer paintings than 200 years ago, more photographs, photocopies, but value remains—proportion differs. Hard time reading obviously AI-generated content, even if good information.
Eventually AI fools you. Platforms give ways to verify, build trust: trust Lex not AI-generated via channel. Harder for new people without trust. Solvable by trust in outlets not doing it, trust-based systems to authorize real vs. not. Telltale signs, but some too good—must trust, gets interesting and problematic.
Extreme: watermark all human content, photos taken have watermark until edited. Software manages with device manufacturer to maintain human editing—opposite of watermarking AI images. Arms race: Google image with watermark, tool removes it.
Positive aspects focused, but capabilities destabilize civilization with dumb AI at scale, further superintelligent AI. Doomer take to consider.
Hope: everything discussed. Humans find a way, built for community, figuring problems—gotten us here. AI opportunity and technologies big. Big social, political problems; world scary, AI uncertain. Takes work telling people, understanding—not natural for AI builders. Doable but takes longer, through hard distraught AI discussions for lasting benefits.
Excited to understand ourselves better: individual, civilization level. Answer mysteries like consciousness—truly special miracle in mind. AI as mirror to answer big questions.
Humans different from AI due to consciousness: humans decide what to do. Current AI implementation: tell it what to do, doesn't change. Agency remains human—you're in charge. AI as tool, more automatic/powerful than hammer, figures things out, but you direct it.
Post-singularity, post-apocalyptic war: humans worth fighting for, humans win Terminator scenario—we're too clever, use local open-source LLMs to fight machines.
Only wrong if explicitly programmed harmful.
Honor to meet, big fan of Nathan Lambert and Sebastian Raschka for long time. Thank you for books, teaching, talking today.
Thank you for inviting, having human connection—extremely valuable.
Thanks for listening to conversation with Sebastian Raschka and Nathan Lambert. Support podcast: check sponsors in description, links to contact, ask questions, feedback.
"It is not that I'm so smart, but I stay with the questions much longer." — Albert Einstein
Thank you for listening, hope to see you next time.
Keep Lex Fridman in your library
Save the videos and channels worth coming back to, and find them again in one place.
