Back to Lex Fridman

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

Lex FridmanJanuary 31, 20264h 25m

Topics60

Introduction to State of AI in 20260:00 DeepSeek Moment and International AI Competition2:01 Models Winning 2024 and Predictions for 202510:30 Programming Use Cases21:30 Guests' Backgrounds and Books24:02 Verifying Math vs Code and Using LLMs for Reading and Programming25:02 Landscape of Open LLM Models28:32 Tool Use in Models like Jamba32:31 Interesting Architectural Ideas in Open Models37:01 Transformer Architecture Basics40:03 Where AI Advancement Happens Despite Stable Architectures45:01 Scaling Laws48:00 Inference Time Scaling and RLVR49:34 Data Processing and Research at Frontier Labs1:13:16 Licensing Gray Areas and Proprietary Data1:15:01 Anthropic Lawsuit and Data Compensation1:16:30 LLM-Generated Content and Human Verification1:18:32 LLM Summaries vs. Original Voice and Insights1:21:31 User Attachment and RLHF Challenges1:24:33 Designing for Global Human Experience1:28:03 Finding Agency with AI1:29:02 Developer Survey on AI Code Usage1:30:36 Learning, Expertise, and Goldilocks Zone1:35:00 Post-Training Advances: RLVR1:37:31 Reinforcement Learning with Verifiable Rewards (RLVR)1:39:11 Learning Fundamentals and Entering AI Research2:02:59 Key Ideas in Post-Training: Character Training and RLHF Challenges2:07:02 Compute for Research Contributions2:13:30 Competition, Culture, Burnout in AI Labs2:23:01 SF AI Memes and Advice for Young People in AI2:27:06 Scaling Text Diffusion Models as Alternatives to Autoregressive Transformers2:29:02 Future of Tool Use in LLMs2:34:32 Continual Learning and In-Context Learning2:39:01 Memory Mechanisms for LLMs2:42:33 Context Length Innovations2:44:01 Broader AI Excitement and World Models2:50:32 Modeling Code Environments and Protein Structure Prediction2:52:25 Robotics: Locomotion, Manipulation, and Model-Based Methods2:54:03 Safety in Robotics and Deployment Challenges2:58:00 AGI/ASI Timelines and Definitions2:59:31 World of Fully Automated Software Writing3:05:30 AGI/ASI Profundity and Scientific Moonshots3:13:32 Challenges in Specifying Arbitrary Tasks for LLMs3:18:59 Compute Concerns and New Ideas for AGI3:21:02 Space-Based Compute and Plateauing Concerns3:23:01 General Systems, Specialization, and Amplification3:25:31 Democratizing Knowledge and Learning Strategies3:28:31 Monetization via Ads and Incentives3:32:33 Business Moves, Consolidation, Acquisitions3:36:33 Future of Frontier Companies and Commoditization3:41:01 Llama Implosion and Open Source Debates3:45:54 NVIDIA's Dominance and Jensen Huang4:00:02 Future of AI Architectures and Deep Learning4:12:12 World in 100 Years: Robots, Interfaces, and Human Interaction4:12:32 Job Loss and Human Suffering from AI4:16:33 Value of Physical Goods, Events, and Anti-Slop Reaction4:17:32 Trust, Verification, and Watermarking in AI Content4:19:00 AI Risks and Hope for Civilization4:20:30 Human Agency vs. AI and Consciousness4:22:30 Closing Remarks4:24:04

AI-Generated Notes

These notes were generated by AI and may contain inaccuracies.

Conversation about state-of-the-art in artificial intelligence, including technical breakthroughs over the past year and predictions for the upcoming year. Technical at times but accessible without dumbing down. Features Sebastian Raschka and Nathan Lambert, machine learning researchers, engineers, communicators, educators, writers, and X posters.

Sebastian Raschka authored Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). Best way to learn machine learning/computer science is to build it yourself from scratch.

Nathan Lambert is post-training lead at Allen Institute for AI, author of definitive book on Reinforcement Learning from Human Feedback (RLHF).

Both have great X accounts, Substacks, YouTube courses (Sebastian), podcast (Nathan).

DeepSeek moment: Early 2025, open-weight Chinese company DeepSeek released DeepSeek-R1, near state-of-the-art performance with allegedly much less compute, much cheaper. AI competition accelerated insanely on research and product levels.

Who's Winning: China or US Companies?

Sebastian: Winning is broad. DeepSeek winning hearts of open-weight model workers by sharing open models. Multiple timescales: today, next year, 10 years. No company has exclusive technology access in 2026 due to researchers changing jobs/labs. Differentiating factor: budget and hardware constraints, not proprietary ideas. No winner-takes-all.

Nathan: Labs put different energy into efforts. Hype over Anthropic's Claude 3.5 Opus insane—used it, built stuff, feels like a meme, organic hype. Earlier, Gemini 1.5 from Google had high marketing/wow factor, seemed like Google's moment to retake advantages, but hype faded despite being fantastic. Differentiation lower now.

Agrees ideas fluid, but bottlenecked by human effort and organization culture. Anthropic bets hard on code (Claude Code), least chaotic, advantage if sustained.

China: More labs than DeepSeek—kicked off movement like ChatGPT in US. Zhipu AI (GLM models), MiniMax, Kimi Moonshot shining recently. DeepSeek still strong but losing crown as preeminent open model maker. DeepSeek narrative: platform for more Chinese companies releasing strong frontier open-weight models.

Chinese open-weights risk US business models, but US has paying AI software users; historically, China/other places don't pay much for software.

Future of Chinese Open-Weight Models

Sign in to read the full notes

Get access to AI-generated notes, topic timestamps, and more.