The math behind how LLMs are trained and served – Reiner Pope
In a Nutshell
Reiner Pope explains LLM training and inference using roofline analysis, revealing that latency and cost stem from balancing compute time (linear in batch size and active parameters) and memory time (dominated by fetching total parameters and KV cache, linear in batch size and context length). Optimal batch sizes (~2K sequences) amortize fixed weight fetches, favoring sparse MoE models like DeepSeek V3 (high sparsity enables larger batches and quality gains), with hardware scale-up (e.g., Blackwell's 72-GPU racks) unlocking bigger models via fast intra-rack all-to-all communication over slower scale-out. API pricing reflects these dynamics—higher costs for fast modes, outputs vs. inputs, and long contexts—while equalizing training, RL, and inference compute predicts over-Chinchilla scaling, and KV cache limits long contexts without bandwidth breakthroughs.
These notes were generated by AI and may contain inaccuracies.
Interview with Reiner Pope, CEO of MatX, a new chip startup. Previously worked on TPU architecture at Google. Blackboard lecture format in new studio on model architecture, ML infra. Understanding training and inference in a cluster explains why AI is the way it is, architectures, API prices, AI progress.
Reiner is an angel investor in MatX, unrelated to podcast.
Companies like Claude, Codex, Cursor offer Fast Mode: 6x price for 2.5x speed token streaming. Mechanically, why pay more for faster latency? Could pay 100x more for much faster? Could have Slow Mode for minutes wait, cheaper prices?
Batch size is big effect on latency and cost. Also speculative decoding or multi-token prediction.
Roofline analysis on Blackwell NVL72 cluster (rack of 72 GPUs): memory bandwidth and compute performance.
Two model factors: time to operate on weights, time on context (KV cache).
Estimate inference time approximation: time ≥ memory fetches time, ≥ compute time.
Compute: multiply by all active parameters, work on attention.
Time_compute ≥ (batch_size * active_parameters) / FLOPs (chip compute throughput). Accounts for weight matrix multiplies; attention compute small, ignored.
Batch: serving many users simultaneously. Without batching, cost/economics 1000x worse.
DeepSeek V3: 37B active parameters, 700B total.
Memory: fetch all total parameters (weights), plus KV cache fetch (depends on batch_size).
Sign in to read the full notes
Get access to AI-generated notes, topic timestamps, and more.