Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club
In a Nutshell
Multi-GPU kernels require fine-grained compute-communication overlap using TMA for all-gather GEMM and register instructions for all-reduce, with Parallel Kittens achieving hand-optimized performance in 50-100 lines of code versus thousands. Intelligence per watt has improved 3x in two years for local accelerators, with 80-90% of queries potentially routable to local open-source models for 50-70% energy savings. Heterogeneous inference disaggregates prefill-decode and attention-MLP phases across specialized hardware, while GPU-accelerated batch simulators deliver 100x throughput gains for RL environments using entity-component-system architecture.
These notes were generated by AI and may contain inaccuracies.
The event is themed as the YC kernel and chip club due to the trend of specialization happening at the chip level. The TPU8 I and T version represent specialization into different ASICs (Zebrafish and Sunfish), driven by sufficient demand for such splits. Training data centers have fundamentally different requirements than inference data centers, with training centers not requiring any bandwidth in and out, while inference centers require proximity to users.
There is significant optimization potential remaining on the CUDA side, kernel side, algorithm side, software side, chip side, and data center side. The hosts include Stuart Soul from Stanford's Hazy Research lab, researcher at Cursor training Composer; John from the CS PhD lab at Stanford, co-advised by Aelia, focusing on LMS, MLIS, and hardware, pioneering intelligence per watt and intelligence per joule; Mark, former PyTorch maintainer, co-founder of GPU mode with Casey Elward, co-founding core automation with OpenAI VP research Jerry Torque; Misha with two decades of hardware-software co-design experience, previously running AI infra at NVIDIA and working on hardware-software co-design at Meta; and Brennan from Stanford working on RL for self-driving cars, focusing on GPU acceleration for simulators.
The proliferation of specialization at the chip level is driven by sufficient demand for tokens. The Zebraish Sunfish TPU V8 represents the first visible split, expected to become more pronounced. Pre-existing patterns include inference workloads using Nvidia for prefill and Cerebras for decode engines. A critical but under-discussed distinction exists between batch size one inference and throughput-optimized inference, particularly relevant for latency-sensitive applications like voice agents where 8-second delays are unacceptable. Batch size one prioritization leads to GPU exhaustion and high costs, representing a chip-level issue requiring specialized solutions.
Sign in to read the full notes
Get access to AI-generated notes, topic timestamps, and more.