Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis
In a Nutshell
Dylan Patel argues that hardware-software co-design across model architecture, systems software, and silicon delivers the real 100x efficiency gains in AI, far outpacing isolated improvements at any single layer. SemiAnalysis built InferenceX as a living benchmark platform to track these cross-layer optimizations, revealing that models like DeepSeek are explicitly shaped for specific chips (Hopper, Blackwell, TPU) while the CUDA moat has shifted downstream to model optimization. The core bottleneck remains memory bandwidth and power density stuck at ~1W/mm², with future scaling requiring co-packaged memory, space-based compute by 2040, and differentiated infrastructure for batch versus latency-sensitive workloads.
These notes were generated by AI and may contain inaccuracies.
SemiAnalysis has 90 people with a large portion being technologists and engineers across the semiconductor supply chain, alongside former hedge fund professionals. The team engages in organic debates where engineers push for the coolest technology while others emphasize cost considerations. The environment is informal and Dylan Patel enjoys moderating these discussions.
Sean from Sequoia introduces Dylan Patel at the SemiAnalysis office alongside partner Sonia Huang. Five years prior, semiconductors were not considered sexy in the West though they remained attractive in the East. Dylan went very long on the sector and built the premier research company in the space, covering technical details, supply chain, and the bigger picture. Rumors indicate SemiAnalysis recently passed $100 million in revenue.
Dylan grew up in a small business environment where his parents owned a motel and gas station. He jokes that his first neural network training involved racially and visually profiling customers entering the gas station to determine which cigarettes to recommend. The cigarettes were displayed across the top of the counter, requiring him to move a step stool based on customer demographics including age, profession, and race.
On his 8th birthday in May, Dylan asked for an Xbox 360 for Christmas instead of a birthday gift. When his cousin from Alabama visited during spring break, Dylan wanted to impress him but the Xbox experienced the red ring of death hardware defect. Dylan opened the device and shorted the temperature sensor to fix it after trying other methods unsuccessfully. This experience opened Pandora's box for him regarding hardware.
By age 12, Dylan was active on forums, posting and moderating communities focused on Android, Apple, Google, and hardware. He tracked Intel, Nvidia, and AMD developments extensively. Smartphones evolved from simple devices to becoming technologically more advanced than PCs architecturally. He built PCs and maintained an economic perspective from his small business upbringing, always analyzing price-performance ratios and margins.
Dylan personally owned an AMD GPU for its price-performance value but maintained that Nvidia was technically superior because they achieved better performance and power efficiency with smaller chips while maintaining better margins. He frequently discussed Nvidia's margin advantages over AMD in GPU discussions during his teenage years.
Dylan achieved grandmaster rank on the North American ladder in Starcraft 2. He describes himself as obsessively good at multiple things. His grades were decent with mostly A's, though he performed poorly in Spanish class despite later becoming fluent in the language. His grades were sufficient for Asian parents but he did not maximize for all A's.
Dylan attended school and earned degrees unrelated to semiconductors. He worked as a quant for two years at a small quant risk firm. In early 2020, multiple events converged: he was denied credit for generating over $10 million in risk-free revenue for his company, his grandmother developed dementia and passed away after falling down stairs, and he experienced personal relationship difficulties. COVID lockdowns began, prompting him to stay with his brother in Nashville.
During COVID, Dylan continued posting extensively online and made significant profits trading COVID-related market movements. Semiconductor shortages occurred around this period. He became involved in an online argument that resulted in his anonymous account being doxed. After initially stopping posting for three weeks, he decided he did not care and launched a real blog called SemiAnalysis on his 24th birthday, posting two articles that received significant traction due to the real name and increased effort invested.
Dylan packed his truck with a tent and air mattress, driving around national parks across America. He negotiated $30 per night motel rooms during weekdays while working, then spent weekends reading semiconductor and AI textbooks and listening to audiobooks in national parks. This six-month period significantly expanded his education while he continued posting blogs.
Dylan attends over 40 conferences annually across the entire supply chain. He particularly enjoys NEURIPS due to the 20,000 AI researchers in his age range, the presence of leading researchers, and the social atmosphere. He also attends highly technical niche conferences like SPIE advanced lithography and SPIE photomask events in Japan where communication can be challenging due to language barriers.
Dylan identifies his ability to bond with anyone regardless of background as a key skill, finding common ground often through technical discussions. He notes that the most interesting conferences are typically the largest ones where major developments occur, though niche conferences provide deep technical insights.
SPIE conferences are exceptionally deep technically. On first attendance, Dylan understood approximately 10% of content. After reading extensively and making contacts, he understood about 50% on the second visit and 75% on the third. Even with continued attendance, he still does not understand everything discussed, contrasting this with NEURIPS where one can quickly map concepts after a few visits.
Conference attendance serves multiple purposes beyond understanding published research. The critical value comes from understanding how research intersects with current technology and differs from existing solutions. Attendees learn unpublicized supply chain relationships, chemical costs, tool consumption rates, and shortage horror stories that reveal single points of failure in the supply chain.
At a Japanese conference, a Japanese engineer shared that in the 1980s his father worked in an industry where the world's only factory producing a specific chemical burned down, causing memory prices to double or triple. Dylan notes the parallel to modern supply chain vulnerabilities.
Dylan agrees that token usage will be the biggest market beyond oil, with inference representing many percentage points of global GDP. The value created from tokens through AI adoption represents the most important development, with inference whether from open or closed models becoming one of the world's largest markets.
SemiAnalysis conducts extensive research for institutional clients while also pursuing interesting public projects. Inference benchmarking was previously point-in-time, becoming outdated as models, software libraries like PyTorch, vLLM, and SGLang, drivers, and optimizations updated frequently. Model costs for equivalent quality have dropped approximately 60x per year due to relentless efficiency improvements.
To address the need for living benchmarks, SemiAnalysis built InferenceX with contributions from CoreWeave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google, and OpenAI providing compute resources. They collaborate with SGLang, vLLM, Radix, and Inferact on open source efforts, and work with Nvidia, AMD, Google, and Amazon on TPU and Trainium support. The platform has received over $50 million in donated hardware across approximately 15 different chip types.
The platform runs daily automated benchmarks on the best models from Moonshot, Alibaba, and other Chinese labs alongside US open source models like GPT-OSS and Neotron. All results and configurations are public, creating a Pareto optimal curve across the interactivity versus batch size spectrum.
Traditional inference comparisons often compare suboptimal points for one system against optimal points for another. InferenceX provides open-source containers for optimal points across the entire interactivity curve, allowing users to download and run the best configuration for their specific model and achieve near-peak performance.
The throughput versus interactivity curve is considered the most important metric in hardware infrastructure, with everything downstream dependent on it. Some workloads require super low latency regardless of cost, utilizing techniques like speculative decoding and multi-token prediction. Other workloads involve batch processing documents where cost efficiency matters more than individual response speed.
Current AI infrastructure treats all workloads identically, but future systems will differentiate between batch workloads and instant response requirements. Anthropic's Claude code fast mode costs significantly more than regular mode, and OpenAI offers priority queue options, demonstrating early movement toward curve-aware pricing.
On one end of the curve, a batch size of 100 might achieve 10 tokens per second per user for 1,000 total tokens per second. On the other end, a single user might achieve 250-500 tokens per second. The average user wants 50-100 tokens per second with appropriate batching. Some workloads accept 4x cost increases for speed when user time or feedback loops are expensive, while others prioritize the 4x cost reduction for batch processing.
Dylan Patel believes space data centers will not matter in the next 3-5 years but projects that in 20 years the vast majority of compute will occur in space. The key variables are the cost and time frame of building terrestrial power versus space-based alternatives. He forecasts that by 2030, OpenAI and Anthropic combined will exceed 100 gigawatts of inference compute, with Meta, Google, and others adding significantly more. By 2040 he expects terawatts of inference capacity, with more than half of incremental compute potentially moving to space while 2030 levels remain under 1%.
Intelligence per watt has improved substantially, with roughly a 40x gain over recent years compared to a 60x cost reduction for equivalent benchmark performance. While current systems remain many orders of magnitude away from human biology in general intelligence efficiency, Patel notes this gap matters less because powering computers is far simpler than powering human brains, which face issues of sickness, disease, food preferences, and sleep.
Efficiency improvements come from three distinct layers: hardware improvements that make chips more efficient, low-level systems optimizations such as kernel-level improvements and matrix multiplication libraries, and high-level model and algorithmic improvements. Patel disagrees with the view that hardware delivered most gains in the last three years. From Hopper to Blackwell, the most optimized deployments achieved roughly 30x improvement on DeepSeek benchmarks over three years, but model-layer advances contributed substantially more when comparing GPT-4 three years ago to current smaller Qwen models with 27 billion total parameters and 2 billion active parameters.
The most important gains occur at the co-design layer between model architecture, hardware, and infrastructure software. DeepSeek V3 experts were explicitly shaped for Hopper, while V4 shapes are optimized for Blackwell and Huawei chips. TPUs perform poorly on DeepSeek despite excelling at other models, demonstrating deep co-optimization of matrix shapes, network IO patterns, collectives, and attention mechanism arithmetic intensity. These elements cannot be easily disentangled across layers.
OpenAI models were previously optimized for Hopper and are now shifting toward Blackwell. Anthropic trains primarily on TPUs but performs inference on both Cerebras and GPUs. Google optimized Gemini 2 for TPU v6e, Gemini 3 for subsequent versions, and the next Gemini release for TPU v7. Running these co-optimized models on older hardware yields poor results.
True breakthroughs arise when multiple layers are co-optimized simultaneously. Separate 2x improvements at the model, software infrastructure, and hardware layers multiply to 8x when kept independent, but co-design across all three can deliver 100x gains. Nvidia co-optimizes from the model layer downstream to silicon. TSMC co-optimizes fabrication, components, consumables, and tools all the way upstream to customer chip designs.
Memory capacity and bandwidth remain major constraints. NAND cells were invented 25 years ago and DRAM cells 40 years ago with no fundamental cell-level breakthroughs. Recent HBM improvements have focused on stacking more layers and increasing speed, but new approaches will stack memory directly on the chip, dramatically increasing bandwidth.
Power density has remained stuck at roughly one watt per square millimeter for two decades across data center and desktop chips. Current Nvidia and TPU silicon still operates near this limit, with chips reaching 1400 watts today and next-generation Nvidia parts targeting 2000 watts. Future designs aim to exceed one watt per square millimeter, reducing required silicon area despite higher power draw and potential efficiency trade-offs. Engineering challenges include thermal management and electrical interference.
Simple energy solutions exist beyond traditional bottlenecks. Millions of diesel truck engines the US can manufacture could be converted to run on gas and coupled to electrical motors operating in reverse to generate power. Although servicing hundreds of such units at a data center site would require mechanics drawn from car shops, the approach represents a straightforward way to expand generation capacity.
Both Nvidia GPUs and Google TPUs will scale massively, with Google projected to produce over 10 million TPUs annually at over $100 billion in value and Nvidia producing tens of millions of GPUs at over $500 billion. Each architecture offers distinct advantages. Nvidia emphasizes general-purpose capability and switch-based connectivity. TPUs provide higher energy efficiency and optimized networks for certain architectures.
The choice depends on hardware-software co-design. OpenAI's increasingly sparse model architectures may perform poorly on TPUs. Anthropic and Google models, being less sparse, may be better suited to TPUs. Fundamental differences include matrix multiply unit sizes, attention mechanisms, expert structures, and network topology. Nvidia's NVLink connects 72 GPUs while Google's ICI connects 8,000 chips at high bandwidth without switches, requiring traffic to pass through other chips.
The CUDA moat narrative has shifted. Model companies now willingly write custom kernels for alternative chips, aided by strong coding capabilities from models like Claude and Codex. With only tens of frontier labs rather than thousands of customers needing compatibility, the traditional CUDA premise weakens.
However, the real moat is downstream model optimization rather than CUDA itself. DeepSeek, Kimi, Zhipu, Alibaba, and Xiaomi models are co-designed for GPUs, making them perform poorly on TPUs. Google maintains the Gemma family as its open-source offering. Downstream users including inference API providers and RL companies therefore default to Nvidia hardware because the available open models are GPU-optimized.
Large labs have already forked PyTorch or built custom infrastructure and no longer rely on open-source implementations. They select the best hardware and co-design their full stack including models and infrastructure software, leveraging AI assistance for kernel development.
Cerebras excels at fast inference, a capability SemiAnalysis uses almost exclusively. The market splits between high-value tasks that justify premium pricing for speed and many workloads that do not require it. The primary risk for Cerebras is model scale: current SRAM-based designs struggle with very large models at long context lengths. If frontier models reach 10-plus trillion parameters with million-token contexts, fitting them on Cerebras hardware becomes difficult. Most revenue at labs continues to come from their best models even when prices rise.
SemiAnalysis differentiates itself by bridging economics and technology, which most analysts fail to do well. The company has 90 people with a large portion being technologists and engineers across the semiconductor supply chain, alongside former hedge fund professionals. Internal debates naturally emerge between engineers focused on cutting-edge technology and finance-oriented team members emphasizing cost considerations. The environment is informal, which allows these perspectives to compete organically.
Certain topics provoke strong reactions. Claims that memory is the bottleneck are acknowledged as true but treated as memes. The assertion that AI has no ROI is particularly infuriating. Denials of model progress also trigger pushback. When people claim models are not improving, not reasoning, or will plateau, the counterargument points to consistent upward trends in capabilities. Saturated benchmarks are replaced by new ones that show rapid improvement.
Semiconductors are complex with many layers of abstraction. Even after 18 years of study since moderating forums at age 12, new information emerges constantly. A recently discovered chemical generating $100 million in sales within a couple-hundred-billion-dollar industry proved essential to every chip despite its relatively small scale. The industry contains roughly a thousand process steps.
The most frustrating occurrences are when people have all facts available yet reach completely incorrect conclusions.
On a 10-year timeframe, space represents an exciting frontier including space data centers and asteroid mining aligned with SpaceX's vision. Co-packaged optics will occur by the end of the decade, with debate centering on timing between 2027-2030.
Companies like Navian are pursuing simultaneous innovation across silicon, software abstraction, and model layers. This represents a long-term bet rather than a two-to-three-year timeline. Approaches combining analog compute with energy-based models are considered exciting despite uncertainty about rapid success.
Deaveen is recognized for consistently helping younger talent and identifying promising individuals. He was ahead of his time with Mosaic. The first interaction occurred in 2019 when anonymous forum activity led to direct messages and eventually a call.
The end state will see continued attempts at vertical integration, but supply chain realities and technology access will drive diversification as the industry scales. Current chip architectures share similar structures with a central logic compute die, HBM on multiple sides, networking on top, and PCIe/IO on the bottom. This pattern appears across Tranium, TPU, and Nvidia designs. Most startups follow this approach except for Grok and Cerebras.
Future development will produce greater bifurcation between hardware and model architectures, enabling co-optimization. Some designs will reach local minima while others reach global optima. Nvidia maintains advantages in general-purpose AI compute due to broad customer feedback across diverse use cases.
Labs do not know their architecture plans even one year ahead. They maintain multiple research bets. Breakthroughs in model architecture such as replacing attention mechanisms could shift optimal hardware requirements. This uncertainty favors maintaining buckets of general-purpose compute alongside specialized ASICs.
Google pays $11 per GPU-hour to xAI despite having TPUs, indicating workloads that perform better on alternative architectures. Google runs three distinct TPU design programs with Broadcom, MediaTek, and an undisclosed third architecture that differs significantly from the first two. This reflects recognition that local minima can become traps.
Every major player will develop its own ASIC program deploying billions to tens of billions annually, with Google potentially reaching hundreds of billions yearly. However, workloads will persist that favor other architectures. AI for science may exhibit different algorithmic patterns than AGI models, supporting continued diversity. Niches will emerge allowing companies to profit even when Nvidia, TPU, and Tranium capture majority share.
Current conditions reflect both demand and supply constraints. Long agent demand is skyrocketing while data center buildouts face delays. Deployment continues accelerating with 20 gigawatts coming online this year and over 30 gigawatts next year despite delays.
Whether compute remains constrained depends on model progress. Model capability improvements such as from Opus to Mythos 5 or Fable 5 expand the addressable market far beyond 2x. Compute capacity has not doubled or quadrupled in corresponding periods, yet the value of tasks AI can perform has grown substantially.
Anthropic achieved net income profitability in Q2 excluding stock-based compensation and may reach full profitability including SBC in Q3. Opus token margins exceed 80% at API prices, though corporate gross margins face some pressure from Bedrock and Vertex deals. High per-token margins enable paying above-market rates for compute. Every additional GPU, TPU, or Tranium can be immediately monetized at positive margins. Doubling compute costs while maintaining 50% gross margin remains viable.
High leverage and growth warrant caution. If models fail to expand valuable work faster than compute capacity increases, conditions will shift. Recent months have shown models expanding their work TAM faster than compute growth, driving prices upward.
Conversations with Anthropic and OpenAI teams indicate continued belief in model progress. Current methods show line of sight to rapid improvement. Models improve faster than six months or a year ago through pseudo-recursive self-improvement where models assist in writing infrastructure and accelerating subsequent launches.
Capital remains a constraint. Google raised despite substantial SpaceX holdings worth approximately $100 billion in liquidity potential. Meta announced a capital raise despite stock price reaction. All major companies will raise debt or equity. Current dynamics show every added GPU, TPU, or Tranium generating gross profit.
Significant heterogeneity exists across gigawatts being deployed. Tranium rental rates reach sub-$10 billion per gigawatt for Anthropic and OpenAI. GPUs typically command $12-13 billion per gigawatt, with recent SpaceX deals reaching $25 billion per gigawatt or $25 million per megawatt annually. Amazon subsidized Tranium pricing initially.
Colocation power pricing has risen from $60 per kilowatt per month to $120-160, with premium facilities reaching $200 for lower-credit customers. Lower-quality facilities in regions with unreliable grid or connectivity transact at $80-100.
Many announced data center projects fail due to repeated delays. Construction teams vary significantly in execution capability. Google's approach allows 1.5 gigawatts of hardware within a 1-gigawatt data center through superior workload-to-power management, achieving higher effective utilization than the typical 60-70% power consumption rate.
Google negotiates utility deals allowing 2 gigawatts on grids rated for 1 gigawatt except for three days annually, using on-site batteries and generators to manage exceptions. These capabilities enable selling more gigawatts or commanding premium positioning. Superior operators extract value through power smoothing, optical switches, and workload management that less experienced players cannot replicate.
A gigawatt given to Anthropic is objectively worth more revenue than a gigawatt given to OpenAI. Both companies could sell every gigawatt they have right now given rate limit problems and token max limits. Codex 5.5 improved capabilities significantly.
SpaceX likely makes better use of hardware than most organizations. They have extensive networking experience from Starlink and power management experience from Tesla. People like Brett Mayo are considered exceptional in this domain.
CoreWeave builds GPU compute that is objectively better than Amazon, Google, or Microsoft's in terms of performance and reliability. The performance and reliability have been tested. However, Google sells capacity six months before it is operational. They use the signed paper contract to obtain debt financing with credit backing, then turn around to pay for the purchase order already issued.
SpaceX operates on a different model: they require running infrastructure before purchase. Having a balance sheet to operate this way creates a significant discrepancy and enables much higher revenue per megawatt.
Five years ago, the expectation was that hyperscalers would own the AI cloud market. The opportunity for NeoClouds emerged because hyperscaler expertise from the traditional CPU cloud world became less relevant or even detrimental for AI workloads.
Amazon was previously considered the best cloud provider due to Nitro NICs that offered tenant isolation with the hypervisor running on the NIC, allowing all cores to be sold. They built custom SSDs by buying raw NAND for lower cost and developed custom Graviton CPUs that drove down per-core cost. These advantages applied to traditional CPU cloud workloads.
In the AI cloud, many of these optimizations hurt performance. Nitro NICs remain worse for performance despite iterations to improve them. Security features designed for time-slicing users across sockets do not apply when customers rent entire 8-GPU servers or 72-GPU racks with long-term contracts rather than short-term rentals.
Network performance at Google and Amazon relied on custom networks optimized for traditional CPU workloads, which did not work well for AI. Microsoft would save money by building their own data centers, but their data center teams were not actually strong. When forecasts doubled, they fell short and had to obtain NeoCloud capacity.
Time to market represents another advantage for NeoClouds. Massive organizations lack individual incentives for faster delivery. Crusoe team members including Chase are hyperlevered equity owners who get rich if they deliver compute faster.
Many NeoCloud operators come from Bitcoin mining backgrounds and learned operations in high-fluctuation markets. Crusoe's main data center lead came from Microsoft.
Jensen Huang absolutely hates a world where hyperscalers hold all the power. He invests in random AI labs and pumps them up, telling investors globally to back these companies because he wants to create a multipolar world. A world where only OpenAI, Anthropic, and Google models exist is one where Nvidia is disadvantaged.
A world where only hyperscalers build compute also disadvantages Nvidia. Therefore, Nvidia allocates resources to NeoClouds, helps backstop their clusters, and supports them in any way possible. While a GPU sold to Crusoe, CoreWeave, Google, or Amazon costs the same price today, five years from now the existence of Crusoe and CoreWeave means Google TPU and Amazon Trainium will be weaker. More inference done with non-closed model labs benefits Nvidia.
The NeoCloud ecosystem consists of wild west operators. NeoLabs have received investments from Nvidia. Some will fail, but some will emerge as strong teams. Crusoe consists of crypto guys who started building data centers and doing flared gas work. CoreWeave started with New York hedge fund guys and crypto operators. Many similar teams started around the same time but failed.
Thinking Machines has a few hundred million dollars of ARR, which is impressive given the product launched less than six months ago despite media coverage about talent departures.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
