Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper
In a Nutshell
Ex-NVIDIA engineer is building Sal Research to deliver the industry's cheapest AI tokens by optimizing for throughput over latency, targeting background agent workloads that run for hours or days rather than interactive chatbots. The core thesis is that making intelligence 1000x cheaper unlocks unbounded token consumption in proactive agents, using every available chip, power source, and distributed data center in the US. Key insight: the shift from human-in-the-loop to background processing removes attention constraints, making throughput-optimized stacks and scavenger strategies for underutilized chips/power economically dominant.
These notes were generated by AI and may contain inaccuracies.
The speaker's job is to make tokens as cheap as humanly possible through every layer of the stack. This includes using every chip, every source of power, and every piece of suitable land in the United States. The goal is to move away from treating agents as expensive consultants that you only consult for hard questions. Instead, the aim is to get intelligence into as many hands as possible.
Sal Research operates as a token factory with an API where anyone can send requests to use open source large language models for any task. The service delivers tokens at prices that are unbeatable in the market. The company also supports building agents on top of this infrastructure by hosting what they call sandboxes—long-running agent virtual machines in the cloud designed for agents that run for hours, days, or weeks.
The core positioning is being a peer company to others that serve different kinds of inference, with the goal of being the absolute cheapest provider and enabler of a certain kind of intelligence use.
The company's theme is abundance—delivering intelligence as a new commodity to as many people as possible at costs sustainable for almost every industry. The belief is that making something 10 times cheaper creates a new product category. The aspiration is to do this for tokens. The profound shift is that machines can now think, and the job is to make as many machines as possible work toward thinking.
Token cost is the current north star, with the goal of having the lowest cost per token in the industry by a significant margin. While tokens are not considered the final unit of work or intelligence, they are what is used today. The speaker expects to move toward outcomes as the unit of measurement. Currently, when consuming tokens through an agent, users do not control how many tokens the agent uses for reasoning. The vision is for agents to self-administer token budgets as opposed to companies setting budgets for how many tokens engineers can spend per month.
Two tailwinds support the company. The first is the rise of open source. Customers increasingly care about owning intelligence and having control and sovereignty over what they depend on. This has created a robust market for customized models or vanilla open source models where users always have the weights and the right to deploy them however they like.
The second factor involves market positioning. Companies like Base 10 and Fireworks have focused on low latency inference, driven by one important customer: Cursor. While that was the right choice a year ago, six months ago it became clear that low latency was not the only requirement for agents. Agents need more persistence and support for long horizon tasks.
The future of agentic inference is long horizon tasks where machines run for hours or days at a time. Token speed becomes less critical—10 tokens per second may be acceptable if it comes with corresponding efficiency advantages. The best latency is no latency at all. The vision is proactive agents that work in the background so that when users wake up, the work is already done overnight.
The more users stay in the loop prompting agents and waiting for responses, the more they become the bottleneck. The goal is for agents to operate on more human time scales, similar to how colleagues are managed—you give high-level tasks and check in once a week rather than every 5 minutes.
The key early indication is test time compute scaling—the idea that giving an agent more time produces better answers. This was theorized about two years ago but only became viable to bet on late last year with Opus 4.5. Opus 4.5 was the first agent suitable for longer horizon tasks, though mediocre at first. More recent models show agents are capable of running for an hour at a time. Average turn or task length keeps getting longer, indicating agents are worth running for extended periods.
The market is unbounded because there is no human in the loop, allowing unlimited token consumption in the background. Human attention span creates limits—if asked to consume 10x as many tokens at coding tools, users may already be at capacity. The unbounded opportunity is tokens consumed in the background or proactively. The projection is ending the year at roughly 50/50 background versus real-time workloads, moving toward 90/10 in favor of background.
Deep research is a top category, especially questions requiring definitive answers over 10,000 sources or more. One customer, Parallel Web Systems, wants to build an index over the whole internet and monitor it in real time for changes—this requires exabyte-scale intelligence.
Cybersecurity is increasingly following this direction. There are exponentially more ways to break code than to generate it. Customers are building agents that break software autonomously to find bugs and proactively patch them. When Fable and Mythos first came out, there was a push to run Fable against every line of code to look for bugs in 20 different ways, including memory errors, business logic errors, and network vulnerabilities.
Security has become proof of work—the best indication of how secure software is comes from how many dollars were spent on Anthropic's APIs trying to break into it. The frontier of intelligence is jagged—Fable does not find a superset of all bugs, and different model sizes find different types of bugs, encouraging diverse sampling approaches.
For individual users, the excitement centers on proactive intelligent agents. This includes a Siri-like system running in the background that understands all emails and text messages received in a day and has an encyclopedic view of a user's life. The goal is for the machine to understand how users interact with it and proactively surface next actions when they open their phones.
The key requirement is incredibly cheap intelligence—being willing to spend tokens without any promise of return. The long view is that intelligence can tackle any verifiable problem, including most software, formal math proofs, and scientific discovery. A dollar cost is now attached to these long horizon tasks, moving from millions to thousands, potentially hundreds or even tens of dollars in the near future.
Human taste remains in the non-verifiable category. The speaker has not solved human taste and does not know if it fundamentally can be solved. The focus remains on quantitative problems, leaving the quality of writing and beauty of art to people.
The approach starts with software. The opportunity on today's chips and data centers is to improve efficiency. The first step was building the entire LM software stack around peak GPU efficiency using Nvidia GPUs to squeeze more tokens from the same chip than anyone else.
This begins at the lowest level of programming kernels. The speaker's background is in GPUs and kernels, having worked at Nvidia starting in college in 2016, witnessing how tensor cores earned their place on the chip.
Tensor cores are specialized units on the GPU that accelerate matrix multiplication. Matrix multiplication is the atomic unit of computation because it provides a succinct way to mix two blocks of numbers together and have them interact. Linear algebra turns out to be a very compact representation of arbitrary relationships in data.
Nvidia started as a graphics company with market share dominance in gaming. Starting in the mid-2010s, they began skunk works projects to make graphics processors suitable for machine learning. Jensen had conviction by 2015-2016 to double down, allocating more silicon die area to this emerging capability.
The first generation of tensor cores received about 5-10% of die area to accelerate convolutions, the fundamental operation for computer vision models at the time. The software team squeezed all possible performance from the chip, learning Nvidia's ethos of chasing speed of light—the edge of what's possible on any piece of hardware.
Nvidia has incredible retention. Many people the speaker worked with in 2015-2016 are still there. Engineers are extremely motivated and passionate, believing in parallel computing through its various incarnations. The company is very frugal—no free lunch, and even requiring employees to chip in a dollar monthly to the milk club for coffee milk.
The GPU is fundamentally a throughput machine, happiest when given a lot of work to chew through at peak utilization. However, AI has been pushed toward interactive chatbot use, prioritizing spitting answers out as quickly as possible. This creates a fundamental trade-off between throughput and latency optimization.
The belief is that the most profound change in the next year will be moving from chatbots to proactive or background agents, making it more sensible to build a stack around throughput.
The trade-off between throughput and latency is foundational in computer science—narrow and fast versus wide and slow. For GPUs specifically, batching is key. Grouping many users' work into a batch allows parallel processing, but any individual token or request in that batch spends longer on the GPU being carried with other traffic.
The bus versus car analogy illustrates this: a private car takes a direct path, while a bus must serve many people and takes a slower path that works for everyone.
NVLink represents a major innovation in GPU interconnect technology that allows large matrix multiplications to be sharded across up to eight GPUs. When a matrix multiply is divided and distributed across multiple GPUs, each GPU processes 1/8 of the work, but this does not yield eight times the speed due to sublinear scaling. Communication overhead prevents strong scaling, resulting in approximately four to five times the speed instead. Every GPU becomes slightly less efficient when working on smaller tiles of work compared to larger tiles.
Alternative parallelism schemes such as expert parallelism and pipeline parallelism offer better approaches for most workloads. These methods allow communication latency to be overlapped and hidden in ways that tensor parallelism cannot accommodate, particularly for low-latency server applications.
NVLink is mandatory for low-latency inference applications, as it enables rapid communication between GPUs. However, this technology comes with significant challenges—it is difficult to scale and productionize. Alternative chip vendors that excel at foundational compute operations like matrix multiplies but lack strong interconnect capabilities can still serve valuable roles in stacks optimized for compute per dollar rather than latency.
The optimization focus shifts to evaluating chips based on FLOPS per dollar and operational costs per hour. Other chips may rank higher than Nvidia on this metric despite lacking the interconnect capabilities needed for tensor parallelism.
Companies like Cerebras have made strategic bets on building fundamentally different accelerators that prioritize a unique memory hierarchy. Their approach maximizes on-chip SRAM for extremely fast access to weights and KV cache. SRAM is integrated directly onto the logic die using TSMC's standard cell library, but this comes with severe density limitations—Blackwell's 800 square millimeter die might only contain single-digit gigabytes of SRAM storage.
DRAM offers dramatically higher density through a fundamentally different manufacturing approach using capacitors rather than transistor cells. DRAM requires constant refreshing every 50 milliseconds, managed by memory controllers, but achieves orders of magnitude higher capacity. Stacked DRAM around Nvidia logic dies can provide hundreds of gigabytes, compared to the 500 megabytes of SRAM on the logic die itself.
Cerebras circumvents the 800 square millimeter reticle limit by connecting every die across an entire wafer via scribe lines, achieving approximately 50 gigabytes of SRAM per wafer. Stacking multiple wafers enables up to a terabyte of extremely fast memory, delivering 21 petabytes per second of bandwidth—orders of magnitude higher than HBM's 10 terabytes per second.
The KV cache represents dynamic conversation history that grows with usage and often exceeds model weights in size. Unlike static weights, KV cache requirements are unpredictable and depend on user volume and conversation length. Models trained primarily on 8,000-16,000 token conversations struggle to maintain intelligence at 200,000 token contexts despite some extended context training.
Cerebras-type chips function best as accelerators paired with traditional GPUs that provide off-chip memory capacity. The MLP layers containing world knowledge can reside on high-speed SRAM chips, while attention layers handling dynamic conversation adaptation benefit from GPU memory capacity for long contexts.
Transformers excel at learning unsupervised sequence data by finding patterns through the attention mechanism, which dynamically reweights input relevance for each prediction. Unlike previous approaches, transformers make no human priors about data structure—they simply identify patterns through massive parameterization.
The architecture scaled from computer vision's 150 million parameter models to trillion-parameter systems by efficiently utilizing increased compute and data. Transformers represent any pairwise relationship between tokens, providing complete modeling flexibility until more selective attention mechanisms are developed.
Overparameterization followed by compression represents the core learning strategy: first prove relationships exist by overfitting, then compress to achieve generalization.
The internet provided a one-time data subsidy of approximately 30 trillion high-quality tokens, with broader definitions reaching 300 trillion tokens. Models have exhausted human-generated internet data, shifting focus to model self-improvement through reinforcement learning environments.
Random human preference data no longer provides valuable signals since models exceed typical human capability. Expert human preference and verifiable task environments now drive progress. Coding and mathematics problems offer self-grading systems where models receive feedback on task completion.
Frontier labs increasingly allocate more resources to RL environments than traditional data collection, recognizing recursive self-improvement on verifiable tasks as the path toward artificial general intelligence through stacked specialized intelligences.
Modern deep learning depends on Flash Attention kernels, but new model architectures require kernel modifications when embedding schemes change. Kernel fusion combines operations like matrix multiplication and addition to avoid DRAM write-read cycles.
Current kernel development involves whiteboard conceptual design followed by natural language descriptions to models that generate the actual GPU implementations. This approach may become fully automated within six months as models improve at kernel engineering tasks.
Kernel engineering focuses on optimizing the performance of existing hardware chips. The frontier models set the baseline capability, with everyone else working downstream of those capabilities. Historical energy efficiency improvements provide context: the percentage of usable energy extracted from coal increased from roughly 10% to 95% over time, demonstrating the potential for similar efficiency gains in compute utilization.
Current GPU efficiency sits at approximately 70-80% of peak utilization when performing large-dimension matrix operations. This efficiency is limited by power and thermals rather than software constraints. Nvidia's quoted peak FLOPS figures are optimistic because power throttling prevents reaching theoretical maximums.
Modern GPU programming requires thinking at the rack and cluster level rather than individual GPUs. Nvidia has begun shipping complete rack systems called NVL 72, containing 72 units of the Grace Blackwell 300 chip. This represents a fundamental shift toward programming entire data centers as unified computing systems.
The competition centers on who can program rack-scale computers most efficiently. Nvidia currently leads in both lowest latency and highest throughput for these systems.
The market for bleeding-edge chips like Blackwell resembles a commodity market with extreme scarcity. Nvidia takes a strategic long-term approach rather than simply raising prices to clear the market. They recognize that compute is power today and carefully allocate chips to avoid concentrating too much capability with single customers.
Relationships and financial backing determine access to chips. New startups struggle to secure large orders because suppliers need assurance of payment capability. This creates barriers for companies without established connections or substantial backing.
AMD chips offer significant opportunities because the perception of inferiority creates pricing advantages. The reality is that AMD chips are not inherently inferior, but fewer people understand how to program them effectively. This creates alpha for teams willing to invest in optimization.
Major companies including Meta and OpenAI have purchased substantial quantities of AMD chips. New chip companies like Etched, Senova, and DMatrix are emerging, though they face challenges securing adequate wafer allocation from TSMC.
The approach involves making any chip work at the right price point. This requires building systems that can utilize multiple silicon architectures and aggressively optimize performance across different hardware types. The willingness to build infrastructure around new chips without being constrained by existing data center limitations provides competitive advantage.
Semiconductor companies now represent 19-21% of the S&P 500, up from historical levels of 2-4%. Investors worry this represents a bubble that will revert to historical norms. Memory stocks like Micron and SK Hynix have seen significant gains, but concerns persist about long-term sustainability.
The key difference from previous technology booms is that current AI consumption is not speculative. Unlike the networking equipment investments of the late 1990s that anticipated user demand that never materialized, token consumption today is immediately valuable. Users purchase tokens because they provide direct utility rather than speculative future value.
Training spending was inherently speculative, focused on building models without guaranteed returns. Inference spending follows different dynamics because it's tied to immediate, measurable value creation. Inference spend increases monotonically without the speculative component that characterized training investments.
Traditional data centers were designed for training workloads, which require concentrated high-bandwidth networking. Inference workloads can operate effectively with distributed smaller data centers. A diseconomy of scale exists in data center construction, making gigawatt-scale facilities extremely difficult to build in the United States.
Power availability follows a hierarchy: gigawatt data centers are nearly impossible to construct, 100 megawatts is increasingly difficult, 10 megawatts sits at the edge of possibility, while 1 megawatt facilities are relatively plentiful. This distribution favors inference workloads that can operate across multiple smaller locations.
Liquid cooling enables high power density, with one megawatt of compute fitting into approximately eight refrigerator-sized racks. The strategy involves purchasing any available chips globally for any duration and operating them in small distributed data centers optimized for inference workloads.
This approach accepts lower uptime guarantees, potentially 95% instead of the 99%+ expected from traditional data centers. The system design accommodates failures by moving workloads to other locations, with customers accepting occasional latency increases in exchange for superior economics.
Solar and wind power represent untapped resources for data centers due to their intermittent nature. The approach involves tolerating extended outages measured in days or weeks by modeling weather patterns and shifting workloads to other locations when power becomes unavailable.
Cheap chips reduce concern about capital costs for idle hardware, enabling acceptance of power sources that traditional operators would reject. This creates access to power capacity that competitors avoid due to reliability concerns.
The business model focuses on scavenging both chips and power sources that larger competitors overlook. Rather than competing directly for concentrated supply, the approach involves aggregating smaller, less desirable resources into an economically competitive system.
The founder perspective favors maximum vertical integration, owning everything from power generation through chip design to final token delivery. The CEO perspective emphasizes pragmatic approaches, starting with software leverage and gradually earning the right to control more of the stack as scale increases.
The strategy targets the largest compute market in computing history, with trillions of dollars flowing into inference infrastructure, creating opportunities for custom solutions throughout the technology stack.
The focus on inference efficiency drives the development of custom solutions. The approach involves identifying opportunities where custom hardware makes sense, then working with partners to build these solutions. If partners cannot deliver, the work is done internally.
Compute scaling is already highly efficient—modern models activate only 10% or less of possible experts, with frontier models closer to 1%. Attention and memory usage represent the primary inefficiency, particularly the KV cache which stores excessive data per token. The entropy analysis suggests current KV cache storage is off by an order of magnitude or two from optimal.
DeepSeek has published work on KV cache compression, demonstrating order-of-magnitude improvements annually. The broader inefficiency lies in compute orchestration across the world, where many GPUs sit idle in private pools or warehouses rather than being utilized effectively.
Doubling chip production would create cascading bottlenecks beyond TSMC capacity. The key insight involves fabs' rigid process controls that maintain tight characterization between best and worst chips. There's potential to accept greater variance in chip quality—taking more rejects and working with chips that fall outside tight process corners—in exchange for reduced manufacturing costs and increased supply.
The strategy involves dramatically expanding power supply across the United States to create demand for chips that wouldn't otherwise justify data center placement.
The approach emphasizes long-term thinking over immediate efficiency, focusing on where systems could be in 6-12 months rather than accepting current machine states as fixed. All bottlenecks are characterized and documented to inform both vendor feedback and future purchasing decisions.
Hiring criteria prioritize curiosity above technical experience. The inability to teach love for performance engineering—specifically the drive to understand every microsecond of machine operation—makes this trait the primary hiring filter. CUDA or AI experience are considered red herrings given how rapidly these technologies evolve.
Frontier labs pay premiums for 3-6 month leads, which makes business sense given their current position. The distillation debate is reframed: since an increasing percentage of internet artifacts are AI-generated, latent distillation becomes inevitable. GitHub repositories created by AI coding tools represent one example of this diffusion.
Enterprise adoption patterns suggest the premium for being 3-6 months ahead may not persist, as enterprises move slowly and many remain on older model versions. The fundamental impossibility of preventing capability diffusion means the question becomes whether scaling laws hold indefinitely.
The goal is abundant tokens and diverse harnesses, enabling every company and user to build their own agent through in-context learning rather than weight fine-tuning. The approach involves using every chip, power source, and suitable land in the United States to achieve this abundance.
Current thinking treats agents as expensive consultants to consult sparingly, but abundant intelligence should be available without cost constraints limiting access.
The contrarian position centers on extreme memory offloading, particularly moving KV cache to flash memory rather than being constrained by HBM shortages. This involves redesigning model architectures to support offloading while maintaining acceptable serving speeds of 1-10 tokens per second.
The view that Nvidia faces fundamental limits in performance per watt improvements challenges assumptions about process node advances. From Hopper through Blackwell to Rubin, and across TSMC's 5nm to 2nm nodes, performance per watt gains are smaller than commonly assumed.
This leads to a contrarian geopolitical stance: losing access to TSMC would create supply shocks but wouldn't be catastrophic, as Western processes like Intel's are at most 2x worse in performance per watt—smaller gaps than typically portrayed.
Success requires deep understanding of the 3-5 critical supply chain bottlenecks: TSMC wafer capacity, HBM capacity, advanced packaging, and power availability. Companies must articulate specific strategies for arbitraging around each bottleneck rather than competing directly with Nvidia's balanced approach.
The most promising attack vector is HBM shortage, given memory manufacturers' reluctance to invest in cyclical capacity expansion. The world will respond by making everything else more expensive rather than solving the fundamental shortage.
Nvidia deliberately avoids competing with customers by not selling tokens directly. This preserves goodwill and maintains a diverse ecosystem of NeoClouds and inference providers who compete to drive Nvidia demand. Jensen Huang's approach of making friends billionaires creates a resilient customer base that would fill any gaps if individual players vertically integrate or switch vendors.
The most valued mentorship came from professors who emphasized understanding the entire technology stack—from gate-level silicon through internet-scale services—rather than specializing in single components. This perspective continues to influence the approach to system design and optimization.
Keep Invest Like The Best in your library
Save the videos and channels worth coming back to, and find them again in one place.
