Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper

Name: Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper
Uploaded: 2026-08-25T12:00:09.000Z
Duration: 1 h 23 min 36 s
Channel: Invest Like The Best

Invest Like The BestAugust 25, 20261h 23m

Topics45

Token Cost as the Core Mission0:00 What Sal Research Builds0:32 The Theme of Abundance1:30 Token Cost as the North Star2:01 Why the Opportunity Exists2:31 Long Horizon Tasks as the Future4:00 Test Time Compute Scaling5:30 Market Share Projections6:02 Background Task Examples6:31 Dream Use Cases9:00 Non-Verifiable Tasks12:31 The Master Plan: Software, Hardware, and Power13:00 Tensor Cores Explained13:31 Nvidia Culture and Retention17:00 Throughput vs Latency Trade-off18:02 Why the Trade-off is Unbreakable19:00 GPU Parallelism and Interconnect Strategies20:47 NVLink's Role in Latency Performance22:01 Cerebras and Alternative Memory Architectures23:31 KV Cache Limitations and Hybrid Architectures28:01 Transformers Architecture and Scaling Properties32:31 Data Evolution and Self-Improvement36:32 Kernel Development and Software Optimization39:01 Kernel Engineering and Efficiency Optimization42:10 The Shift to Rack-Scale Computing43:30 Chip Market Dynamics and Allocation44:30 Alternative Chip Markets and AMD Opportunities46:00 Heterogeneous Computing Strategy47:32 Investor Concerns and Market Valuation49:00 Training vs Inference Spend Patterns50:30 Data Center Evolution and Power Distribution53:00 Physical Scale and Distributed Architecture55:00 Power Source Innovation and Intermittency Tolerance59:00 Scavenger Strategy and Vertical Integration1:00:00 Custom Hardware for Inference1:03:14 Ranking Inefficiencies in Token Production1:03:31 Fab Capacity and Process Optimization1:06:01 Building a Performance-Focused Organization1:08:01 Closed vs Open Source Dynamics1:10:30 Vision for Abundant Intelligence1:13:01 Divergent Technical Views1:14:30 Consensus Challenges1:16:31 Advice for Compute Startups1:19:01 Nvidia's Strategic Positioning1:20:30 Mentorship and Full-Stack Understanding1:21:30

In a Nutshell

Ex-NVIDIA engineer is building Sal Research to deliver the industry's cheapest AI tokens by optimizing for throughput over latency, targeting background agent workloads that run for hours or days rather than interactive chatbots. The core thesis is that making intelligence 1000x cheaper unlocks unbounded token consumption in proactive agents, using every available chip, power source, and distributed data center in the US. Key insight: the shift from human-in-the-loop to background processing removes attention constraints, making throughput-optimized stacks and scavenger strategies for underutilized chips/power economically dominant.

AI-Generated Notes

These notes were generated by AI and may contain inaccuracies.

The speaker's job is to make tokens as cheap as humanly possible through every layer of the stack. This includes using every chip, every source of power, and every piece of suitable land in the United States. The goal is to move away from treating agents as expensive consultants that you only consult for hard questions. Instead, the aim is to get intelligence into as many hands as possible.

Sal Research operates as a token factory with an API where anyone can send requests to use open source large language models for any task. The service delivers tokens at prices that are unbeatable in the market. The company also supports building agents on top of this infrastructure by hosting what they call sandboxes—long-running agent virtual machines in the cloud designed for agents that run for hours, days, or weeks.

The core positioning is being a peer company to others that serve different kinds of inference, with the goal of being the absolute cheapest provider and enabler of a certain kind of intelligence use.

The company's theme is abundance—delivering intelligence as a new commodity to as many people as possible at costs sustainable for almost every industry. The belief is that making something 10 times cheaper creates a new product category. The aspiration is to do this for tokens. The profound shift is that machines can now think, and the job is to make as many machines as possible work toward thinking.

Token cost is the current north star, with the goal of having the lowest cost per token in the industry by a significant margin. While tokens are not considered the final unit of work or intelligence, they are what is used today. The speaker expects to move toward outcomes as the unit of measurement. Currently, when consuming tokens through an agent, users do not control how many tokens the agent uses for reasoning. The vision is for agents to self-administer token budgets as opposed to companies setting budgets for how many tokens engineers can spend per month.

Two tailwinds support the company. The first is the rise of open source. Customers increasingly care about owning intelligence and having control and sovereignty over what they depend on. This has created a robust market for customized models or vanilla open source models where users always have the weights and the right to deploy them however they like.

The second factor involves market positioning. Companies like Base 10 and Fireworks have focused on low latency inference, driven by one important customer: Cursor. While that was the right choice a year ago, six months ago it became clear that low latency was not the only requirement for agents. Agents need more persistence and support for long horizon tasks.

Sign in to read the full notes

Get access to AI-generated notes, topic timestamps, and more.

Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper

Token Cost as the Core Mission00:00

What Sal Research Builds00:32

The Theme of Abundance01:30

Sign in to read the full notes