Baseten CEO Tuhin Srivastava on Custom Models, and Building the Inference Cloud
In a Nutshell
Baseten, the AI Inference Cloud, has scaled 30X in a year to over $1B in projected revenue, powering 95% custom models for AI natives like Abridged and Cursor amid severe compute shortages managed via 90 clusters across 18 clouds. Inference dominates as the "last market" due to Jevons paradox—cheaper intelligence drives explosive demand for agentic workflows—while open-source models (including top Chinese ones like DeepSeek at 20% cost) enable companies to own specialized post-trained models using proprietary user signals. Key challenges include capacity crunches requiring multi-year GPU contracts, sticky software layers for 400% NDR, and building an inference-training loop with sandboxes and runtime optimizations for long-horizon agents.
These notes were generated by AI and may contain inaccuracies.
Elad and host interview Tuhin Srivastava, founder and CEO of Baseten, the AI Inference Cloud. Topics include capacity constraints for AI compute, why inference is the last market, changing workloads, open-source and multi-chip future, and 30X scale in a year.
Baseten grown 30X over the last year, expecting more than a billion dollars in revenue this year. Over last 24 months, realization that AI can be put everywhere. Options from closed-source to open-source models. Open-source models crossed chasm in baseline capability. RL techniques and post-training for specialized models mainstream with examples. Customers realizing they can own their inference.
Long tail of models coming through, customers in-housing intelligence. Application layer growing massively; Baseten indexes on that demand.
Existential question: does independent application layer exist versus labs? Application layer exists because value to companies is user signal only they can gather, encoded in workflows rather than models.
Example: Abridged, ambient scribe used by physicians in almost all US hospitals. Deep integration into clinician workflows and EMR. Frontier model companies lack access to that user signal. Companies with signal can post-train models on reward signal, build long-horizon agentic models.
Another example: support tasks at companies like Baseten involve 1-20 actions per ticket, allowing specialized models.
Two versions: new companies like Abridged, Decagon using AI, selling to customers; enterprises building in-house. By inference count, 99% is enterprise fall yet to come online. Majority of enterprise adoption ahead.
Transition happening: enterprises using AI tools, then closed-source APIs, then custom model adoption.
Majority of Baseten customers today: AI natives like Abridged, Cursor, Open Evidence at scale. Learn by building with highest scale customers pushing technologically.
Build for highest scale, like Stripe evolution serving enterprises after pulling frontier. Also build for companies serving enterprises: Abridged, Open Evidence, Decagon, Writer, Gamma. Get translation of enterprise requirements: data retention, deployment locations, GPUs, latencies, transparency.
Serving these prepares for healthcare enterprises like latent health.
Evolution: Mistral, then Meta Llama, misperforming models, Chinese origin. Customers (fastest-growing AI companies) want frontier models: GPT OSS, Moonshot, DeepSeek, Canopy, Orpheus (text-to-speech). Start with capability, then optimize cost. Baseten has visibility to run them well.
Chinese models fantastic, teams awesome. No evidence of agendas or biases crossing network boundaries beyond early models caught quickly. Important for US to develop own models; currently five labs in China creating open-source vs US struggling.
DeepSeek moment: treat as if from Meta. Two scenarios: America fails at open-source (fundamental problem) or succeeds and must be ready.
Chinese government subsidizing models, passing surplus to US enterprises indirectly.
Economics: DeepSeek runs at 20% cost of Anthropic models with comparable/better latency, reliability. Losing access massive loss; cheaper intelligence enables more embedding.
State-of-the-art frontier still closed-source: Anthropic, OpenAI, Google.
95% tokens on custom models. Three businesses: dedicated inference (custom, customer SLA), shared inference (shared endpoint/SLA), training. Almost all dedicated: customers modify models with own data, specialized; compile differently for quality/performance, not vanilla open-source weights.
Acquired PaaS (former customer doing post-training on Baseten) for research capability. Baseten infrastructure/product people; market moving to post-training. Accelerates market, gets closer to customers earlier.
Demand high for post-training software loops/expertise. PaaS team (Australians) working with customers. Links inference/post-training: quantization timing affected by training; paired problems.
Inference begets post-training via data/evlas/reward functions, closing loop.
Prove with best-in-class model you have something worth optimizing. No post-training pre-product market fit. Work with at-scale customers with user signal/value. Then optimize better/faster/cheaper; specialized models fit niches (e.g., support over coding).
Severe supply crunch; little slack compute. Baseten runs large clusters at mid-90s utilization. 90 clusters across 18 clouds worldwide. Technology unifies runtime fabric across clouds for reliability, latency, failover.
Enables onboarding new providers/countries in half a day with full stack. Daily 4 PM meeting to manage capacity for demand.
Suppliers operationally crunched; few reliable (Baseten runs more than most, 3-4 in gold tier with redundancy).
Can buy capacity years ahead (e.g., January 2028), but market moves fast. H100 (4.5 years old) price still rising, useful life ~9 years. Term lengths up: 1,024 B200s requires 3-5 year contract, 20-30% TCV prepay.
Need demand to fill + low cost of capital. Impacts public timing; demand for public infra companies. Interesting working capital/financing; exploring debt.
GPUs-as-service not sticky/commodity. Inference with software layer incredibly sticky: top 30 customers never churned, 400% annual NDR.
Software layer key value.
Access to inference computers is a strategic advantage. Labs pursue owning compute: if we have all the compute, good luck running inference. In a world of constrained compute, the number one thing to own is compute. Owning it as an asset is underappreciated. You can't make good hot chocolate without milk—unless you're vegan. No one wants vegan inference. People might want alternative milk: H100 is great, but they want B200, GB200, tons of NVIDIA.
Several years out, is there a multi-chip world? Diversification everywhere: want a world of many models, many most things. Everyone would be sad if it didn't happen. There will be inference-specific chips, decode-specific chips. NVIDIA mentioned this; Grok LP thing. People underestimate NVIDIA's supply chain, CUDA, developer ecosystem. As an infrastructure company, moving fastest with NVIDIA today. Given NVIDIA's scale, hard to see competition in next couple years. Other players tie up supply with one buyer, hard for ecosystem to form. Big labs with proprietary deals get 90% supply, push for 95%, everything built for them—no one else can use it.
Reacting to market: code agents, long horizon agents big deal; CPU compute, video inference different; sandboxes. For Baseten, runtime stuff critical: chips run on, workloads supported. Get good at diffusion transformers. Coding agents need sandboxes. New speculation techniques for faster imprints. KV cache away routing. Seeing massive gains at runtime level.
Create loop between inference, push training—begets more inference. Partner on everything: best eval companies like Braintrust integrated into/around Baseten. Partner with All on the sandbox, build best sandbox experience. Best training APIs for continual learning as solved problem, not discrete. Core Baseten thesis: build that loop. Ensure biggest possible: access to compute, infrastructure, own compute. Primitives off that: incredibly marginal creative for us/customers. Sandboxes, async batch inference drive utilization with first-class batch experience. Inference cloud: very good at inference, do tangential things/loop into inference, partner/build when necessary. Own core inference story, unblock supply, create margin, go off stack to unlock value.
Surprises at scale: scale limitations with hyperscaler products (AWS, GCP)—not infinite scale. Edge cases become real at big scale. Examples: kernel panic from fluent bit worker creating too many logs into one node—happened to two terms by two workers simultaneously. Systems/kernel level problems. LLM runtimes immature: KV cache use less sophisticated. Limitations at scale for security, performance. More systems level than LLM specific.
Capacity. Market so big, be as aggressive as possible. Grown ton last 12 months/few months. Answer always: go bigger, faster. Fun, exhausting, uncharted territory. Big one: compute. No world with enough capacity/compute for LLM value next 5-10 years—or invent new stuff.
Learning scaling 30X: brought in talent like Danny, Samir, Stephen Day on technical/go-to-market. Were very flat until 12-18 months ago. Need leaders. Contrary to engineer view: everything overhead. Once said: have engineers instead of salespeople—bad idea. Having leadership team you trust critical. Give people whole problems. If micromanaging, cop-out—don't have right people.
Be very clear optimizing for. Not generic 'smartest, hardworking'. Care about first principles thinkers, work high priority, very kind/nice, collaborative, low ego, no hero culture. If need manager, not right place. Clear rubric makes fits/apparent. Hired amazing people, low unnecessary turnover—clear on what we want.
Operations hard in cloud. Slept with pager under pillow decade. Phone buzzing now. Concern: do people get it? Infrastructure company culture: AWS execs' pagers went off multiple times in 45-min meeting. Get used to it—changes speed. Amir's (co-founder) 7-year-old asks if pager is P zero. Rejects people who don't fit quickly—engineers avoiding pager. Joke: siren in office for incidents.
Jevons paradox: decrease cost of intelligence, demand goes up (price elasticity, availability). Developers insert more intelligence if cheaper—insert more anyway, but hell of a lot more if cheaper. Agents longer running. With inference cost down, run longer, more work, larger end. Compute scales from inference perspective. Customers: start with quality/amount needed or base model. Drive down costs: more intelligence = better UX, answers, experiences, actions, revenue. Inference down begets more. Last market: even with AGI, all left is inference. No 'this answer/action enough'—keeps going long time.
Biggest market: shift from software seats/digitization to intelligence, units of cognition, agentic workflows.
Everything smarter. Doctors better care with tools. Build more software, ton more—not less. Not slowing software engineer hiring. Better tools for consumers. Everybody has own team: agent for doctor, learning, organizing life—concierge for everything. Concierge education: personalized access.
Embrace or extinction moment. Workflow/software companies figure intelligent inserted versions driving user value. Very exciting.
Keep No Priors: AI, Machine Learning, Tech, & Startups in your library
Save the videos and channels worth coming back to, and find them again in one place.
