ElevenLabs' Mati Staniszewski: How Voice Becomes the Interface for AI
In a Nutshell
ElevenLabs, founded in 2022 by childhood friends inspired by poor Polish dubbing, builds frontier audio AI models—including text-to-speech, speech-to-text, dubbing, voice agents, and music—achieving >$400M revenue with a flat, remote team of 400+ by prioritizing rapid launches, global talent, and data annotation from 1,000+ experts for long-term emotional intelligence. Key applications span customer support (e.g., Deliveroo, Deutsche Telekom), citizen services (Ukraine gov), and education (interactive Feynman/Ramsay agents), with viral moments like dubbing world leaders and celebrities in new languages. Founders' lessons: keep teams <10 people, embed engineers everywhere, focus on domain-specific models, ecosystems (20k+ user voices), and full-stack integration beyond raw research for defensible moats.
These notes were generated by AI and may contain inaccuracies.
ElevenLabs started in 2022 with childhood friend co-founder P (Polish names are complicated). Met in high school, became best friends, took same classes, traveled, studied, and worked together. Still best friends today. Inspiration from Poland, suburbs of Warsaw: foreign movies dubbed in Polish use one single voice for all characters, male or female, often monotone, forcing viewers to interpret emotions themselves. This persists today for most content. Highlighted need for everyone to speak any language with same emotion and intonation.
Problem spans audio domain: narrating content, books not in audio form, news articles, language barriers, future humanoids/robots where voice is primary interface.
ElevenLabs builds frontier models for audio without billions in funding. Started in 2022, year of crypto/metaverse, pre-AI boom. Audio was niche with few researchers. Advantages: excited about future, others undervalued it, audio models smaller (less compute), big data needs but solvable via transcription/annotation, architectural innovation.
Co-founder top researcher, assembled best audio talent. Untraditional: started in London with Warsaw presence, fully remote to hire globally via GitHub scraping, sharing samples. Launched product quickly, monetized early for revenue to fund models, stayed healthy margins for independence. Later raised external money as ambitions grew. Niches remain untackled for startups.
Started with text-to-speech understanding context for right emotion/intonation (happy sentence sounds happy, dialogues pronounced naturally). Expanded to break language barriers: dubbing requires transcription, translation, text-to-speech. Added speech-to-text. Combined into localization/dubbing/orchestration/voice engine for agents with turn-taking.
Added real-time streaming for interactive experiences as reasoning models improved. Tackled hardest: music production. Spans full audio research: text-to-speech, speech-to-text, combinations, localization, voice agents, music.
First: replicating speaker's voice (with weird accent) from sample—initial denial, then side-by-side confirmation. Second: laughter, pauses making experiences human—topped Hacker News as first AI that laughs.
Viral 2023-2024: Javier Milei speech translated to English retaining voice; Narendra Modi, President Zelenskyy; Matthew McConaughey's newsletter in Spanish/Portuguese (first time family heard him in their language).
Recent: emotional intelligence in voice agents—detects user stress/excitement/speed, responds reassuringly/matching/slowing. Audio general intelligence: combine narration/singing in one continuous voice stream.
Customer support common (phone tree replacement). Shift to revenue: sales (inbound/outbound amplification). Examples: Deliveroo voice agents call restaurants for opening times to update riders/drivers/orders; Deutsche Telekom inbound inquiries via voice vs. forms—simpler, captures more info (use cases, issues) for better follow-up. ElevenLabs uses internally.
Overlooked: citizen support (gov access: taxes, travel policy); Ukraine gov app voice agent for frontline info, education lectures, safety amid war. Education: 24/7 incredible teachers (Karpati, Richard Feynman) for physics etc. Masterclass interactive: Gordon Ramsay teaches cooking (shouts in kitchen); Chris Voss teaches negotiation by live practice.
400 people, >$400M revenue. Keep teams <10 people (research, product, GTM, ops, talent). Flat structure, ~10 direct reports max. Non-technical teams (people, GTM, legal) embed engineers for automation/upskilling/VIP coding review/security.
Examples: hiring/recruiting scraping/analysis; legal scoring system for negotiations (indemnity, liability cap, clauses)—points per customer size, automated. No titles—optimize for impact, tenure irrelevant. 4-year-old company.
Early: order-taking/pricing, not true negotiation. Startups: event organization (call venues for prices, renegotiate with budget). Emotional intelligence key (delivery, pauses). Agent interruptibility needed. Hackathon: agents detected each other, switched to efficient non-voice transmission. Future: voice vs. other depends on infrastructure.
Voice essential for devices/robots instruction/interaction. Bottleneck: communicating with intelligence—voice/visual unlock value. Human-human interaction value increases (events, artists). Trust critical: authenticate real humans/agents (encoding/decoding/watermarking). Future: detect authenticated real, assume fake by default.
Bad: emotional interaction (doesn't get emotion well, too slow); music (good production, not top charts even with artist input—changes in 1-2 years).
Training prioritizes long-term impact/revenue correlation, not short-term. Label data extensively (what/how of audio: emotions, voice description, music)—assembled team of 1000+ people.
ElevenLabs assembled a team of over 1,000 people—including voice coaches, musicians, and artists—to annotate data behind the scenes. This will not provide value in the next 6 to 12 months but is expected to in 12 to 24 months. Collecting such data is frequently not accessible.
Question: From the model AI perspective, what are the modes with audio models? Where are the labs going, not going? In making a real good frontier audio model, what are the main defensible parts?
Response: ElevenLabs develops a variety of models. Recently met Jensen, who commented that speech-to-text models are technology and text-to-speech is artistry: "we are all artists." Gained a client for life.
To fix text-to-speech and emotionality, focus intensely on that space: get in front of users, collect data and preferences, fine-tune models, and apply domain specificity (healthcare differs from financial services, education, or experiences) for production.
Continuous advantage comes from caring about quality and spending time on model work. However, in many use cases, models are just a small part of the stack.
Beyond research, on the product side:
- Understand the user's problem and workflow.
- In voice agents, combine audio models with knowledge inside the system.
- Integrate with telephony systems for cross-channel interaction.
- Evaluate, test, and monitor.
Build ecosystems: distribution and trusted brand, pre-existing work like templates for agents or creative workflows, and over 20,000 user-contributed voices across languages, styles, and voices. This caters to diversity, eases starting, and supports workflows.
Handing back to Kazine Mi. Thanks to Andrew for being a partner.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
