FFmpeg: The Incredible Technology Behind Video on the Internet | Lex Fridman Podcast #496
In a Nutshell
FFmpeg and VLC power nearly all internet video (YouTube, Netflix, Chrome), built by volunteer open source communities excelling in handwritten assembly (e.g., dav1d's 240k lines, 80% assembly outperforming compilers 62x) for decoding on 3 billion devices, rejecting backdoors, millions in buyout offers, and corporate exploitation to stay ad-free and robust. Key takeaways: Handcrafted low-level C/assembly optimizes every CPU cycle amid stagnant hardware; maintainers (10-15 core) demand excellence via harsh reviews, enabling bit-exact support for obscure codecs through reverse engineering; intertwined projects like x264 democratize pro video tools, preserve archives eternally, and pioneer ultra-low-latency streaming (Kyber) for robots/drones. Future-proof ethos: Passion-driven contributors (including teens) prioritize maintainability, psychovisual quality over PSNR, and royalty-free codecs (AV1/AV2), ensuring multimedia survives 1000+ years as humanity's Rosetta Stone.
These notes were generated by AI and may contain inaccuracies.
Excellent code defines the community regardless of contributor background. Contributors come from diverse, often introverted backgrounds.
FFmpeg is one of the biggest CPU users worldwide. Every sentence about it represents someone's lifetime work, with books written on each aspect. Complexity is inordinate. FFmpeg has 100,000 lines of assembly for all codecs; one codec alone has 240,000 lines. Every cycle matters, running on 3 billion devices decoding video nonstop. 30% of Netflix video is AV1, 50% of YouTube.
Peak video codecs: 79.9% assembly, 19.6% C, 0.5% other.
Tweets showing these facts provoke outrage for two years: intrinsics, compiler auto-vectorization are claimed sufficient, but hundreds of handwritten assembly examples prove otherwise.
Intelligence agencies asked to backdoor VLC; response was firm no, would shut down software instead.
Regrets are attacks on your mind.
Conversation with Jean-Baptiste Kempf (president of VideoLAN, key figure in VLC and FFmpeg) and Kieran Kunhya (codec engineer, FFmpeg contributor, FFmpeg Twitter/X account).
FFmpeg: open source backbone for YouTube, Netflix, Chrome, VLC, Discord, every video/audio platform. Decodes, encodes, transcodes, streams, plays almost any format. Built by volunteers.
VLC: open source media player for any format/platform, no ads/tracking. Downloaded over 6 billion times. Iconic traffic cone logo.
Modern civilization rests on volunteer software obsessed with engineering craft, not fame/money. Open source infrastructure enables billions consuming video daily via invisible machinery.
VLC records VHS via capture cards (Peritel/RCA), controls VCR camcorders. Supports DVD-Audio with custom encryption. FFmpeg supports game codecs, e.g., bit-exact 10-second Star Wars opening from one disc.
VideoLAN conference competition: weirdest file (MKV by Derek with changing frame resolution, aspect ratio, rotation per frame; animated SSA subtitles on black frames). VLC played all.
File that's valid ZIP and valid MP3 simultaneously.
Best worst logo: iconic worldwide, known in India/China. 25% website traffic from "cone player" searches. April 1st joke about changing to caterpillar construction got 10,000 emails opposing.
Distinctive (orange, bright, weird, ridiculous, absurd, hilarious), becomes meme/culture. In 20 years, cones will evoke video player memory.
FFmpeg/VLC ensure archival: in 1000 years, VLC opens surviving videos; cockroaches and aliens see footage via VLC.
Memes claim VLC plays pancakes in DVD drives; tried, doesn't work.
Physical codec: 3D-printed cone with RFID chip playing Star Wars.
FFmpeg underlies 90%+ video workflows (YouTube, Netflix, Chrome, Firefox, VLC). VLC downloaded 6.5+ billion times (likely more).
Pressing play in VLC: from file/stream to pixels/sound.
Stages:
- Access URL (HTTP, file, DVD) to byte stream via OS.
- Demuxer (container) separates video/audio frames from data blocks.
- Parse frames to detect codec (GPU decodable? 45% files not). Probe variants, vendor capabilities.
- GPU: pass through.
- Software fallback: deentropy (Huffman/arithmetic coding removes math compression).
- Intra prediction (I-frames, spatial domain).
- Residual (frequency domain, quantized); inverse transform to spatial, apply residuals.
Video compressed 100-200x. Codecs mimic human ear/eye (YUV: luminance + chrominance, matching cones/rods; scale down color resolution halves size undetectably).
Degrade signal perceptually optimally using frequency domain (DCT), block splits (visible if wrong). Each codec generation: 30% better quality at same bitrate, but 10-100x more CPU/GPU.
Programming in C and assembly.
Full pipeline: address → byte stream → demux (separate audio/video/subtitle tracks) → decode (codecs) → raw → display (graphics/audio cards).
Every stage: thousands of industry workers, books, academic/practical subtleties.
Containers (mux/demux): collect multiple tracks (file format). Examples: MP4, AVI (Microsoft), MOV (Apple→MP4), Matroska/MKV (Steve Lhomme, future-proof).
Common confusion: MP4 (container) vs H.264 (codec). H.264 = MPEG-4 Part 10 (AVC); MPEG-4 meta-spec includes AAC (Part ? audio), Part 2 video.
MP4 typically H.264 + AAC (99%). VLC/FFmpeg ignore .MP4 extension, probe content (real world mismatches).
Extension hints priority (try MP4 demuxer first), but untrusted.
VLC originated as client for 1990s VideoLAN UDP streaming (damaged packets). Doesn't trust inputs for security/broken files.
Popular for pirated incomplete AVI (metadata at end). VLC interprets broken files.
Codecs remove spatial/temporal redundancy using math, 100-200x compression. Asymmetric: encode costly (once), decode cheap (many viewers). Error resilience for UDP (recoverable, join streams mid-way).
To give a more image to people who are not familiar, right? Like, when you're going to see any type of movie, right?
Camera panning reveals background elements like clouds remaining the same for 30 seconds to a minute, allowing reuse from frame to frame. More memory and power enable more comparisons for greater compression. Modern codecs exploit this.
Video consists of grids of pixels (often RGB with three values per pixel) at 24, 30, or 60 frames per second. Compression targets 100x to 1,000x reduction by eliminating redundancy—elements humans wouldn't notice if missing, such as unchanged clouds between frames or uniform black backgrounds.
For uniform areas, reference pixels (e.g., top left and top right) can represent the whole without repeating values. Frames can reuse elements from previous frames, limited by memory and compute power. Comparing pixels across 200 past frames in 4K requires massive compute. Decompression occurs during playback.
Codec development couples encoding and decoding with distinct trade-offs: heavier compression might complicate decoding; complex encoding eases decoding; fast encoding burdens the player. Multiple codec types exist due to these challenges.
Modern codecs like AV1, AV2, VVC are collections of tools, not single codecs, adapting to content like screen shares, video, or animation (e.g., Zoom switching from PowerPoint to video).
FFmpeg provides low-level libraries for codecs (compression/decompression), muxers/demuxers, and filters. Core libraries enable pipelines to process any video files. Integrated into VLC, Chrome, smart TVs, and online video platforms. Depends on libraries like x264, libvpx.
Philosophically, FFmpeg levels the playing field: home videos and trillion-dollar corporations use the same stack, often with 3,000-line commands or API calls.
Tools include command-line FFmpeg, FFprobe; libraries libavcodec, libavformat, libavfilter. Command-line FFmpeg is legendary for parameters allowing full customization—like a programming language.
Users specify input/output formats, but advanced commands span thousands of characters, sometimes AI-generated due to complexity. Supports numerous filters on the command line.
Everyone uses FFmpeg: YouTube (server/client via Chrome), OBS recording, professional equipment.
Examples: add intro/outro videos with fades (dip to black), cross-dissolve audio (quiet to loud), bake in captions with custom fonts, layer audio/video. Handles any codec seamlessly. Capable of Adobe After Effects-like operations via command line—no equivalent for images (ImageMagick is similar but lacks depth; no Photoshop CLI equivalent).
Vision: make insanely complex multimedia easy for everyone. Drop file in VLC—it plays; complex FFmpeg filters just work.
FFmpeg democratized video: podcast/YouTube/streaming revolutions enabled by it replacing 1990s car-sized, hundreds-of-thousands-dollar equipment.
Video's rise (shorts, Reels, TikTok) empowers individuals worldwide—real freedom.
FFmpeg, VLC, x264, VideoLAN are fully open source. Analogy: open source provides the chocolate cheesecake plus recipe, oven-building instructions, and rights to modify/resell—software as billions of tiny instructions.
2,000–3,000 contributors to FFmpeg since inception, like Linux kernel's 10,000. Communities collaborate online to fix codecs/files for everyone's benefit, working for the greater good.
Licensing allows billion/trillion-dollar companies as wrappers. Cloud providers run open source tools via API; some (Mongo, Elastic) changed licenses to prevent this.
FFmpeg's license is a social contract; changing it requires all contributors' agreement (thousands, some deceased). Enables forks/merges (e.g., GCC/EGCS; KHTML → WebKit → Blink).
Permissive (MIT, BSD, ISC, Apache): few conditions, do whatever (popular for JavaScript, BSD OS). Some require attribution.
Copyleft (share-alike): GNU GPL, AGPL (strong), LGPL, Mozilla Public License, Eclipse Public License. Vary by strength.
Public domain doesn't exist worldwide; licenses use international copyright law.
FFmpeg and VLC mostly GPL or LGPL; Linux kernel GPL, Android Apache; BSD kernels BSD.
Can move from permissive to restrictive (e.g., MIT to GPL) but not vice versa. LibVLC core changed from GPL to LGPL for third-party integration (e.g., phones/tablets, game engines) without forcing full open-sourcing.
LGPL (Library GPL): link into closed-source apps, but return modifications to the library. Enables commercial use/companies.
Apple App Store complicates GPL; easier for LGPL. VLC on iOS/Apple TV uses MPL.
Re-licensing requires contacting all contributors (joint/collaborative works; individuals retain copyright even on deleted contributions). Jean-Baptiste contacted 350+ people 14 years ago—emails, travel, even to factory worker whose son (contributor) died. Explained open source intent; emotionally challenging but done correctly to preserve community.
Some projects re-license aggressively, destroying community.
Diverse: Syrian war zone with part-time electricity, all walks of life. Mostly introverts; code quality defines community—background irrelevant ("maybe you're a dog. I don't care").
Rejects poor patches regardless of credentials (e.g., engineers from large companies).
Linus Torvalds: created Linux (powers Azure 70-80%, Android), Git. Harsh (mostly to maintainers), but ensures high quality. Core teams small: VLC (5 people), FFmpeg (10-15) maintain 1,000+ contributors' code.
There's a 1% chance that someone comes and stays. You will change jobs, wives, have children, accidents in life. You're not going to come back. FFmpeg maintainers are the ones maintaining your code. It needs to be maintainable and excellent. Sometimes rework good code to make it excellent because few people maintain something critical for everyone.
There's spiciness and harshness to maintain high excellence. FFmpeg and VLC are low-level and extremely technical. It's a subculture where newcomers are unknown. In real life at VideoLAN DevDays (VDD) every year, people are fun and love it. Online tone can be harsh, but it's a culture. Gaming culture is harsh too—ways of showing love and respect differ. Book clubs are sweeter; high-stakes open source used by millions is different. Not often insults like in gaming. Linus' tone is unusual even for open source—harsh on results: "No, this is not good. This is crap." Focus on code, not person. Matter-of-fact. FFmpeg developed almost entirely by volunteers. Someone does a hard day's work, comes home tired—terseness isn't personal. Busy but care about open source; can't handhold every detail. Most don't speak English natively, especially Europe-centered projects like FFmpeg and VLC. US people unhappy about tone don't know better. English has subtleties in tone lacking in other languages. Cultural and language differences complicate communities.
JB repeatedly turned down millions to keep VLC open source, free, without ads. It's a meme on Reddit and 9GAG—JB as wizard in VLC hat refusing tens of millions. People tag him; he replies "Good morning" with 24K upvotes, great karma.
Refused dozens of millions several times; could be multimillionaire on a beach. Not moral or right. Works for greater good, people. Not fully legitimate alone. In France, top excellence schools outside university: engineering, business, lawyers, medical. Two years crazy math/physics to enter. École Centrale Paris (name changed) moved south of Paris post-WWII. Alumni nonprofit bought land—campus managed by students: radio, TV, supermarket, library, rooms. Learned running campus at 22 or no electricity.
'80s: IBM/3Com-sponsored token ring network—no routers, ring where message passes neighbor to neighbor. Slow: every computer opens message. OK for Telnet/mails in '80s. '90s video games (Doom, Duke Nukem 1994-95) need low latency. Students demand faster network. University: campus yours, figure it out. See CIO of Bouygues (French company doing TVs): future satellite. 1995 first satellite dish. Instead of one dish/decoder per 1500 students, enormous dish, one decoder, stream video on network—first video streaming. Built hacked Network 2000 project: crashes after 45s (demo 40s), memory leaks, 64MB RAM instead of 8-16. MPEG-2 TS transport, MPEG-2 video/audio. Got 155 Mbps ATM network, best in Europe. Stopped project.
Six months/year later, two students (Christophe Massiot, friend of Kieran and JB) think others care about local network video streaming, create VideoLAN. Not open source yet; 3 years to make school agree (university wanted to monetize MPEG-2 decoders due to IP/copyright). Main application: streaming on local network, 10 years before YouTube. Pentium 60/75, main machine 486DX 33MHz. TV dominant; satellite with dozens/hundreds channels groundbreaking, especially university with nationalities (Maghreb, Middle East—multiple dishes/satellites). VideoLAN has crazy solutions like multicast on unicast. Client became VLC. Strong-armed university; open sourced early 2001. JB joined 2003 at university—not creator, emerged from VideoLAN (student project), continuity of Network 2000.
2003: created open source nonprofit VideoLAN, took everything from university for sustainability. Spent most time on VLC/VideoLAN. 2005: project nearly died, only 2 active developers. JB made it continue—great tech, useful. Grew from hundreds of thousands to millions, now billions of VLC versions worldwide. Contributors from Syria, India, everywhere. Offers: bundle toolbars (spyware, change browser/search), ads inside VLC—refused. Not against money (created startups), but ethically. Shady ad companies only; Netflix integration different story. Would make money, but project dies in 3 years, forked. Dishonesty compromises spirit. Selfishly sleeps happy—right/wrong. Wouldn't betray contributors. Internet thanks—inspiring sacrifice for open source freedom/symbol. 2000s normal: downloads secretly install spyware in faint license text.
Last offer obscene: "Use money for new open source." Mind trick hard, but not right. Takes guts/vision to refuse thousands/millions.
FFmpeg built by volunteers.
Kieran spicy on Twitter celebrating developers/codecs/assembly. Google biggest open source supporters long-term. Some things overboard. FFmpeg homepage: processes untrusted data, security issues normal. Google used AI for security reports on FFmpeg. Volunteers dealt with limited funding. Went to media first announcing AI good before fixes. Public: AI-found vulnerability reported publicly pre-fix, 90-day deadline ignoring volunteer nature. On obscure 1990s game codec.
From Google's view: resources on ubiquitous open source security, compute/expensive researchers—contributing.
Security like architects never on site—day-to-day construction beneath. Aggressive tone: "You will get popped" (hacked). Padlock protects home, not Fort Knox. AI picks locks at scale, says "not secure" without fixing/patches/funding. AI reports wordy (DoS), everything high priority on niche 1990s game codec/disk from 1993. Hobby project safety matches hobbyist view—no big scary "critical." Another: filter integer overflow, wrong pixel color—high 7.5 severity red. Can't cry wolf—leads to password stickers on PC. Modus operandi: scare/fear. Google uses FFmpeg at massive scale (millions CPU cores), contributes to own (VP9, AV1), disproportionate overall despite funding students/Summer of Code. Alex Strange (former FFmpeg dev) on Hacker News: > "The problem with security reports in general is security people are rampant self-promoters (Linus once called them something worse). Imagine you're a humble volunteer open source developer. If a security researcher finds a bug in your code, they're going to make up a cute name for it, start a website with a logo. Google is going to give them a million-dollar bounty. They're going to go to DEF CON and get a prize, and I assume go to some kind of secret security people orgy where everyone is dressed like they're in The Matrix. Nobody is going to do any of this for you when you fix it."
Disproportion discovery vs. patching means. Post-debacle: Google sends patches, reward tools for fixes—changed positively. Other companies demand fixes for critical bugs in products.
FFmpeg tweet: "The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion-dollar corporations expect free and urgent support from volunteers. Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority. After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead. This is unacceptable." Image/details show reality. Think open source = vendor with SLA; public bug tracker = third-party Jira. Heinous: name-dropping Microsoft, visible product.
If this was just a general bug report, it would have made it a lot better. They literally said, "This is a big deal because a lot of people are using it in Microsoft." Companies think of FFmpeg as a vendor that Microsoft is paying a huge amount of money to. Nobody thinks, "Shouldn't we be giving millions of dollars to FFmpeg?"
This is a very big problem in large companies. The person was just a manager on one project in Microsoft Teams. He had never discussed with the open source community. Usually there are OSPOs (Open Source Program Offices) supposed to discuss with open source communities, but they often don't explain that correctly internally. We are not your supplier. If you want me to be your supplier, I'm very happy. I will send you a contract and SLAs. I created five companies doing that around open source projects.
Some of the spicy tweets from Kieran produced positive results. Donations have increased substantially. They're still not enough to cover even a single full-time developer, but on awareness level and technical level, there's substantially more technical awareness and awareness of FFmpeg's importance as a result of X and what's happened. It solved its purpose. People realize the level of importance FFmpeg has.
On VideoLAN it's the same. For more than a year, we couldn't update VLC on Android because of a bug on the Play Store. The only way we got someone to answer was a spicy tweet saying we are going to stop distributing VLC for Android, with around 100 million people using that. Then someone from Android came and discussed with us. We had the same issue with Microsoft, saying we were going to stop distributing VLC on the Windows Store. We are so small that our only strong power to solve issues is blaming on social networks because it snowballs and they listen. Large companies often have difficulty talking to us. VLC is probably one of the top 10 software used on Windows. I'm not part of Microsoft ISV programs. I don't have a point of contact at Microsoft, while Adobe, Spotify do. Raising awareness works. It's sometimes very spicy, lot of drama. X and Twitter are okay for that, but it's efficient.
Everybody listening should follow FFmpeg on X, follow VideoLAN on X. Go donate to FFmpeg.
Thank you, Lex. Over the years, several years you've been a supporter of FFmpeg and VideoLAN on X, giving shout-outs, appreciating what we do.
FFmpeg for life. Tim Sweeney, Carmack, and a few others, very high-level people, have raised awareness on our X accounts, and that helped a lot. Karpathy as well.
Outside of so many people using it, it's impactful on the world, and a great representation of a great open source project. The value of assembly and C and taking programming seriously for real world systems.
It's celebrating people like Andreas Rheinhardt who do maintenance, unpaid as a volunteer. He's doing massive refactorings. Andreas Rheinhardt and Anton Khirnov rewriting ffmpeg.c with threading. Celebrating untold labor that doesn't change anything from the user standpoint. The files are exactly the same, but the airplane has been rebuilt whilst it's in the air.
Christian Garcia said, "As a teenager running this account," referring to the FFmpeg account, and you responded, "Teenagers have written more assembly in FFmpeg than Google engineers." There are incredible contributors who are teenagers. We don't care who you are, where you're from, what you do. Teenagers have written thousands of lines of assembly over the years. Shout-out back in the days to Daniel Kang. Highlighting work of people like Ruikai Peng, a 16-year-old, some of his first contributions to FFmpeg, actually doing and putting security researchers to shame by finding issues and fixing them and being 16. There's no barriers. You don't have to study at college under this person. Learn C from the K&R book. Learn assembly. You can contribute to world-class technologies.
In VLC one of the oldest contributors Felix does everything on Mac and iOS. He started working on VLC at 16. Edward Wong, Google Summer of Code student who stayed for three years around VideoLAN, was 14, part of Google Summer of Code and Google Code-in, programs with students or high school. We wrote a ton of assembly for x264 and for VLC and for FFmpeg. Everyone can contribute.
He did a good job because he didn't play the alarmist CVE game, create a public exposure of security and big scary red 7.5 high priority. He just fixed an issue in Git after three days and fixed it. He didn't need to play a big security drama. The kids are all right. There's a portion of the security community that likes to hype themselves up by creating drama. They would have raised "This is a high priority CVE 8.0" on an issue that was in Git, not even in a release, in development, fixed three days later.
Much love and respect to Google engineers. Some of the best software engineers in the world, they contribute a lot, even on the security front. Big fan of Theo. He was part of this debacle and drama. When you zoom out on the grand arc of human history, the drama contributed positively. Donations went up. It brought more attention, allowed everybody to bicker and figure out what FFmpeg is all about.
It's a rap battle at the end of the day. X is a perfect place for international rap battle. You say stuff, I say stuff about your mama, but it doesn't mean I have an actual personal issue. The Theo situation went a little bit too far. It's just a bit of fun. It's WWE. Everyone's having a bit of fun on X. It doesn't need to be taken seriously. The teenagers thing, that Google employee said, "Hey, there are other ways to run an open source business." Just have a bit of fun. That's the point of this account. If you can teach people about open source projects, assembly, by doing that, there's a lot to be offered. It's not dunking for dunking's sake. It's showing the story. These are not big corporate open source projects like Kubernetes with hundreds or thousands paid to develop. These are people in their basements in spare time. If you can address that in a fun and entertaining way, that's the value of X and the reach.
Even at Google, so many different people. Ton of Google engineers we work with all the time, from YouTube to Chrome to Chrome Media. What we do is efficient. For Theo, it went a bit too far. I calmed everyone down, had him on the phone, said this goes too far. In the end it's a rap battle, but positive for the project. Awareness on true open source from communities has increased dramatically in the last two years.
What motivates the incredible contributors? They're sitting in the basement. What's the driver?
Many drivers, but weirdly the main one is that multimedia plays videos, and video is cool. So many people arrive because they loved watching anime. Advice when people ask, "What should I work on in open source? How do I start?" Work on something you love. I work on VLC because I love movies, watching the same movies over and over.
Because it's interesting, a topic you like, that's where people come to VLC and FFmpeg. Second, we search for excellence, best school ever of programming. If you're good in C in FFmpeg, if you know how to write assembly, you're going to be one of the best programmers ever, even writing TypeScript. You get reviews by the most seasoned programmers who look at every part of your code and tell you why it's not great. Best teachers ever.
Andrew Kelley started Zig. He was an FFmpeg developer and started Zig after his FFmpeg school. Place to learn so many aspects of programming in the real world, in a thing used by billions. Nowhere to hide. Have to be open and honest about flaws and how to learn and be better.
In multimedia you have 16 milliseconds to display a frame. Not like a game engine where you can slow down. You need to be good, else no video. If you miss a frame, you destroy the look because of codecs. Need to be perfect.
Not just pure programming mathematically. Need to understand how computers work. Writing assembly, understand CPU pipelining, SIMD, ALU, IO. Missing in many engineers today: computer architecture. Debates like should we use this assembly call or this one? It's three cycles on this CPU, massive impact.
FFmpeg is probably one of the biggest CPU users in the world, running on order of magnitude 100 million, maybe even a billion CPUs as we speak. Every instruction matters. Impact in terms of CPU is massive.
First you come because interesting subject, stay because excellent, end proud because it's in hands of everyone. So many jobs like consulting company portal to download invoices for PG&E. Not going to tell that to grandma. But say, "I do this so you can play video on your laptop," they understand. Working on VLC, FFmpeg, H.264, in hands of hundreds of millions, you have impact. Can be proud. In addition to great resume, that's why people contribute.
The world is a museum of passion projects. Everything out there is a passion project. Open source multimedia and open source in general, you can do that so much faster. Faster network effect.
I can open a cafe passion project, but building codes, build building, location. In software, passion project moves quickly, amplified by network effect, more than sum of parts. Find people interested in obscure things, make something truly amazing.
Tim Sweeney said in a reply complimenting JB, "Many things in the world only happen because an awesome person decides to do it. This is the case with VLC."
Small number of people, sometimes one person, can create something incredible in software. JavaScript incredible thing created initially by single person. Programming languages like Python, C, Java, one person vision, design, sometimes over a weekend initial spark.
Linus built Git in two weeks. Changed the world. Linus' passion project. "Hey, I'm uploading this tarball to an FTP, deal with it."
Not just in software. Believe in individuals changing the world with good vision. Useful. Whether build train, cars, rockets, people who believe in themselves and have vision can have huge impact for humanity.
FFmpeg and VideoLAN, VLC coexist, no central point of importance. Binary star system. They succeed because of each other.
VLC is to FFmpeg as Android is to Linux. They depend on each other, coexist because of each other.
A lot of FFmpeg pipelines involve x264 project, a VideoLAN project. 80-plus percent of those pipelines dependent on a VideoLAN project. VLC uses FFmpeg, gives it reach, exposure to weird files, historically used some donation money to fund FFmpeg development. Reverse engineering later. Many developers shared. No central location. Virtuous cycle.
x264 is the encoder for H.264 video standard. H.264 standard, x264 open source implementation used by basically everybody for everything. Main driver. MP4 file with H.264 codec, if from software environment like data center, chances are created with x264. Under VideoLAN flag.
VideoLAN has a bunch of stuff. VideoLAN website, bunch of icons: libdvdcss, libdvdnav, libdvdpsi, libvlc, vlc-unity, libbluray, many more. Lately dav1d project, everywhere. libspatialaudio lately announced. checkasm, insane but amazing project. x264 one of those VideoLAN projects.
x264 most amazing encoder ever designed, helped adoption of FFmpeg. Large companies went through FFmpeg because wanted x264, increased popularity. VLC had popularity because played files done by FFmpeg. Many projects intertwined.
On X, when VLC mentioned, people say "quick reminder it's FFmpeg inside doing actual work." Not the case. We work together.
When I compiled VLC for Windows, around 16 million lines of code. One million inside VLC repository, FFmpeg around two million. So many dependencies outside. FFmpeg integrates third-party libraries like x264, LibOpus, many others. We all depend on each other.
Shout-out to people. History of FFmpeg, key figures. Eras of FFmpeg, key eras and people.
Fabrice Bellard creating the concept. 2000 era: Michael Niedermayer. Key things: exhaustive support for DivX and Xvid, all weird variants of MPEG-4 Part 2 (predates MPEG-4 Part 10). 2000s video codecs, flavor after flavor of weird decoders. Needed new player for every file format: Windows Media Player for Windows Media, RealPlayer for RealMedia. Key: native decoders for those. As teenager, figured out one player that could decode without separate bloated players. Downloaded RealPlayer had ads, other stuff. Simple library that was fast.
2008 onwards big change: H.264 maturity, beginning of high definition video. Key decoder.
Beginning single player playing Xvid, DivX, Windows Media, RealPlayer massive achievement without codec packs.
VLC 1.0 out around 2009-2010, where it exploded. Without codec packs, it just works across all codecs, plus other modules.
2000s weird codec packs with DLLs from everywhere, spyware, unreliable.
Task in 2000s Michael did was Sisyphean. Edge cases poor beyond comprehension. Chinese CCTV system weird variant of MPEG-4 Part 2 (MPEG-4 ASP), fix without breaking everybody else, times a million.
Reverse engineering started in 2000s with Windows Media (proprietary), RealMedia with Benjamin Larsson, Kostya Shishkov. Key groundwork.
2010s Paul Mahol, Kostya era, most difficult codecs. JB talk about GoToMeeting 4 and GoToMeeting 5.
Amazing Ukrainian guy Kostya, living in Germany at the time, in love with Sweden. Lot of people in community very clever. He's borderline genius.
He reverse engineered extremely complex codecs. One megabyte binary blob to reverse engineer is an order of magnitude a month of work; this guy does 20-30 megabyte blobs for difficult, obscure codecs.
GoToMeeting was a big problem for VLC, the number one feature request. JB put a bounty. A guy said, "Okay, JB, I'm going to do it," and in two months explained how: "Oh, I looked at the code, like this looked like a DCT that I used to see on WMV and so on." His code is full of jokes, including JB, Kempf, and Kostya references. The code is beautiful.
Developers at assembly language level make everything sound easy due to humility from the high skill required.
In the community, most impressive are reverse engineers and assembly folks. x264 became amazing due to Loren Merritt from University of Washington, who made it great and fast with tons of assembly. This was the golden era.
Kostya views the world as a binary specification, figures it out without documentation. He uses the phrase "binary specification."
For GoToMeeting, record a meeting—how to play back without their player? Find the decompression module in the video conferencing client. Dump YUV data from the module: open in disassembler, guess hooks, run module natively to decode a sample, hook to output raw YUV for bit-exact comparison.
In disassembler, use intuition to find DCT, entropy coding. Patterns exist, e.g., screen codec tools. GoToMeeting has variants 2, 3, 4. Older recorded sessions in WMV files require special decoder; without it, Windows Media Player shows black screen or audio only.
GoToMeeting.exe is Windows 32-bit, but people use Android, iPad, RISC-V, Arm. Such reverse engineering is exceptionally useful for humanity, enabling future support.
Reverse engineering is mind-blowing, like archaeologists with little signal inferring from experience and code structure.
Kieran reverse engineered CineForm before open sourcing. Use samples; find ones with flat blocks (e.g., animation) to start without complex tools. Track by observing sample and machine code: "Ah, I have byte, this byte is six. Take this branch." In different samples, different paths.
For GoToMeeting, two orders of magnitude more complexity. Work in black box: many steps from entropy decoding, intra prediction, motion prediction, IDCT. Debug purely in memory; wrong coefficients lead to rabbit holes. Step through debugger instruction by instruction on tens of megabytes, millions of instructions. Pause VM, dump memory for encryption/DRM.
Jon Lech Johansen broke DVD with DeCSS, then MP4 FairPlay from Apple, debugging Windows in VM—mind-blowing.
People help with samples. VideoLAN tweeted for obscure samples. Sometimes get goldmine: company with 100,000 files for bit exactness testing across coding tools.
Bit exactness: From 2000s, codecs require every implementation to produce exactly the same bits for a given sample. H.264 implementations (Lex's, JB's, Kieran's) must match bit exactly. MPEG-2 in '90s lacked this—a big industry mistake. Yuri Reznik acknowledged it.
FFmpeg is unique: winner-takes-all for multimedia codecs, unlike browsers with multiple engines (Firefox, Chrome).
Yuriy Reznik: PhD computer science Kyiv University, 150+ papers, 80+ US patents, contributed to H.264, MPEG-4, AVC-H.265, MPEG-4 ALS, G.718, G.711 (telco), RealAudio, RealVideo. Worked at Zencoder, Brightcove, Contex. Very nice; gave ideas for JB's startup Kyber at Mile High Video conference in Denver.
FATE (FFmpeg Automated Testing Environment): Tests insane combinations of OSs, compilers, instruction sets (e.g., macOS/iOS/tvOS variants, RISC, PowerPC, DEC Alpha, ARM, Visual Studio, Clang, GCC, Intel, Apple Clang). All volunteer-run; Kieran hosts Macs.
Ensures no miscompilations (compilers sometimes compile C wrong). Logs all compilations/tests; all tests pass for codecs, filters.
Key because local tests miss breaks on specific configs (e.g., GCC 11 on Mac). Small output changes cascade to glitches due to frame dependencies.
Kieran's company Open Broadcast Systems (unrelated to OBS streaming) builds broadcast equipment for 10-bit video between stadiums/studios. 10-bit not native on CPU (use 16-bit, 6 wasted bits); packing formats save 40% bandwidth (e.g., PCI Express). 5x5 or 6x6 matrix of format conversions, handwritten assembly for CPU generations—traumatic.
Assembly 62x faster than C.
VLC/FFmpeg support: Windows XP to 11, macOS 10.7 to latest, iOS 9 to latest (mix Xcode/SDK versions for Arm32 iOS 9), Linuxes, BSD, Solaris, OS/2 (10 users). Small team supports more OSs than Microsoft/Google/Apple.
iOS worst: Frankenstein Xcode for iOS 9. Impacts users with old hardware (e.g., iPad 2).
Assembly: Write processor instructions directly (vs. C compiled to assembly/machine code). SIMD (single instruction, multiple data): Add 5 to vector of 16 numbers in one instruction—perfect for video pixels. FFmpeg avoids abstractions/intrinsics; direct register control yields 10x-50x (e.g., 62x) speedups.
dav1d: VideoLAN AV1 decoder by Alliance for Open Media (Google, Netflix, Amazon, Mozilla). Launched saying too complex for software, needs hardware. Team (JB, Ronald Henrik, Martin) made extremely good software decoder: 30,000 lines C, 240,000 lines handwritten assembly (FFmpeg total assembly ~100,000 lines across codecs). Motto: every cycle matters.
Used in VLC, software AV1 stacks on 3 billion devices (30% Netflix, 50% YouTube video). 720p decode on 1-2 cores without hardware.
79.9% assembly, 19.6% C, 0.5% other. Tweets cause backlash: "intrinsics fine, compiler/auto-vectorization." But handwritten beats by orders of magnitude.
Handcrafted assembly outperforms C by orders of magnitude. Modern compilers have auto-vectorization for SIMD, but it's multiple times slower, not just 5-10% slower.
Karpathy learned assembly from the tweets and documented his work.
Philosophically, hardware speed increases have ended with Moore's law. Limitations in AI and memory require going down the stack to optimize. Requests for CPU, GPU power are exploding while hardware speed stagnates. Solutions include adding more cores (up to 250), taking every inch of the machine, and abusing it in unexpected ways, like using cryptography instructions in video processing.
In dav1d, they don't use the operating system's function calling convention to save registers and CPU state when calling functions. dav1d creates its own calling convention for speed, bypassing standard rules since calls stay within the library.
Loren Merritt created a lightweight abstraction layer for assembly to handle calling conventions across x86 variants: Linux 32-bit, Windows 32-bit, Windows 64, Linux 64. dav1d goes further with custom conventions internal to itself.
Custom conventions aren't special to each OS but must handle different instruction sets: RISC-V, ARM64, SVE, SME, x86 AVX-512, AVX. Runtime processor detection sets function pointers based on capabilities, from 2008 laptops to RISC-V machines.
By not respecting OS calling conventions, they avoid saving/loading registers to L1/L2 cache, gaining speed. Understanding CPU architecture is key; this handwritten approach is unique to dav1d, called an art by Kieran.
Unlike high-frequency trading on specialized hardware, dav1d runs on billions of devices across all hardware. Counterintuitively, huge value exists in assembly for performance.
Programmers in C/C++, Python, JavaScript, vibe coding with natural language need intuition: go low-level for more power per dollar. Like LLM quantization (FP8, FP4, Microsoft Phi in 1.5-bit) constrained by memory/hardware, real-time processing demands optimization. dav1d runs billions of hours per day; optimize mission-critical hot paths, not glue code like FFmpeg CLI.
Kyber focuses on ultra-low latency where every millisecond counts. Hardware constraints (cost, power, CPU/RAM/networking) return; AI handles business programming, but hardware optimization resists vibe coding.
Traditional assembly teaching is grammar-focused, listing every instruction (even irrelevant ones), like learning language structure before use. Instead, learn by solving real problems, like asking names to communicate.
Assembly in FFmpeg differs from embedded devices (low-power). Requirements: high school math, C (really just pointers). High schoolers like Daniel Kang contributed to FFmpeg via these lessons.
asm-lessons starts with basics, explains jargon/syntax, vector-focused, covers registers (general purpose, vector). Uses FFmpeg-style examples. Some assembly is beautiful, like flying a Spitfire—pure aviation, pushing beyond design with cryptography instructions.
Person-to-person teaching (like blacksmithing/wizards) doesn't scale. LLMs understand somewhat but hallucinate modifications, fail bit-exact tests; lacks training corpus like Stack Overflow.
Speaker started with Itanium (Intel/HP 64-bit, lost to AMD64/x86-64). Itanium had high FMA/float power (6B ops/sec) but memory bus limited to 1.5B, requiring crazy register reuse/packing no high-level language handles. Intel books full of jargon.
asm-lessons target real problems. Students submit patches. Includes x86inc framework by Loren Merritt (from x264), abstracts calling conventions; enabled x264 contributions.
Learn assembly to understand computers: registers, L1/L2/L3, RAM, SSD, disk—builds better programming culture.
Kieran and speaker differ. Value in memory safety, but can't match handwritten assembly speed (nor C potentially). Rust has Esperanto/Sinclair C5 vibe—utopian, self-important, not solving real problems as well/better than existing.
Rust rewrites hit 85-90% features; last 1% takes 99% time (Elon: prototypes easy). Needs flawless unit testing, stable ABI, multiple compilers. No objections to Rust in FFmpeg if equal/better.
Speaker added Rust modules to VLC (small core, many modules in C/C++/Objective-C). Kyber (startup, open source) mainly Rust. Rust is better C++ for memory ownership; great for new projects, parsing/networking (memory/boundary checks). Poor for interop with existing code.
Rust community pushes rewrites; advice: don't rewrite. Writing code 10x easier than reading/understanding complex code (undocumented wisdom, business logic). Even author's best code unreadable by self after 6 months (Perl example).
Rewrites reach 80-90% fast, then stall. For dav1d/x264 (heavy assembly), Rust C-part improves security, but assembly jumps anywhere, breaking model. Need secure assembly: compile-time checks (checkasm like dav1d/x264/VideoLAN) to prevent invalid jumps/inline assembly.
Henrik Gramner (Sweden): knows Intel x86 better than Intel; all SIMD cycles across P4, Nehalem, Core 2, etc. Nicest person.
Martin Storsjö (Finland): ARM/Neon expert for iPhones/Android. Codes assembly on phone virtual keyboard at playground watching kids.
Requires deep architecture knowledge. x86 (out-of-order) simpler; ARM varies wildly (A72, Apple variants)—write efficient code across all.
Open source splits happen (disagreements allowed by license). Examples: GCC 2 vs EGCS (→GCC 3), KHTML → WebKit/Blink. VLC: fork for features, merge back.
FFmpeg split to Libav; community merged back, moved on. Forks improve projects (e.g., GCC fork sped it up). FFmpeg better post-fork. Spent time reuniting community.
Reasons for the FFmpeg and Libav split in 2011 were project governance, leadership style, and development processes, not fundamental technical disagreements. FFmpeg absorbed Libav's work while Libav withered, and most distributions and developers returned to FFmpeg. From a Linux user perspective, distributions like Ubuntu (versions 12, 14) briefly switched to Libav then back to FFmpeg, creating confusion. Similar to Apple replacing GCC with Clang. Despite heated drama, Libav developments were merged back into FFmpeg, resulting in a superset of features. Debates like code reviews were settled in FFmpeg per community agreement. Active Libav contributors returned, strengthening FFmpeg.
Forks are less straining than maintainer burnout. Daniel Stenberg, maintainer of curl and member of the European Open Source Academy, opposes AI slop—fake reports and bad patches that burden maintainers. The XZ fiasco involved one maintainer overwhelmed by two attackers sending nonstop questions at odd hours, leading him to grant commit access to an attacker. Burnout stems from maintenance demands, not just forks. Speaker now maintains multiple multimedia and non-multimedia libraries after original maintainers quit. Requires tough skin to handle complaints felt personally. Google underestimates dependency on small open source projects, as shown in the Nebraska meme where FFmpeg underpins all modern digital multimedia infrastructure, maintained by a handful of 10-15 core developers. Compare to XZ (one person), libxml (unmaintained XML parser handling all edge cases), and the single maintainer of all time zones (possibly Nebraska or South Dakota). Large corporations ignore maintainer mental health.
Support open source financially and spiritually. Celebrate maintainers; security community wrongly criticized hobby projects for edge cases. Human civilization runs on unpaid passionate contributors doing hard, beautiful work. Speaker received death threats on VideoLAN around 2009-2010 for dropping VLC PowerPC support during Apple's transition from PowerPC to Core Duo (started 2006), near VLC 1.0 release with only four developers. Received letter with powder (flour, mimicking anthrax threats) saying "PowerPC forever." Police involved; forged resilience despite initial shock and family distress.
Positive feedback like Reddit thanks for VLC shared in IRC (still used daily, no ads/tracking; lacks threads, emojis, message editing unlike Slack; IRCv3 has them but unused; old-school emojis with colons/dashes). Always respond to thanks to validate open source appreciation. Celebrate FFmpeg, VLC, Linux, and quality software/companies. Celebrate human endeavor to build excellent, useful tech that solves hard problems and eases lives. Engineers love building challenging, cool things (speaker wanted to build cars young).
Kieran: First programmed Microsoft QBasic on Windows 3.1/95, building multiplication/counting loops, then ambitiously planned a soccer video game. Jean-Baptiste: BASIC, Turbo Pascal end of elementary; serious programming at 11 in Florence, Italy, using Logo with turtle graphics for complex designs, sparking computer passion.
H.264 (encoded by x264) dominates internet video, Blu-ray, broadcasting. Emerged with HD, fast Intel Core 2/Nehalem CPUs enabling real-time. Key: focus on psychovisual metrics over PSNR (peak signal-to-noise ratio, mean squared error/log causing blurring by averaging errors). Hobbyists (often anime fans) pioneered psychovisual rate distortion (block energy compensating human perception) and adaptive quantization (bias bits to complex areas like trees vs. grass). Tested by eye on laptops, not pro screens.
ParkJoy (Swedish TV, early HD film, no-expense-spared): runners by river with reflections, high-info textures (trees, water, grass, motion, leaves, lighting)—sorts encoders; PSNR blurs, psychovisual/adaptive quantization excels. Planet Earth birds flock (millions, complex constant bitrate). Anime banding fixed by Loren Merritt. x264 remains reference for AV1, AV2, VVC, HEVC. Used in Blu-ray (e.g., Cinema Paradiso, Fringe box set by Chris Henderson/Warner Bros.), preferred over proprietary for quality on consumer TVs.
Started by Laurent Aimar at École Centrale Paris (VLC birthplace). Contributors: Loren Merritt, Jason, Måns, Henrik, Anton (pioneered assembly in FFmpeg, dav1d). Mostly never met, many anime lovers. Pre-Crunchyroll, fans ripped Japanese DVDs, fan-subbed (Aegisub for anime/South Asian/Japanese subtitles with ruby diacritics). Anime has unique gradients, textures, banding from digital production.
All codecs since MPEG-2 share concepts: inverse transform, intra prediction, motion compensation, entropy coding. Each generation: 25-50% compression gain. MPEG-2 → DivX era → H.264 leap → HEVC/VP9 (royalty-free like VP9, vs. H.264 licensing costing hundreds of millions). AV1 (Alliance for Open Media): 40-60% less bandwidth than H.264 at same visual quality/given bitrate—increase quality or reduce bitrate. Needed as SD→HD→4K→4K HDR multiplies sizes by 2-4x.
Better compression requires more coding tools, bigger blocks, and lots more sub-partitions, making it exponentially more complex. The encoder searches more possibilities, such as predicting a block in directions like left, right, bottom, up, and more refined ones like north, northeast—starting with 8 directions, then 16, 69, or 128. Each increase means the encoder spends more time checking which tools compress better. AV1 encoding takes two orders of magnitude more CPU cycles than H.264.
CPUs are not getting faster; instead, more cores are thrown at the problem. Encoding happens once for hundreds of millions of users. YouTube encodes almost everything in H.264, but popular videos get re-encoded in AV1 because it costs more to encode but serves millions, trading encoding time, server-side CPU, and client-side efficiency. Smaller file sizes benefit battery, modem, etc.
Top five: H.264 in MP4 containers, AV1 in MP4/WebM containers, ProRes for nonlinear editing in MOV containers. ProRes is Apple's codec for editing, originally for Final Cut Pro, designed for fast decode and seek since editors move quickly. It has no temporal compression—it's intra-only.
I-frames (key frames) are complete frames like JPEGs. P-frames (predicted) take parts from previous frames (e.g., blocks 5, 7, 42) and add differences, requiring a prior I-frame to decode. B-frames (bi-predicted) depend on past and future frames, so decoding order differs from display order—the encoder buffers and decides dependencies. ProRes is intra-only (all I-frames).
Default FFmpeg GOP for H.264 is 250 frames (several seconds). Intra-refresh (used in Kyber) builds an I-frame gradually without full I-frames after the initial one, refreshing parts of the image. B-frames are mind-blowing as they require waiting for future frames.
Decoders worldwide produce bit-exact results despite complexity.
FFmpeg provides access to parameters like resolution, frame rate, codecs (H.264 to AV1), bitrate-quality trade-offs (constant bitrate, constant quality like CRF, QP), GOP length, number of B-frames. Thousands at YouTube, Netflix, Meta optimize these for movies, user-generated content, phone videos, screen recordings, editing—different needs per type.
YouTube delivers at scale: uploads from everywhere (even low-view videos), minimal latency, all resolutions—like web version of VLC.
Google Video used VLC plugin in browsers via ActiveX. Now, VLC WebAssembly compiles VLC/FFmpeg to run in JavaScript.
WikiLeaks Vault 7 revealed CIA used modified VLC with a plugin (psapi.dll) to read/encrypt/send documents from folder during playback—clever as fans spin and CPU spikes normally. VideoLAN issued press release: only download from official site.
Chinese hackers targeted Indians with fake libVLC DLL, leading to VLC ban in India (fought and lifted). Open source vulnerability: fake sites or targeted ads trick downloads.
Germany has fake VLC site (12+ years, German language, dark SEO) ranking high on Google despite reports—Google ignores as binary too big for analyzer. It installs dormant service activating after 3 weeks for spyware/adware (replaces ads).
Make sure wherever you're downloading stuff, it's legitimate.
VLC has ~500 plugins (including FFmpeg), supports many formats/protocols/filters/architectures, calls drivers (Intel, NVIDIA, AMD hardware decoders). Security issues in shaders, VLC, FFmpeg, or drivers could crash and exploit (e.g., via malformed media like browsers/PDFs, launching ransomware/botnets).
Desktop apps like VLC/Adobe have full file access. Solution: sandbox to protect from integrated third-party code. Mobile apps are sandboxed, but VLC needs broad access (defeats purpose). Split into processes: decoding, demuxing, filters—each sandboxed like Chrome tabs. Challenge: sustain gigabits/second memcopies (vs. web's MBs).
With millions of users, every feature gets used—e.g., puzzle filter (jigsaw video, movable pieces) by bored viewers of French movies/love triangles. Added 2010 by French math teacher for Bézier curves; 2015 email complained 16x16 (256 pieces) too simple—increased to 256x256.
Command-line VLC/ASCII art useful for debugging multicast networks: SSH routers, check black/green frames.
Streaming less complex than codecs; hardest was 90s/2000s satellite broadcasting (fixed bandwidth, no bursts). Modern live streaming allows pause; real-time encoding needed.
Adaptive streaming for CDN congestion: player switches resolutions. Basic algorithm: download segment, if >50% time to download, drop quality. Harder: deciding when to increase. Encode 7 resolutions with bitrates (less strict now).
Human psychology: tolerance to video changes lower than audio (e.g., AAC to Spectral Band Replication sounds tinny/jarring). Audio glitches noticed immediately. Sports at 30 FPS tolerated (should be 60). Live sports: satellite to central encode (older resolutions, real-time, no QA), push to CDNs, add DRM, support devices/browsers vs. controlled set-top boxes. 10-20s latency acceptable.
Kyber targets milliseconds for feedback control: drones, humanoid robots, hovers, cloud gaming, remote cars (e.g., Waymo fallback). Evolution: FFmpeg for files → streaming → lower latency. Pushes network limits; prioritizes latency over quality.
Kyber is an SDK platform for end-to-end control of machines, particularly in robotics contexts like teleoperation for training robots via machine learning. It uses a single QUIC protocol connection based on UDP for low latency, avoiding TCP head-of-line and HTTP head-of-line problems. It's ciphered by default and sends multiple streams on the same connection: audio, video, commands (mouse, keyboard, gamepad), while maintaining synchronization and coherence.
Clock Drift and Synchronization
Clocks drift across devices like robot cameras, sensors, GPS. Kyber accounts for clock drifting, learned from VLC, broadcast, real-time, and MPEG-TS. When recording a Kyber stream from a robot, it's predictive for playback, ensuring coherent data for AI training. Existing solutions work for one camera but become complex with five or six. Visual snapshots must match the exact time they happened. For control, actions must occur precisely when commanded. Server (robot) has re-timestamping mechanism accounting for clock drift.
Use Cases
Remote drones (defense or non-defense), remote cars, remote submarines, industry, remote surgery where experts can't go due to danger or cost. Goal: make distance disappear for projection of skills or power. Streaming for devices like Meta Ray-Ban glasses requires GPU power from cloud or phone. Needs real-time latency for video, with encoders toying to encode a frame in 4 milliseconds. Kieran's company also achieves under that latency, optimizing local latency (encoder, decoder) added to networking time.
Reliability and Low Latency Techniques
Forward error correction: over-transmit a few percent of data to allow packet loss reconstruction on client side without retransmission latency, crucial over internet for distant control. Demo at CES in Las Vegas: 3D-printed rover (small car with telescopic arm), controlled from France using webcam and small PCB server, streaming to the other side of the planet. Use cases include AI controlling many drones. Requires excellence in video, networking, every millisecond in networking/encoding/decoding, low-level integration, and syncing everything.
Goal: 4 milliseconds glass-to-glass latency. Glass-to-glass: from computer running program (e.g., video game or robot) capturing at 1000 Hz (240 Hz equivalent) through network replication to display. Achieved 7 milliseconds Windows-to-Windows or Windows-to-Mac: 3.5 ms in NVIDIA hardware encoder, 2 ms in Intel decoder (total 6 ms). To go lower, need better codecs or encoders. 4 ms is the grail. Achievable with custom hardware like SDI, but Kyber targets internet with small hardware like Jetson Nano or N150 for millions of robots/drones (rolling, flying, swimming). For autonomy, need teleobservation: AI monitors remote cars, alerts on problems, operator intervenes for safety (e.g., humanoid caring for grandma, cars stopping correctly).
Kyber is open source, dual-licensed: commercial and AGPL. For product use, full product must be open source; otherwise, pay commercial license for support, IP, modifications. Allows hobbyists/small users to build open source projects. Building robots: 3D-printed rover, finishing 3D-printed drone wings, sailboat, humanoids (not high-quality, but for accessibility).
AV1 by Alliance for Open Media (Google, Netflix, Amazon, Apple, VideoLAN): royalty-free codec finished 2018, now deploying. AV2: next generation, 30% better (same quality, 30% bandwidth reduction vs AV1). dav1d is AV1 decoder (recursive acronym: dav1d is an AV1 decoder, spelled with '1'). AV2 decoder: dav1d2 (D-A-V-2-D, 'de' is two in French). Demo at CES: VLC running first AV2 demo.
Codec Specifications and Implementations
Specification (e.g., AV2, H.264) defines how codec works. Encoders: AVM for AV2 (like x264 for H.264, x265 for H.265); future SVT-AV2. Decoders: dav1d for AV1, dav1d2 for AV2, ffh264 for H.264, ffhevc for HEVC (H.265) in FFmpeg. H.266 (VVC) is next MPEG after H.264/H.265. Each generation ~30% better: HEVC 30% over H.264, H.266 30% over HEVC (varies by scenario, e.g., 70-80% for screen recording). Main codecs: H.264, H.265, H.266, AV1, AV2.
Dual Naming and Standards Bodies
H.264 is MPEG-4 Part 10 (ISO) and AVC (ITU). HEVC is MPEG-H Part 2, H.265. H.266 is VVC. ISO (private), ITU (UN). Politics/history leads to dual names.
Major differences: encoding cost, patent royalties. Multimedia is patent minefield (like 3G/4G/5G, RF). AV codecs aim royalty-free (fringe patents possible). MPEG: patents not discussed in standardization; post-format, patent pools like MPEG LA aggregate patents, charge users. Codec patents: e.g., using rectangles instead of square blocks. H.264 patents sane; HEVC bloated with rarely-used patented features for pools (MPEG LA, HEVC Advance, Nokia outside). HP removed HEVC support from Windows laptops due to rising costs/uncapped patents. YouTube/Netflix faced hundreds of millions/year, led to AV1/AV2 via Alliance for Open Media (also audio codecs). AV sidesteps patents (no golden frame, S frame; variants like "I can't believe it's not a B frame" to avoid infringement). VVC has all HEVC patents plus new. Double creativity needed: efficiency + patent avoidance.
FFmpeg/VLC and Patents
Don't worry due to France rejecting software patents (mathematical/idea patents invalid). VLC licensing would cost >200 euros/user. Open source helps.
New generation of entrepreneurs in Europe, especially France (French tech). Stigma of failure gone; people now want startups vs big companies. Over-regulation exists, but positives too. VideoLAN has no office; everything client-side, no platform. Responds to court orders without bankruptcy risk (lawyer fees <10k USD/year). Received intelligence agencies requesting backdoors: said no (less politely); would shut down if compromised. Compilation: offline boxes, compile compiler first, double signing. Blocked governmental fake binary push. Can relocate easily (e.g., Malta, Cayman Islands), change domain.
No Censorship or Surveillance
VLC never censors videos; fully offline, no server talk, no telemetry. Can't know usage or block content (e.g., anime). Helped US police with murder case file playback; Afghan War army RTSP movies for troop morale (provided fixed version).
VLC is completely open source, I think it is allowed on the US Army laptops, right? Because I
US military reviewed and approved VLC's processing documentation. VideoLAN only works with authorities for support. No issues seen with how people use VLC.
VLC has a small core contributor team of six to eight people, but Jean-Baptiste Kempf handles everything legally alone. He used to stress a lot about millions of users, military usage, and potential government pressure, but not anymore. They're making complex technology easy for everyone as a neutral tool used for both good and bad.
You cannot blame a tool.
Source of Zen
Worst-case scenario thinking: like a chess player, is anyone dead or hurt? If not, proceed. For startups, worst case is bankruptcy, which is normal. Lawyers unhappy? VideoLAN has only 50 grand; source code is public and unstoppable. Work is for the greater good.
Active open source multimedia archiving community, led by Dave Rice from City University of New York. They value FFmpeg as a Rosetta Stone for playing multimedia a thousand years from now, preserving visual knowledge.
Example: UK's New Domesday Book archived on BBC microcomputers; within 10-15 (or 20) years, no software to play it, requiring reverse engineering.
FFmpeg written in C, closest to mathematics and logic. C compilers will exist in 1,000 years, like Latin—still usable.
Archiving community funded FFV1 codec (lossless, fast software-based encoding). Concerned about compression altering material. Need lossless, fast, resilient storage (e.g., recover from lost bits without losing whole GOP). Funded GPU encoding for faster FFV1.
Many global archival groups use FFmpeg and FFV1. They provide specialist advice, e.g., 1950s colorimetry on specific tapes.
Archivers know deep details on video, colorimetry, colors, storage—things developers don't. Terrified of losing essential content in lossless archiving, deeply understanding what to preserve.
Film scanners are a huge topic; much film degrading due to poor storage. Open source workflows shared with budget-limited countries (e.g., teaching FFmpeg commands to children in India). Model community ethos: participating in something bigger for thousand-year impact.
In 1,000 years, may drown in AI slop; archiving real life is essential. Capturing 20th-21st century transition from data scarcity to oceans of slop.
Losing tons of '30s-'50s films, '70s-'80s tapes—not enough tape heads to read all. Huge moral hazard in deciding what to archive. Digital stewardship ensures playable by everyone, not obsolete devices.
Needle in haystack: archive all, find gems later (e.g., something in corner compressed away). FFV1 mathematically lossless; error recovery on bit flips, previous frames.
FFmpeg will exist in 100 years; VLC maybe.
Next 5-10 years easy; beyond: holograms, brain-computer interfaces (Neuralink), VR. Multimedia expands to brain-consumable data/codecs.
VLC for Neuralink. FFmpeg -i input format human brain.
Codecs for point cloud, volumetric videos, RGBD (depths for robotics/3D). Compression for astronomy, 3D elements.
VLC has VR/XR version; Kyber streams XR for low-power glasses (Apple Vision, Quest): 3D, interactive, low-latency volumetric/point cloud.
Will manage 3D data. Community open, though not all see it; leaders/entrepreneurs do.
Tension in FFmpeg: stick to video/audio excellence? Define multimedia as digital representation of streams for human senses.
Future: odor sensors/diffusers (demuxer track type: odors; stereo smell—left/right nose). Haptic plugin in VLC for 4D cinemas (hydraulic arms in theme parks, synchronized data feed). Many standards exist.
Small maintainer team (150 VLC contributors/year, 300 FFmpeg). Goal: integrate all contributions, maintain architecture for modular additions (e.g., spatial audio demo; easy for next modules). Framework for everything timed/sensible: brainwaves, etc.
FFmpeg/VLC push world to standardize (e.g., brainwaves, haptics for robots/Neuralink). Pattern: new topic → 5 standards → hype down → standardization by followers → leader pressured in.
Example: 3D audio—past hype (Android Cardboard, two formats dead); now back with lessons.
Dolby was amazing, defined sound with great engineers; now mostly lawyers/licensing, less innovation (like HP).
Favorite FFmpeg tweets:
-
Talk is cheap, send patches.
-
FFmpeg, nothing is beyond our reach. (From US military satellite patch for global monitoring.)
FFmpeg on Mars 2020 rover for picture compression (COTS tech, paper written).
Multi-planetary library. VLC in weird places: Formula 1 paddocks (live feed), European Space Agency, SpaceX launches, particle accelerator.
CERN LHC (27km ring): VLC monitored analog cameras via capture cards, streamed multicast. Fixed issue in 2010 with Laurent; toured antimatter, colliders—amazing physics day.
No regretted tweets:
Je ne regrette rien. Regrets are a tax on your mind. Learn from mistakes, don't regret.
Hate is a very expensive emotion. (Johnny Depp quote.)
Gratitude for FFmpeg, VLC, community, spicy tweets, conversation. Sexy hat like wizard.
Lex: > Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program. (Linus Torvalds)
Keep Lex Fridman in your library
Save the videos and channels worth coming back to, and find them again in one place.
