#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

Episode Summary

Executive Summary: The conversation explores FFmpeg, VLC, and VideoLAN as critical open-source infrastructure powering modern media. Jean-Baptiste Kempf and Karen Kunya explain codecs, containers, reverse engineering, and the engineering culture behind these projects, while also defending open-source governance, volunteer maintenance, and the need for low-level optimization, security, and archival preservation.

Main Topics: How video playback and compression actually work (Priority: 5/5): The guests break down the path from URL/file to decoded pixels and sound: demuxing containers, decoding audio/video streams, GPU vs software paths, compression, prediction, transforms, and human-perception-driven tradeoffs. FFmpeg and VLC as foundational open-source infrastructure (Priority: 5/5): They frame FFmpeg as the low-level multimedia toolbox and VLC as the iconic media player/client that makes hard technical problems easy for users, both used across the internet and embedded in major products. Open source licensing, governance, and community norms (Priority: 5/5): A major theme is the social contract of GPL/LGPL/MIT-style licensing, relicensing challenges, contributor ownership, and how community standards and meritocratic review shape project direction. Assembly, performance, and computer architecture (Priority: 5/5): The discussion emphasizes that hand-written assembly and SIMD still matter for extreme performance, especially in codecs, and that understanding CPU architecture is essential for real-world optimization. Security, abuse, and the maintenance burden on volunteers (Priority: 4/5): They discuss AI-generated security reports, public vulnerability drama, vendor-like expectations from trillion-dollar companies, and the burnout caused by underfunded volunteer maintainers. Reverse engineering and codec evolution (Priority: 4/5): The episode highlights historic and ongoing reverse engineering of proprietary formats, the evolution from MPEG-2/H.264 to AV1/AV2/VVC, and projects like x264 and David that push software decoders to the limit. Low-latency control, robotics, and the future of multimedia (Priority: 4/5): Kempf’s new project Kyber extends the same multimedia principles into teleoperation and robotics, aiming for millisecond-level end-to-end control over video, audio, and commands.

Key Arguments: Open-source multimedia works because many tiny maintainer teams build tools that billions rely on, and the projects survive through community, not corporate ownership. FFmpeg and VLC succeed by treating file formats as hints rather than truth, because real-world media is often broken, mislabeled, or partially corrupted. Compression is not like ZIP: codecs intentionally discard redundancy in ways aligned with human perception, using spatial/temporal prediction and frequency-domain math. Handwritten assembly can still outperform C by large factors in critical paths, and modern compilers do not eliminate the need for human performance engineering in all cases. Licensing is a social contract: GPL/LGPL choices determine whether companies can build proprietary products on top while still contributing back. Security work on volunteer projects must respect scale and incentives; large organizations should contribute patches and funding rather than only generate alarmist bug reports. Open source maintainers need both financial and human support; burnout and harassment are serious risks to ecosystem health. The next frontier of multimedia includes AV2, volumetric/video, XR, brain-computer interfaces, and real-time machine teleoperation, all requiring similar low-level rigor.

Data Points: VLC downloads: Over 6 billion / 6.5 billion - The speakers cite VLC as having been downloaded more than 6 billion times, with one estimate given as 6.5 billion. FFmpeg contribution scale: 2,000-3,000 contributors - JB estimates the FFmpeg ecosystem has had roughly 2,000 to 3,000 contributors over time. Core team size: 5 people (VLC core), 10-15 people (FFmpeg core) - They describe the small number of maintainers who carry long-term responsibility for these projects. Files not GPU-decodable: Up to 45% - During the decoding walkthrough, JB says a large share of files may still need software fallback rather than GPU decoding. Compression improvement per codec generation: About 25-50% better; often summarized as ~30% - They repeatedly describe new codec generations as yielding roughly 30% better compression at the same quality. AV1 bandwidth savings vs H.264: 40-60% less bandwidth - AV1 is described as reducing bandwidth substantially relative to H.264 at the same visual quality. AV2 improvement over AV1: About 30% better - They describe AV2 as the next generation after AV1 with around 30% better compression efficiency. David code size: 30,000 lines of C + 240,000 lines of handwritten assembly - The AV1 decoder project David is highlighted as an extreme example of assembly-heavy optimization. FFmpeg assembly size: Around 100,000 lines of assembly - Karen estimates FFmpeg’s total assembly footprint is around 100k lines across codecs. Relicensing outreach: More than 350 contributors contacted - JB describes the effort needed to relicense parts of VLC from GPL to LGPL/MPL. VLC support window: Windows XP through Windows 11; macOS 10.7 onward; iOS 9 onward - They note the unusually broad platform support maintained by the VLC ecosystem. Kyber target latency: 4 milliseconds glass-to-glass - JB states the goal for ultra-low-latency teleoperation and robotics control. Kyber achieved latency: 7 milliseconds - They mention a working Windows-to-Windows / Windows-to-Mac path achieving around 7 ms. Pivotal test artifact: FATE automated test matrix across many OS/compiler/architecture combinations - FFmpeg’s automated testing is described as running on a huge volunteer-maintained matrix.

Pivotal Quotes: "Talk is cheap. Send patches." — Karen Kunya: Used to summarize the project culture: contribution matters more than rhetoric. "We make everything that is very complex easy to use for the normal people, for everyone." — Jean-Baptiste Kempf: Explains the mission of VLC and FFmpeg as simplifying highly technical multimedia infrastructure. "We are not the contributors. We are the maintainers." — Jean-Baptiste Kempf: Describes the long-term responsibility of a small core team maintaining the code for everyone.

Implications: The episode underscores that modern media, security, and even robotics depend on volunteer-built low-level software. The future will reward teams that combine technical excellence, sustainable governance, and open collaboration with real support for maintainers.