Episode Summary
Executive Summary: Prince Kanuma describes how he became one of the main contributors to MLX, Apple Silicon’s open-source inference ecosystem, and how that work led to new packages for audio, embeddings, vision-language models, and a real-time voice system called Marvis. The conversation explores MLX’s GPU-first design, trade-offs versus Core ML and the Neural Engine, model porting/quantization workflows, pruning and fusion research, and the future of multimodal, local AI assistants.
Main Topics: Origin story and entry into MLX (Priority: 5/5): Prince explains how he discovered MLX in late 2023, tested it on an M1 Air, saw its promise for local inference on Apple Silicon, and made his first contribution by porting StarCoder 2. MLX ecosystem and Apple Silicon inference (Priority: 5/5): The discussion clarifies what MLX is, how it differs from Core ML and llama.cpp, why it is GPU-centric, and where it fits in Apple’s broader AI stack. Model porting, quantization, and release workflow (Priority: 5/5): Prince outlines his process for converting Hugging Face models to MLX, generating multiple quantized variants, and deciding which bit-widths are worth testing versus leaving to community feedback. Pruning, fusion, and model behavior research (Priority: 4/5): He discusses research on pruning, discovering sub-networks that preserve or remove capabilities, and his Fusion technique for combining model traits without training. Building MLX Audio, MLX VLM, and embeddings (Priority: 4/5): Prince explains that these packages provide inference tooling and model support for new modalities, helping MLX expand from language models into audio and vision-language use cases. Marvis and real-time speech-to-speech agents (Priority: 5/5): He introduces Marvis, a small real-time voice model intended to enable low-latency, natural conversational agents and voice-driven computer use on Apple Silicon. Future of trustworthy, multimodal AI (Priority: 4/5): The conversation closes on broader AI trends: systems that challenge users, provide reliable answers with sources, and unify text, audio, image, and video into compact local models.
Key Arguments: MLX succeeded because it is optimized specifically for Apple Silicon, giving it untapped potential compared with more general frameworks. Local inference became compelling once MLX showed fast quantization and Apple’s GitHub activity signaled real momentum. The Neural Engine is more power-efficient, but MLX’s GPU-first approach makes large-model support and portability across Mac/iPhone/iPad much easier. A practical MLX contribution pipeline starts with Hugging Face model configs; if a model type exists, porting is straightforward, otherwise the Transformers implementation is translated into MLX. Quantization is the default deployment strategy, with 3-bit and 4-bit variants most carefully tested because some models are too sensitive to low-bit compression. Pruning is not just deleting weights; recovering performance usually requires extra compute, retraining, or distillation. Fusion is a promising zero-training technique that can combine capabilities from two checkpoints and sometimes outperform both source models on downstream evaluations. Audio, vision, and other media models are converging toward unified multimodal systems, reducing the need for separate ASR, LLM, and TTS components. Real-time voice agents need streaming, interruption handling, and low latency to feel natural, which is why Marvis was built from scratch. Prince sees local AI as a way to build practical assistive tools, especially for his blind father and for more usable computer agents. The next stage of AI, in his view, is not just AGI hype but systems that can reliably disagree, explain themselves, and give trustworthy, source-backed answers.
Data Points: MLX launch timing: End of 2023 (around December 2023) - Prince says MLX first appeared then, roughly three years after Apple Silicon Macs launched. Apple Silicon laptop used initially: M1 Air / M1 MacBook Pro-era machine with 16 GB memory - He started experimenting on a small M1 laptop before upgrading. First contribution to MLX: Porting Hugging Face StarCoder 2 to MLX - His first PR became part of a broader contributor merge. Models ported last year: Around 1,000 quantized model versions - Prince clarifies these were mostly quantizations rather than 1,000 distinct architectures. Distinct model families ported: About 20 to 30 individual models - He distinguishes unique model architectures from quantized variants. Model porting speed: About 30 minutes for a newly released model - After gaining experience, he could adapt new releases very quickly. Large model size mentioned: 100 billion parameters - He references an early Cohere model that was impossible to run on his 16 GB MacBook. Pruned model result: 6B parameters from an 8B Llama 3 model - He says he released the world’s first Llama 3 6B through pruning and continued pretraining. Performance recovered after pruning: >75% of the 8B model’s performance - Reported for the pruned 6B Llama 3 model. Throughput on Mixtral across two Apple Silicon devices: ~4 tokens/sec - He used Thunderbolt 4/5-style offloading ideas to test MOE inference. Community notifications on Hugging Face: 1,700 notifications - These are primarily requests and feedback on MLX ports. Total notifications across platforms: >10,000 - He notes a very high ongoing support burden across GitHub, email, and other services. Train-on-quantization techniques: DWQ and AWQ - He describes both as methods used to preserve performance after quantization. Marvis model size: ~250 million parameters - He says the first version runs real time on Apple Silicon. Marvis first-audio latency: ~80 milliseconds - The decoder enables very fast start of spoken output. Marvis development cost: $700 - Initial training of the first version required this spend before later optimization. MLX audio model example: 82 million parameters - He cites Kokoro as a tiny text-to-speech model available in the ecosystem. Real-time quality target for Marvis: Real time on M3 Max; close to real time on M1 Pro and potentially M1 after quantization - He frames current and future performance goals.
Pivotal Quotes: "if you're going to do this MLX thing, you have to be the best, the best of the best." — Prince Kanuma: He recounts his partner challenging him when he decided to start contributing to MLX. "I want a system that I can trust." — Prince Kanuma: He explains the future direction he finds most exciting in AI: reliable systems that can challenge users and provide sourced answers. "we have our first model. It is about 250 million parameters, and it can run real time on Apple Silicon." — Prince Kanuma: He announces Marvis, the real-time voice model he is training from scratch.
Implications: MLX is becoming a serious local-AI platform for Apple devices, especially as multimodal and voice systems mature. The next wave of useful AI may be smaller, faster, and more trustworthy, not just bigger.