Episode Summary
Executive Summary: Nicholas Carlini discusses his path from systems security into ML security, focusing on two major risk areas: adversarial examples and privacy leakage in generative models. He explains why many defenses fail, how adversarial training only partially helps, and how large models can memorize and leak training data. He also covers data poisoning risks in web-scale datasets and practical mitigations like hashing source files.
Main Topics: From systems security to ML security (Priority: 5/5): Carlini explains that he entered ML security by looking for an underexplored area with open problems, then applied classic security thinking to neural networks. Adversarial examples and failed defenses (Priority: 5/5): The conversation reviews how tiny input perturbations can fool classifiers, why many proposed defenses did not work, and why adversarial training became the main partial defense. Practicality and real-world relevance of adversarial attacks (Priority: 4/5): Carlini argues that while adversarial examples are not the most common attack in the wild, they reveal fundamental limits and matter in systems where ML is the main gatekeeper. Privacy leakage and memorization in diffusion models (Priority: 5/5): He describes membership inference and extraction attacks, showing that large generative models can memorize and reconstruct training examples, including sensitive images. Black-box vs white-box attack settings (Priority: 4/5): The interview contrasts query-only attacks with access to internal weights, noting that black-box methods already work well for several privacy attacks and white-box access has been less useful than expected. Data poisoning in web-scale training pipelines (Priority: 5/5): Carlini explains how uncurated internet data and expired domains create realistic poisoning opportunities, and how simple defenses like hashing can help but reduce usable data.
Key Arguments: Adversarial examples are fundamentally hard to solve; hundreds of defenses have been proposed, but only a few meaningfully work. The standard way to attack a model is gradient descent on the input rather than the weights. Adversarial training is the first broadly effective defense, but it only partially restores robustness and is far from a complete solution. Even if adversarial examples are not the most common real-world attack, they identify the ceiling of what security for ML classifiers can achieve. Privacy threats are more serious when models are trained on sensitive data such as medical images, emails, or text messages. Large diffusion models can memorize and leak training images, though the exact memorization rate observed was low under a strict definition. Memorization and extraction matter because leaking even a small number of sensitive records can be unacceptable. In practice, the most vulnerable examples were often duplicated training images, but the exact memorization pattern remains poorly understood. For some privacy attacks, black-box querying is already enough; white-box access has not proven substantially better in most practical settings. Web-scale dataset construction via URLs is vulnerable because expired domains can be bought and turned into poisoning vectors. Checking hashes of downloaded images is an effective mitigation for one poisoning path, but it can also remove a large amount of usable data if exact bytes are required.
Data Points: Adversarial training accuracy on CIFAR-10: ~60% - Carlini cites adversarial training as improving robustness from 0% under attack to around 60%, still far from ideal. Standard CIFAR-10 classifier accuracy: 99% - Baseline accuracy without adversarial attacks. Accuracy under adversarial attack: 0% - Carlini says strong attacks can reduce standard classifier accuracy to zero on CIFAR-10. Stable Diffusion prompts tested: 350,000 - Black-box extraction experiments on Stable Diffusion. Stable Diffusion memorized images found: 109 - Exact pixel-level memorization examples recovered from the 350,000-prompt search. Imagine prompts tested: 1,000 - Smaller query budget due to slower model access in Google’s Imagine model. Imagine memorized images found: 23 - Memorization examples recovered from the 1,000-prompt search. Stable Diffusion model size: ~900 million parameters - Used to argue the model cannot literally memorize a 10 TB dataset in full. Stable Diffusion training data size: ~10 terabytes - Compared against model size to show memorization must be limited. Observed memorization rate: ~0.00001% - Carlini describes the fraction of training data found as strict verbatim memorization as extremely small. CLIP-scale poisoning rate: ~1 in 10,000 to 1 in 100,000 images - He gives rough poisoning thresholds for multimodal contrastive self-supervised models. CLIP training images: ~400 million - Used to estimate how many malicious images would be needed for poisoning. Poisoning images required in a large CLIP experiment: a couple thousand - For a 400M-image setup, Carlini says only a few thousand poisoned images may suffice. Domains acquired for poisoning study: 6 domains - Three popular and three random domains were purchased to demonstrate expired-domain poisoning risk. Estimated cost of domains: ~$100 - Cost to buy the expired domains used in the poisoning study. Conceptual Captions 3M total images: 3.3 million - Dataset used to illustrate how many URLs no longer match exact bytes. Conceptual Captions 3M still available online: 2.9 million - Images still retrievable from the web. Conceptual Captions 3M exact-byte matches: 1.1 million - Only exact hashes matched when requiring byte-for-byte identity.
Pivotal Quotes: "The way you attack a machine learning model is with gradient descent. Only instead of taking the gradients with respect to the weights of the classifier, you take the gradients with respect to the input." — Nicholas Carlini: Explaining the mechanics of adversarial examples and why tiny perturbations can fool models. "Defense works, we mean like if the attacker tries like a million times, they'll still fail. You know, here you try twice, and probably one of them works." — Nicholas Carlini: Comparing normal security expectations to the partial protection offered by adversarial training. "If your medical images were one of the 109 medical images extracted, you wouldn't go, fair enough, oh, it's fine, like it's okay that you memorized my images." — Nicholas Carlini: Arguing that low memorization rates can still represent serious privacy failures.
Implications: ML security is moving from theory to operational risk: models can be fooled, leak training data, and be poisoned through web-scale pipelines. Builders need stronger defenses, dataset provenance checks, and realistic threat modeling—not just benchmark gains.