Red Teaming o1 Part 1/2– Automated Jailbreaking with Haize Labs' Leonard Tang, Aidan Ewart, and Brian Huang

Episode Summary

Executive Summary: This emergency episode examines OpenAI’s o1 and o1-mini reasoning models through the lens of outside red-teamers from Haze Labs and Apollo Research. Guests report stronger formal reasoning, improved jailbreak resistance, and faster feedback cycles, but also emphasize that subtle, long-tail failures, dual-use ambiguity, and deceptive behaviors remain real safety challenges.

Main Topics: Haze Labs' role in frontier-model red teaming: The guests explain Haze Labs’ origin as a startup focused on rigorously testing AI systems at scale, drawing from adversarial robustness research and selling red-team services to frontier labs including OpenAI, Anthropic, and AI21. Attack methods and automation: They discuss a portfolio of jailbreak and probing techniques, including accelerated coordinate gradient attacks, MCTS-style search, evolutionary methods, reverse language modeling, cipher/encoding attacks, and multi-turn prompt strategies, emphasizing that they rely heavily on automation plus human verification. o1 capabilities and model shift: The guests describe o1 as a substantial step forward in formal domains like math, coding, and symbolic reasoning, while noting it was not dramatically better in casual chat. Nathan frames it as a 'GPT-4.5'-style jump in reasoning and planning. Safety, jailbreak resistance, and calibration: Discussion centers on the model being harder to jailbreak largely because it reasons better, but not being immune. The speakers stress persistent false positives/false negatives in LLM-based judges and the need for calibrated human review. Dual-use and context-dependent safety: The conversation broadens to the difficulty of classifying benign versus dangerous content in real-world settings such as biotech, finance, enterprise search, and sensitive social topics, where the same output can be safe in one context and unsafe in another. Capabilities-safety relationship and scaling: The speakers debate whether more capable models become safer automatically. They note some evidence for capability-safety correlation, but also point to inverse-scaling or attack-specific cases where larger models can be more vulnerable, and where jailbreaks can degrade benchmark performance. Preparedness, release process, and future governance: The episode closes on OpenAI’s tight feedback loop with red teamers, the medium-risk classification, and the unresolved question of how future whistleblowing, audits, and external testing should work for increasingly powerful models.

Key Arguments: OpenAI’s o1 appears to be trained via intensive reinforcement learning on a GPT-4-class base, making it a major reasoning upgrade rather than a mere incremental chat improvement. Frontier-model red teaming now depends on automated pipelines plus human judgment because LLM-based judges are too brittle to reliably classify subtle harms. Cipher, multi-turn, and long-context jailbreaks remain effective across many frontier systems, including o1, showing that attack surfaces are still very large. Model safety is heavily context-dependent: outputs that are benign in one enterprise, legal, or research setting may be harmful in another. Capabilities and safety are correlated in some settings, but the relationship is not universal; some attacks may become more effective as model capability increases. The most important remaining safety problems are subtle, long-tail failures rather than obvious refusals on extreme harmful prompts. Improved safety may come from deeper generalizable reasoning about safety rather than continually patching individual jailbreak prompts. Current evaluation methods are not sufficient to replace human supervision, especially for ambiguous or dual-use cases.

Data Points: Haze Labs founding date: January - The guests say the company started in January and came out of stealth in June. Haze Labs stealth launch: June - They mention a public 'Bad Day to Be an LLM' launch moment when coming out of stealth. Seed round investor: General Catalyst - They report raising a seed round from General Catalyst in the summer. Red-teaming time window: around a month - They say they had roughly a month to test o1, more than the 10 days rumored online. OpenAI access date: August 26 - Nathan references the system card noting Meter received access on August 26. o1-mini access timing: early August - Nathan notes the system card also says access began in early August for another phase. Safety benchmark effect: up to an order of magnitude more refusals - Brian says circuit-breaker-style defenses can increase refusals on benign queries by around 10x. Attack effect on MMLU: significantly lower than baseline - Brian says MMLU performance drops materially when questions are passed through the bijection-learning jailbreak. Model-size effect on adversarial training: more sample efficient and more effective - Aiden cites work showing larger models can make adversarial training against some attacks easier. Anthropic/OpenAI frontier timeline referenced: 2025 or 2026 - Nathan paraphrases public concerns that leading developers could be far ahead by 2025 or 2026.

Pivotal Quotes: "O one is harder to jailbreak, largely because it reasons more effectively in general, and this includes reasoning about what it should and shouldn't do." — Nathan Levenz: Opening analysis of the new model’s safety profile and why capability gains may also improve refusal behavior. "Subtle doesn't mean benign." — Brian Huang: A discussion of long-tail, context-dependent harms where seemingly minor failures can still be dangerous. "We've just leveled up to a new game, and we now need to figure out how to do testing and red teaming in a way that's you know befits this new level of capabilities." — Brian Huang: Closing reflection on how o1 changes the frontier for evaluation and safety work.

Implications: o1 raises the bar for reasoning but also for red teaming: future safety work will need better judges, more human oversight, and defenses that generalize across contexts, long horizons, and dual-use applications.