Episode Summary
Executive Summary: Jan Leike discusses technical AI safety at DeepMind, focusing on making machine learning systems robust, aligned with human preferences, and secure against failure modes like distribution shift, adversarial attacks, and unstable reinforcement learning. He argues AI safety is a neglected, high-impact field where empirical ML research and strong research skills can meaningfully reduce long-term risks from powerful AI.
Main Topics: Why AI safety matters: Leike argues AI could deliver enormous benefits, but only if we understand and mitigate risks before building more capable systems. Safety research aims to ensure advanced AI does what humans intend. Learning objectives from human preferences: He explains DeepMind’s work on reward learning from comparisons of behavior, such as ranking video clips to train a reward predictor for reinforcement learning tasks like a backflipping robot. Failure modes: distribution shift and reward misspecification: A major concern is that learned reward models can break when the system enters novel states or when humans stop giving feedback, causing confident but wrong behavior. Machine learning security and adversarial robustness: Leike highlights how deep networks can be fooled by tiny perturbations, including black-box transfer attacks, and says defenses are still weak. Other technical AI safety problems: Beyond preference learning, he points to safe exploration, minimizing side effects, stability of deep RL, and robustness as open research areas with low-hanging fruit. Career advice for entering AI safety: He recommends a strong math/computer science foundation, early research experience, and often a machine learning PhD or equivalent to build the skills needed for technical AI safety. Working at DeepMind and collaboration: Leike describes DeepMind as collaborative, intellectually dense, and actively building its technical AI safety team in cooperation with OpenAI and FHI.
Key Arguments: AI safety is one of the most pressing problems because increasingly powerful AI could have large positive impacts or serious unintended consequences if objective functions are misaligned. Learning human preferences from rankings is a practical step toward building systems that better reflect what people actually want rather than what is easy to specify. The backflipping-noodle project revealed that learned reward predictors can fail under distribution shift when feedback is reduced or the environment changes. Neural networks often produce overconfident outputs in unfamiliar settings, so they do not naturally recognize uncertainty or know when to defer. Adversarial examples show that modern ML systems can be surprisingly easy to attack, including in black-box settings where the attacker does not know the original model. Technical AI safety is still underpopulated, so skilled ML researchers can have unusually high marginal impact by joining the field now. Empirical machine learning work is currently especially valuable for AI safety because many problems are underexplored and amenable to experimentation. A PhD is valuable not as a credential per se, but because it develops the package of research skills needed to contribute effectively to AI safety.
Data Points: EAG San Francisco timing: Second weekend of June - Announced as the next Effective Altruism Global event Early bird deadline: 18th of March - Listeners were told they could save money by applying before this date DeepMind collaboration: With OpenAI and Future of Humanity Institute - Leike says the technical AI safety team collaborates with both organizations Research task example: Backflip - The human-preference learning paper trained a small robot/nothing-like 'backflipping noodle' to do a backflip Feedback format: Two video clips, left better/right better/same - Current human ranking interface for training the reward predictor Postdoc duration: About six months - Leike says he was a postdoc at FHI for around six months before joining DeepMind Technical safety team size: Not that many people yet - He says the team is still small and wants more researchers PhD length in the US: Five or six years - Leike describes typical US machine learning PhD duration PhD length in Europe: Three or four years - European PhDs tend to be shorter but usually require a master's degree Career path example: One paper before finishing master's - He recommends publishing at least one paper before completing a master's degree if possible Recommended conferences: ICML, NIPS, ICLR - He suggests meeting researchers and reading papers from top ML conferences Typical random-seed experiment: 10 different random seeds - Used as an example of instability in deep RL performance
Pivotal Quotes: "I think we really need is to have well-reflected, sane, and informed debates about it." — Jan Leike: On the public debate around AI risks and the need for less polarization "I would love if you're really good at machine learning, you should come work with us." — Jan Leike: His closing appeal for more talent in technical AI safety "So, in a way, I think the working on AI safety is exciting because it's kind of not a very well-established field yet. There's lots of different low-hanging fruit that are just really ripe to pick." — Jan Leike: Why the field is attractive to researchers now
Implications: For listeners, the episode frames AI safety as a concrete research career path with outsized impact. For the industry, it highlights urgent needs in robustness, alignment, and security, plus a severe talent shortage in technical AI safety.