Episode Summary
Executive Summary: Sam Charrington and Kamyar Aziza Denishelli review 2021 reinforcement learning, arguing the field made major progress in robotics, control, risk-sensitive optimization, and structured MDPs, even if it received less public attention than NLP. The conversation emphasizes a shift from general-purpose RL toward problem-specific, theory-backed methods that are safer, more robust, and closer to real-world deployment.
Main Topics: RL progress in 2021 and why it was less visible (Priority: 5/5): Kamyar argues RL had a strong year with major theoretical and practical advances, but fewer conference interactions during the pandemic meant fewer ideas were publicly shared and amplified compared with NLP. Robotics and guaranteed RL control (Priority: 5/5): A major theme is the adoption of RL in robotics, especially drones and walking robots, where guarantees on stability and performance are becoming possible through techniques like Lipschitz-constrained neural networks and online adaptation. Control theory meets reinforcement learning (Priority: 5/5): The discussion explains how control theorists are reframing classical control problems as RL problems when dynamics are unknown, leading to new algorithms, new conferences, and revised notions of robustness. Risk-sensitive RL and off-policy risk assessment (Priority: 4/5): The speakers discuss moving beyond expected reward to optimize and evaluate policies using quantiles, variance, CVaR, prospect theory, and other risk functionals, especially for healthcare, finance, and societal decision-making. Structured MDPs and problem-specific RL (Priority: 4/5): Kamyar highlights renewed interest in structured assumptions such as state abstraction, rich observations, linear MDPs, and low-rank dynamics to improve sample efficiency and make RL practical. Self-supervised transfer and generalization (Priority: 3/5): He identifies self-supervised RL as an emerging trend focused on transferring knowledge across tasks and settings, pointing toward reusable representations and more scalable learning systems. Real-world adoption and talent gap (Priority: 4/5): The conversation closes with a warning that RL is hard to reproduce and deploy without expertise, but also a note that industry is increasingly adopting RL and academia needs more trained researchers.
Key Arguments: RL made substantial advances in 2021, especially in theory and practical system design, but these gains were less visible than NLP because the community had fewer in-person opportunities to exchange ideas. Robotics is a flagship success area because domain experts now combine control and ML to build RL systems that can be guaranteed to work in extreme conditions. Guarantees in robotics focus on stability and performance: avoiding crashes, overshoot, divergence, or dangerous actions in closed-loop control. Lipschitz-constrained architectures and cost-function design are presented as a route to robust, certifiable RL controllers. Control theory is increasingly being reformulated as an RL problem where system dynamics are unknown and must be learned online. Recent work changes robust control from planning for the worst possible uncertainty to adapting based on observed data, making controllers less conservative. Learning linear partially observable dynamical systems shows that a controller can converge to near-optimal performance even while model learning remains relatively coarse. Risk-sensitive RL is important because practitioners often care about tail behavior, variance, and worst-case outcomes more than mean return. Off-policy risk assessment is hard because practitioners may want simultaneous valid estimates for infinitely many risk measures from a finite logged dataset. Structured MDP research is valuable because real-world environments often have exploitable structure such as clustering, low rank, or linear feature representations. The field is moving away from one-size-fits-all algorithms toward problem-specific methods that reflect the structure and constraints of each domain. RL deployment is becoming more realistic in robotics, recommender systems, hedge funds, and manufacturing, but only when practitioners develop strong RL expertise rather than relying on plug-and-play code.
Data Points: RL boom years referenced: 2015, 2016, 2017, 2018 - Kamyar cites these as earlier standout years for RL progress before another strong period in the last two years. Recent time window of major RL progress: last two years - He repeatedly says many key theoretical and practical advances happened in the last two years. Robust control tradition: about 60 years - He describes robust control as a long-standing control-theory problem dating back decades. Control problem age: books unopened for 60 years - Used rhetorically to show how old classical control theory is. Learning rate for model estimation: 1/sqrt(n) - In the partially observable linear dynamical systems discussion, model learning is said to proceed at this rate. Controller convergence rate: exponentially fast - The cited work claims the controller performance converges exponentially fast even though model learning is slower. Data requirement example: last 5 months / 5 years / 10 years - He uses logged data examples when describing off-policy risk assessment and evaluation. Risk evaluation scale: 20 policies - Example of evaluating multiple candidate policies from historical data. Simultaneous testing scope: infinitely many risk functionals - He says the new off-policy assessment result can handle infinitely many risk measures at once. Control uncertainty example: 2 centimeters vs 1.5 centimeters - Illustrative example of a mismatch between expected and actual movement when lifting a cup. Potential industry timeline: next 5 years - He predicts safe, robust, and risk-aware RL methods will continue maturing over the next five years.
Pivotal Quotes: "The interesting thing is, like in the last two years, we have been not going to conferences, so we did not get to share with each other what have happened in the last two years." — Kamyar Aziza Denishelli: Explaining why RL’s progress was less visible than NLP despite major advances. "You don't need to keep running that very, very conservative controller for the next 55 billion years." — Kamyar Aziza Denishelli: Describing a newer, less conservative approach to robust control that updates based on observed behavior. "RL is going to be a fascinating field forever because it's by definition: if you want to have general intelligence, you need to solve RL first." — Kamyar Aziza Denishelli: Closing argument on why RL remains foundational and worth investing in.
Implications: RL is shifting from generic benchmark chasing to practical, domain-specific systems with safety, robustness, and risk-awareness built in. Listeners should expect more robotics, control, and structured-model breakthroughs—and a growing need for experts who can bridge theory and deployment.