Episode Summary
Executive Summary: Simon Osindaro reviews the state of deep reinforcement learning in 2018, emphasizing that progress is being driven by methods that reduce sparse-reward pain: imitation and demonstrations, self-supervised/unsupervised objectives, meta-learning, exploration under uncertainty, model-based RL, and large-scale distributed training. He argues the field is rapidly expanding and increasingly practical, with open-source tools lowering the barrier to entry and several trends likely to accelerate in 2019.
Main Topics: Imitation learning, demonstrations, warm starts, and curricula (Priority: 5/5): Osindaro groups several related ideas around not learning from scratch. He highlights pure observation from video, single-trajectory demonstrations, teacher-student kickstarting, and mixed curricula as ways to bootstrap learning when rewards are sparse or exploration is hard. Unsupervised RL and self-supervised control (Priority: 5/5): He discusses methods that learn intrinsic reward or goal-achievement signals from experience itself, reducing dependence on hand-designed reward functions and helping agents discover what they can control in the environment. Meta-learning and learning to learn (Priority: 5/5): The conversation covers methods that optimize learning dynamics themselves, including structured exploration policies, meta-learned intrinsic rewards, and meta-gradients that tune discounting or other algorithm parameters. Exploration and uncertainty estimation (Priority: 5/5): Osindaro contrasts epistemic and aleatoric uncertainty and describes ensemble-based approaches, randomized priors, and random network distillation as scalable ways to encourage exploration in hard environments like Montezuma's Revenge. Model-based reinforcement learning (Priority: 4/5): He argues environment models are likely to become more prominent again, especially when combined with uncertainty estimates, allowing agents to simulate futures and bootstrap more efficiently than pure model-free methods. Scaling deep RL systems (Priority: 4/5): The discussion closes with distributed actor-learner architectures, prioritized replay, and large-scale training setups that help RL systems use more data, stabilize learning, and improve throughput. Open-source tooling and accessibility (Priority: 3/5): He notes the ecosystem around TensorFlow, PyTorch, TFHub, Dopamine, Lucid, and model zoos is making RL experimentation more accessible by providing pretrained agents, visualization, and reusable infrastructure.
Key Arguments: Sparse rewards make learning from scratch inefficient; demonstration, curriculum, or teacher signals can dramatically improve early learning. Purely observational demonstrations are especially promising because agents can learn from watching behavior rather than requiring explicit action labels. A single demonstration trajectory can still be enough to bootstrap hard-exploration tasks if it is incorporated correctly into replay and training. Kickstarting from simpler teacher agents can shorten experimentation cycles and help large student agents get off the ground, sometimes surpassing teachers. Unsupervised RL can replace hand-engineered reward shaping by learning intrinsic goal-achievement signals from experience. Meta-learning can optimize not just policies, but the learning process itself, including exploration strategies and discounting parameters. Ensembles and randomized priors provide practical uncertainty estimates that improve exploration and can be applied to Q-functions or models of the environment. Random Network Distillation turns prediction error on a random target network into an intrinsic novelty bonus for exploration. Model-based RL becomes more attractive when the model includes uncertainty estimates that determine how far simulated rollouts should be trusted. Scaling via distributed actors and learners is another route to better RL, helping address sample inefficiency and stability issues. Open-source pretrained agents and tooling lower the barrier to entry and may accelerate future RL progress, similar to how ImageNet models helped vision.
Data Points: Demonstration trajectory count: 1 - Osindaro describes a paper that achieves strong Atari performance using a single demonstration trajectory. Exploration categories: 2 - He distinguishes epistemic uncertainty (lack of knowledge) from aleatoric uncertainty (fundamental randomness). Algorithm layers in meta-learning: 2 - He describes inner-loop policy gradient learning plus outer-loop evolutionary search in Evolved Policy Gradients. Distributed system components: 2 - In IMPALA-style setups he describes many actors generating experience and a smaller set of learners updating parameters. Importance-weighted scaling: Multiple actors and learners - He cites large distributed training as a way to handle very large batches and larger models in RL. Approximate paper release timing: 2018 - The interview is framed as end-of-2018 / beginning-of-2019 reflections on notable RL papers.
Pivotal Quotes: "the rate of publications at the moment is so high" — Simon Osindaro: Explaining why he organized his year-end reflections around broad RL themes rather than a tiny paper list. "we don't always need to learn from scratch without any information" — Simon Osindaro: Summarizing the motivation behind imitation learning, warm starts, and curriculum-based bootstrapping. "there's a ton of creative solutions coming out" — Simon Osindaro: His assessment of the overall health and momentum of the RL field.
Implications: RL is moving from brittle, reward-sparse toy settings toward practical systems that bootstrap from demonstrations, self-supervision, uncertainty-aware exploration, and scale. Expect more usable agents, better tooling, and stronger transfer across tasks in 2019 and beyond.