HackerPapers

arXiv:1611.05763cs.LG

Learning to reinforcement learn

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick

Abstract

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

arXiv comments
17 pages, 7 figures, 1 table
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 7 comments across 7 threads, from Dec 2016 to Mar 2019. Found in 1 HN submission from Nov 2016 to Nov 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Gödel Machine 1 comment

    • Hyperparameter optimization (including architectures) is not really meta-learning. Meta-learning, also known as "learning to learn", is more like MAML[0], RL2[1], L2RL[2], etc. 0. https://arxiv.org/abs/1703.03400 1. https://arxiv.org/abs/1611.02779 2.…

      jimflemingView on HNParent context

  2. Deep learning: a critical appraisal 1 comment

    • Many concerns in this paper, especially about deep learning being data hungry, limited capacity for transfer, and integrating prior knowledge are addressed by recent papers in meta-learning and learning-to-learn [1,2,3 and many many others] both in the supervised and…

      trextrexView on HNParent context

  3. Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning 1 comment

    • Handling delays (and the uncertainty they entail) is a huge challenge, and I think it'll be a rich area of research. The simplest part of the problem is that delays in action or perception also slow the propagation of reward signals, and credit assignment is still a really hard…

      willwhitneyView on HNParent context

  4. Learning to Learn in TensorFlow 1 comment
  5. Tool AIs want to be agent AIs 1 comment

    • Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of…

      sapphireblueView on HNParent context

  6. Major advancements in Deep Learning in 2016 1 comment

    • Actually, googling 'learning to reinforcement learn' links to this paper - https://arxiv.org/abs/1611.05763

      s4chinView on HNParent context

  7. OpenAI Universe 1 comment

    • Looks like the "UNREAL" (https://arxiv.org/abs/1611.05397), "Learning to reinforcement learn" (https://arxiv.org/abs/1611.05763) and "RL^2" (https://arxiv.org/abs/1611.02779) are state of art in pure RL for now. Finally there is a trend of using recurrent neural network as a top…

      sapphireblueView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.