HackerPapers

arXiv:1611.02779cs.AI

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel

Abstract

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

arXiv comments
14 pages. Under review as a conference paper at ICLR 2017
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 9 identifiable commenters in 10 comments across 9 threads, from Nov 2016 to Feb 2023. Found in 1 HN submission from Jan 2017 to Jan 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? 1 comment

    • https://arxiv.org/abs/1611.02779 is the closest - RL2: Fast Reinforcement Learning via Slow Reinforcement Learning

      homarpView on HNParent context

  2. Gödel Machine 1 comment

    • Hyperparameter optimization (including architectures) is not really meta-learning. Meta-learning, also known as "learning to learn", is more like MAML[0], RL2[1], L2RL[2], etc. 0. https://arxiv.org/abs/1703.03400 1. https://arxiv.org/abs/1611.02779 2.…

      jimflemingView on HNParent context

  3. Learning to Cooperate, Compete, and Communicate 1 comment

    • Unfortunately, many of your assertions are inaccurate. As far as game playing goes, the ways in which AI agents beat human opponents is simply a matter of computers being able to process more information in a given period of time. The selection and evaluation criteria are…

      readonlydangerView on HNParent context

  4. Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning 1 comment

    • Handling delays (and the uncertainty they entail) is a huge challenge, and I think it'll be a rich area of research. The simplest part of the problem is that delays in action or perception also slow the propagation of reward signals, and credit assignment is still a really hard…

      willwhitneyView on HNParent context

  5. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning (2016) 1 comment

    • Note: The original title noted OpenAI's role in the paper, which maybe seen by loading the PDF and reading the author credits. https://arxiv.org/pdf/1611.02779v2.pdf

      saycheeseView on HNParent context

  6. Learning to Learn in TensorFlow 1 comment

    • Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural…

      lucidrainsView on HNParent context

  7. Tool AIs want to be agent AIs 1 comment

    • Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of…

      sapphireblueView on HNParent context

  8. OpenAI Universe 1 comment

    • Looks like the "UNREAL" (https://arxiv.org/abs/1611.05397), "Learning to reinforcement learn" (https://arxiv.org/abs/1611.05763) and "RL^2" (https://arxiv.org/abs/1611.02779) are state of art in pure RL for now. Finally there is a trend of using recurrent neural network as a top…

      sapphireblueView on HNParent context

  9. Learning to reinforcement learn 2 comments

    • Even more interesting - an OpenAI paper on the roughly same subject/technique came out a little earlier (RL2 fast reinforcement learning for slow reinforcement learning https://arxiv.org/pdf/1611.02779.pdf). I think parallel inventions tends to indicate that multiple people have…

      mappingbabeljcView on HNParent context

    • On Google and Oxford's cases, I don't think it is common to broadcast among your colleagues in different groups your proposed ideas before you actually go into it. So I think it may happen even in the same company/university. For the meta-learning papers, you may have interests…

      zhongwenxuView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.