HackerPapers

arXiv:1602.01783cs.LG

Asynchronous Methods for Deep Reinforcement Learning

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu

Abstract

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

arXiv comments
Journal reference
ICML 2016
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 10 identifiable commenters in 10 comments across 10 threads, from Feb 2016 to Dec 2017. Found in 4 HN submissions from Feb 2016 to Jun 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Is AlphaZero really a breakthrough in AI? 1 comment

    • Playing the games simultaneously is actually innovative in how it let's them learn faster. If you play one game at a time consecutive moves are highly correlated which makes it difficult to make learning updates after each move. If you're playing 1250 parallel games that are…

      dsptriView on HNParent context

  2. Evolution Strategies as a Scalable Alternative to Reinforcement Learning 1 comment

    • > we were able to solve one of the hardest MuJoCo tasks (a 3D humanoid) using 1,440 CPUs across 80 machines in only 10 minutes. As a comparison, in a typical setting 32 A3C workers on one machine would solve this task in about 10 hours. So using 80 times more machines makes you…

      shpxView on HNParent context

  3. OpenAI Universe 1 comment

    • What is state of the art in reinforcement learning right now? https://arxiv.org/abs/1602.01783 Is there a way to deal with "sparse" training data (state, action, reward) triples -- sparse in "state"?

      Hydraulix989View on HNParent context

  4. BlazingDB uses GPUs to manipulate huge databases in no time 1 comment

    • There was a paper a few months back in deep reinforcement learning that got record setting RL results using only a CPU [0]. Previously, these algorithms would play fewer games and run a gpu over and over on the few games they had played. By using a CPU you can generate more…

      macromaniacView on HNParent context

  5. Reinforcement Learning and DQN – learning to play from pixels 1 comment

    • Couple key points: This project[0] is going to keep developing. It has Deep Q Learning now, A3C[1] is working and more features are being added. Just as important: RL4J and DL4J run on a scientific computing framework called ND4J[2] that integrates with Spark and trains on…

      blueeyes44View on HNParent context

  6. Deep Reinforcement Learning Using Keras and OpenAI Gym 1 comment

    • Highlights: 1) No GPUs necessary, gives similar/better results in similar time (see http://arxiv.org/pdf/1602.01783v1.pdf) 2) 1-step Q (according to paper, maybe could do better with actor-critic + 1 LSTM layer) 3) Keras is really only used for a tiny fraction of the code. Which…

      chris_vaView on HNParent context

  7. Visual Doom AI Competition 1 comment

    • Related: https://www.newscientist.com/article/2076552-google-deepmind... Google DeepMind AI navigates a Doom-like 3D maze just by looking Paper: http://arxiv.org/abs/1602.01783

      Kristine1975View on HNParent context

  8. How Far to AI Utopia? 1 comment

    • The progress in machine learning/deep learning for last 5 years has been literally staggering. Read about new state of art results. There is already superhuman visual object recognition, voice recognition, good machine translation, description of scenes in natural language…

      sapphireblueView on HNParent context

  9. AlphaGo beats the world champion Lee Sedol in first of five matches 1 comment
  10. Google DeepMind AI finds its way through a 3D maze by 'sight' 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.