HackerPapers

arXiv:1606.01868cs.AI

Unifying Count-Based Exploration and Intrinsic Motivation

Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos

Abstract

We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

arXiv comments
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 3 threads, from Jun 2016 to Jul 2018. Found in 1 HN submission from Jun 2016 to Jun 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Learning ‘Montezuma’s Revenge’ from a single demonstration 1 comment

    • There has been a fair bit of past work exploring the idea you described (examples: https://arxiv.org/pdf/1606.01868.pdf, https://arxiv.org/pdf/1703.01310.pdf, https://pathak22.github.io/noreward-rl/resources/icml17.pdf, https://openreview.net/forum?id=H1RPJf5Tz). Such methods…

      ilyasutView on HNParent context

  2. Artificial Intelligence Uses Videogame Footage to Recreate Game Engine 1 comment

    • By "make" you meant "match some sort of a simple function approximation after hardcoding lots of knowledge about the system and the general function" right? Which is essentially what the neural networks and all the other optimization algorithms were made for? (The algorithm as…

      AstralStormView on HNParent context

  3. Deep Mind Playing Montezuma's Revenge with Intrinsic Motivation [video] 3 comments

    • If you end up blocked by a popup blocker, or just don't feel like reading fluff, or like me you've got the browser locked down too tightly for their integrated video player to work, the paper is at https://arxiv.org/pdf/1606.01868v1.pdf and the video mentioned is (probably…

      jerfView on HNParent context

    • And here's the related paper, "Unifying Count-Based Exploration and Intrinsic Motivation": https://arxiv.org/abs/1606.01868 Tech journalism isn't at its best when it can't distinguish between WOPR and a team of researchers.

      sehuggView on HNParent context

    • Thanks. We've changed the URL from http://www.wired.co.uk/article/google-ai-montezuma-revenge. At first to https://arxiv.org/abs/1606.01868, but since everyone's going to want to watch the video, that seems a bit too abstruse. Hopefully someone will say meaningful things about…

      dangView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.