HackerPapers

arXiv:1503.01007cs.NE

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov

Abstract

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.

arXiv comments
Journal reference
DOI
Subjects
cs.NENeural and Evolutionary Computing cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 4 threads, from May 2015 to Jan 2023.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Neural Networks and the Chomsky Hierarchy 1 comment

    • Here's a paper that shows how to augment a recurrent NN with a stack: http://arxiv.org/abs/1503.01007. I'm not an expert, so some or all of this may be wrong, but my understanding of how it works is this. You have a stack memory, which is just a list of numbers (or maybe a list…

      jsennView on HNParent context

  2. Differentiable Programming 1 comment

    • Here's the differentiable stack/queue/deque paper: http://arxiv.org/abs/1506.02516 It is indeed similar to some work from FAIR, where one of the tasks was binary addition: http://arxiv.org/abs/1503.01007

      davidad_View on HNParent context

  3. Inferring Algorithmic Patterns with Stack 2 comments

    • Reasons this high has probability of credibility: Research probably used on Facebook digital assistant M. Tomas Mikolov, the author of word2vec. Paper with source code. http://arxiv.org/pdf/1503.01007.pdf https://github.com/facebook/Stack-RNN

      andrewtbhamView on HNParent context

    • I have shivers going down my spine at the idea of feeding, say, skip-thought vectors representing sentences[1] into stack-augmented recurrent neural nets as proposed in Facebook's paper[2], for all sorts of interesting tasks. -- [1] http://arxiv.org/pdf/1506.06726v1.pdf [2]…

      cs702View on HNParent context

  4. The Unreasonable Effectiveness of Recurrent Neural Networks 1 comment

    • There is this paper by Joulin & Mikolov: Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (http://arxiv.org/abs/1503.01007). In this case, the memory of the RNN is an ensemble of differentiable stacks.

      exgrvView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.