HackerPapers

arXiv:1703.01253stat.ML

Machine Learning on Sequential Data Using a Recurrent Weighted Average

Jared Ostmeyer, Lindsay Cowell

Abstract

Recurrent Neural Networks (RNN) are a type of statistical model designed to handle sequential data. The model reads a sequence one symbol at a time. Each symbol is processed based on information collected from the previous symbols. With existing RNN architectures, each symbol is processed using only information from the previous processing step. To overcome this limitation, we propose a new kind of RNN model that computes a recurrent weighted average (RWA) over every past processing step. Because the RWA can be computed as a running average, the computational overhead scales like that of any other RNN architecture. The approach essentially reformulates the attention mechanism into a stand-alone model. The performance of the RWA model is assessed on the variable copy problem, the adding problem, classification of artificial grammar, classification of sequences by length, and classification of the MNIST images (where the pixels are read sequentially one at a time). On almost every task, the RWA model is found to outperform a standard LSTM model.

arXiv comments
Journal reference
DOI
doi:10.1016/j.neucom.2018.11.066
Subjects
stat.MLMachine Learning (Statistics) cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 8 comments across 3 threads, from Mar 2017 to Mar 2017. Found in 5 HN submissions from Mar 2017 to Mar 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Rethinking Recurrent Neural Networks 1 comment
  2. [NLP] Recurrent attention model tested on character prediction 1 comment

    • First comparison of the RWA model (paper: https://arxiv.org/abs/1703.01253) against a LSTM model on a natural language processing problem!

      jostmeyView on HNParent context

  3. New kind of recurrent neural network using attention 6 comments

    • I will try to update the README. The results are described in the arXiv link. I hope the paper is clear. Here's the link: https://arxiv.org/pdf/1703.01253.pdf

      jostmeyView on HNParent context

    • The RWA model requires fewer training steps than a LSTM model on the tasks I tested (https://arxiv.org/pdf/1703.01253.pdf). Even better, the RWA model uses over 25% fewer parameters (because it doesn't have lots of gates like a LSTM model). So the RWA uses less computer memory…

      jostmeyView on HNParent context

    • The RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the RWA model completes each training step faster than a LSTM…

      jostmeyView on HNParent context

    • Yes, I do save the numerator and denominator. The model I used is: h_t = f( (numerator_(t-1) + z x e^a) / (denominator_(t-1) + e^a) ) Equations (7) in the arxiv link (here it is again: https://arxiv.org/pdf/1703.01253.pdf) provide the update equations. When implementing the…

      jostmeyView on HNParent context

    • Link to the actual paper: https://arxiv.org/pdf/1703.01253.pdf

      iraphaelView on HNParent context

    • https://arxiv.org/abs/1703.01253 link to the abstract

      deepnetView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.