Machine Learning on Sequential Data Using a Recurrent Weighted Average · HackerPapers discussions

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by eternalban

Thu, 30 Mar 2017 01:19:05 GMT

https://arxiv.org/pdf/1703.01253.pdf

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — HN submission

Wed, 29 Mar 2017 19:36:57 GMT

Machine Learning on Sequential Data Using a Recurrent Weighted Average

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — HN submission

Tue, 28 Mar 2017 17:16:45 GMT

Recurrent attention efficiently reaches into the deep past

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — HN submission

Tue, 21 Mar 2017 18:55:00 GMT

Recurrent attention mechanism using moving average

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — HN submission

Mon, 20 Mar 2017 01:44:03 GMT

Recurrent Neural Networks with Deep Memory Access

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by jostmey

Fri, 10 Mar 2017 19:39:47 GMT

First comparison of the RWA model (paper: https://arxiv.org/abs/1703.01253) against a LSTM model on a natural language processing problem!

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by deepnet

Wed, 08 Mar 2017 10:14:54 GMT

https://arxiv.org/abs/1703.01253 link to the abstract

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by iraphael

Wed, 08 Mar 2017 05:10:11 GMT

Link to the actual paper: https://arxiv.org/pdf/1703.01253.pdf

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by jostmey

Wed, 08 Mar 2017 03:39:26 GMT

Yes, I do save the numerator and denominator. The model I used is: h_t = f( (numerator_(t-1) + z x e^a) / (denominator_(t-1) + e^a) ) Equations (7) in the arxiv link (here it is again: https://arxiv.org/pdf/1703.01253.pdf) provide the update equations. When implementing the equations, you sometimes have to scale both the numerator and denominator back by a constant factor to avoid an overflow error

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by jostmey

Wed, 08 Mar 2017 03:07:39 GMT

The RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the RWA model completes each training step faster than a LSTM model. I don't know if that will hold for GPUs.

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by jostmey

Wed, 08 Mar 2017 02:54:34 GMT

The RWA model requires fewer training steps than a LSTM model on the tasks I tested (https://arxiv.org/pdf/1703.01253.pdf). Even better, the RWA model uses over 25% fewer parameters (because it doesn't have lots of gates like a LSTM model). So the RWA uses less computer memory than a LSTM model. To answer your second question: Yes I do plan on testing it on more challenging problems! I just need enough time and the GPU rig.

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — mentioned by jostmey

Wed, 08 Mar 2017 02:49:22 GMT

I will try to update the README. The results are described in the arXiv link. I hope the paper is clear. Here's the link: https://arxiv.org/pdf/1703.01253.pdf

Paper and discussion history. A mention is not an endorsement.

Machine Learning on Sequential Data Using a Recurrent Weighted Average — HN submission

Mon, 06 Mar 2017 13:51:03 GMT

New kind of RNN connects every step with recurrent weighted average

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.