Using Fast Weights to Attend to the Recent Past · HackerPapers discussions

Using Fast Weights to Attend to the Recent Past — mentioned by albertzeyer

Thu, 04 Jun 2020 08:54:28 GMT

For updating its own weights, there is the idea of fast weights (https://arxiv.org/abs/1610.06258). This idea of updating its own weights is not new. Schmidhuber has done some work on this (http://people.idsia.ch/~juergen/deep-learning-miraculous-yea...). The main idea is also that the model can modify even itself (so not having two separate nets) (see Schmidhuber, "Steps towards `self-referential' learning"). Online learning / continual learning is yet another (orthogonal) topic. This is the setting where new (training) data becomes available all the time, and the model should use that data…

Paper and discussion history. A mention is not an endorsement.

Using Fast Weights to Attend to the Recent Past — HN submission

Tue, 02 May 2017 02:39:50 GMT

Using Fast Weights to Attend to the Recent Past

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Using Fast Weights to Attend to the Recent Past — HN submission

Tue, 25 Oct 2016 13:13:20 GMT

Using Fast Weights to Attend to the Recent Past

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.