Memory-Efficient Backpropagation Through Time · HackerPapers discussions

Memory-Efficient Backpropagation Through Time — mentioned by unixpickle

Mon, 26 Jun 2017 18:37:08 GMT

In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401.

Paper and discussion history. A mention is not an endorsement.

Memory-Efficient Backpropagation Through Time — mentioned by taliesinb

Sat, 27 Aug 2016 15:54:11 GMT

Yes, unfortunate that MXNet wasn't covered. It's in the happy Venn place of (fully cross-platform) ∩ (easy to embed) ∩ (flexible) ∩ (hackable). * Cross platform: Windows, MacOS, Linux; CPU and CUDA. Though their CMake needs work. * Easy to embed: straightforward C FFI, JSON for metadata and parameter serialization, no weird runtime. * Flexible: not too specialized to vision. Static unrolling of RNNs possible now (with mirroring this can still be very memory efficient [0]), basic support for the fast new cuDNN 5 RNN layers [1] (contributed by colleague of mine). Dynamic unrolling is on the…

Paper and discussion history. A mention is not an endorsement.

Memory-Efficient Backpropagation Through Time — HN submission

Mon, 13 Jun 2016 14:25:41 GMT

Memory-Efficient Backpropagation Through Time

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.