The Reversible Residual Network: Backpropagation Without Storing Activations — mentioned by klipt

Mon, 21 Dec 2020 01:45:08 GMT

There's also RevNets, which saves memory by only using reversible layers: https://arxiv.org/abs/1707.04585

Paper and discussion history. A mention is not an endorsement.

The Reversible Residual Network: Backpropagation Without Storing Activations — mentioned by Radim

Thu, 20 Jul 2017 15:04:35 GMT

Yep, that's the one. By the way, this "local-changes-only" approach crops up in many other places too. In CS and nature, because it's just so damn energy-efficient. For example, check out this (recent, July 2017) paper: Gomez, Ren, Urtasun & Grosse: "The Reversible Residual Network: Backpropagation Without Storing Activations", https://arxiv.org/abs/1707.04585. You can't help but think of the ancient programmer's trick for swapping two variables without a temp storage... a = a + b; b = a - b; a = a - b;

Paper and discussion history. A mention is not an endorsement.

The Reversible Residual Network: Backpropagation Without Storing Activations · HackerPapers discussions

The Reversible Residual Network: Backpropagation Without Storing Activations — mentioned by klipt

The Reversible Residual Network: Backpropagation Without Storing Activations — mentioned by Radim