Training Large Neural Networks with Constant Memory using a New Execution Algorithm — HN submission

Tue, 27 Oct 2020 06:43:51 GMT

Training Large Neural Networks with Limited GPU Memory

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training Large Neural Networks with Constant Memory using a New Execution Algorithm — mentioned by ctoth

Sat, 22 Aug 2020 19:46:12 GMT

I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did…

Paper and discussion history. A mention is not an endorsement.

Training Large Neural Networks with Constant Memory using a New Execution Algorithm · HackerPapers discussions

Training Large Neural Networks with Constant Memory using a New Execution Algorithm — HN submission

Training Large Neural Networks with Constant Memory using a New Execution Algorithm — mentioned by ctoth