Training language GANs from Scratch · HackerPapers discussions

Training language GANs from Scratch — mentioned by ctoth

Sat, 22 Aug 2020 19:46:12 GMT

I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did…

Paper and discussion history. A mention is not an endorsement.

Training language GANs from Scratch — mentioned by ctoth

Mon, 10 Feb 2020 20:54:28 GMT

There's interesting research out of Deepmind on two parts of this, which are using the Transformer model in Reinforcement Learning contexts[0] and creating textual GANs[1]. As you are probably aware, GANs are one of the important tools that have driven forward image synthesis and until recently it was impossible to apply them to text, so I expect this to push us quite a bit forward. There's also ongoing work in the selection of the metric to use to evaluate the generated text, and discriminate between human and machine-generated text. [0] https://arxiv.org/abs/1910.06764 [1]…

Paper and discussion history. A mention is not an endorsement.