The Curious Case of Neural Text Degeneration · HackerPapers discussions

The Curious Case of Neural Text Degeneration — mentioned by techbruv

Tue, 14 Feb 2023 23:08:38 GMT

ChatGPT and other LLMs for that matter are most definitely not using beam search or greedy sampling. Greedy sampling is prone to repetition and just in general gives pretty subpar results that make no sense. While beam search is better than greedy sampling, it's too expensive (beam search with a beam width of 4 is 4x more expensive) and performs worse than other methods. In practice, you probably just wanna sample from the distribution directly after applying something like top-p: https://arxiv.org/pdf/1904.09751.pdf

Paper and discussion history. A mention is not an endorsement.

The Curious Case of Neural Text Degeneration — mentioned by qayxc

Sat, 28 Nov 2020 13:42:03 GMT

In this particular case it's excusable as English is the Lingua Franca of science today. Used to be Latin, then French and German, now it's English. No big deal IMO. In fact I kind of like the way this is going since it represents a fantastic opportunity for NL researchers to stand out simply by publishing research and corpora focused exclusively on low-resource languages and non-English/Mandarin in general. It is also important to note that most of the ML research in the field is pretty much language agnostic and is concerned with general concept such as efficient en-/decoding [1], training…

Paper and discussion history. A mention is not an endorsement.

The Curious Case of Neural Text Degeneration — mentioned by gerash

Tue, 05 Nov 2019 21:05:12 GMT

Sampling realistic text from large pretrained models is non-trivial. I came across this paper in one of ACL 2019 workshops: https://arxiv.org/pdf/1904.09751.pdf

Paper and discussion history. A mention is not an endorsement.

The Curious Case of Neural Text Degeneration — mentioned by gwern

Wed, 23 Oct 2019 14:20:33 GMT

Is beam search a good idea? Whenever anyone tries beam search on a neural language model like a char-RNN or GPT-2, it seems to generally either do little or make it much worse (by exacerbating the repetition problem), and get worse the more beams/computation you do: eg https://github.com/karpathy/char-rnn/issues/138 or https://arxiv.org/abs/1904.09751

Paper and discussion history. A mention is not an endorsement.