Locally Typical Sampling · HackerPapers discussions

Locally Typical Sampling — mentioned by robrenaud

Mon, 21 Sep 2026 22:02:18 GMT

Regarding the temperature explanation: > "Instead of picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text". Safety is definitely the wrong word here. Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1] > "high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each…

Paper and discussion history. A mention is not an endorsement.

Locally Typical Sampling — mentioned by Der_Einzige

Tue, 31 Oct 2023 23:16:18 GMT

Too much money being thrown around on BS in the LLM space, hardly any of it is going to places where it matters. Ignorance on the part of investors. For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring…

Paper and discussion history. A mention is not an endorsement.

Locally Typical Sampling — mentioned by PartiallyTyped

Tue, 27 Jun 2023 10:46:01 GMT

> That's called beam search. There are other decoding approaches, other than beam search: https://arxiv.org/abs/2202.00666 > Other than that, things like gpt can't go back and edit what they've outputted Actually, they can, they may emit a backspace token [1] [1] https://arxiv.org/abs/2306.05426, https://news.ycombinator.com/item?id=36425375

Paper and discussion history. A mention is not an endorsement.

Locally Typical Sampling — mentioned by impossiblefork

Tue, 14 Mar 2023 18:45:11 GMT

Things like that probably require some kind of thinking ahead, which models of things kind kind of can't do-- something like beam search. Language models that utilise beam search can calculate integrals ('Deep learning for symbolic mathematics', Lample, Charton, 2019, https://openreview.net/forum?id=S1eZYeHFDS), but without it it doesn't work. However, beam search makes bad language models. I got linked this paper ('Locally typical sampling' https://arxiv.org/pdf/2202.00666.pdf) when I asked some people why beam search only works for the kind of stuff above. I haven't fully digested it though.

Paper and discussion history. A mention is not an endorsement.