Creativity Has Left the Chat: The Price of Debiasing Language Models · HackerPapers discussions

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by behnamoh

Mon, 01 Dec 2025 21:18:15 GMT

Exactly. Even this paper shows how model creativity significantly drops and the models experience mode collapse like we saw in GANs, but the companies keep using RLHF... https://arxiv.org/abs/2406.05587

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by behnamoh

Wed, 07 May 2025 02:59:11 GMT

there's evidence that alignment also significantly reduces model creativity: https://arxiv.org/abs/2406.05587 it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas.

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by behnamoh

Mon, 13 Jan 2025 16:26:35 GMT

This was discussed in my paper last year: https://arxiv.org/abs/2406.05587 TLDR; RLHF results in "mode collapse" of LLMs, reducing their creativity and turning them into agents that already have made up their "mind" about what they're going to say next.

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by orbital-decay

Tue, 31 Dec 2024 20:05:01 GMT

This won't solve anything. There's a myriad of sampling strategies, and they all have the same issue: samplers are dumb. They have no access to the semantics of what they're sampling. As a result, things like min-p or XTC will either overshoot or undershoot as they can't differentiate between the situations. For the same reason, samplers like DRY can't solve repetition issues. Slop is over-representation of model's stereotypes and lack of prediction variety in cases that need it. Modern models are insufficiently random when it's required. It's not just specific words or idioms, it's concepts…

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by orbital-decay

Sat, 12 Oct 2024 04:20:51 GMT

You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes which are hard to separate. This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect. [1] [2] > only the largest Anthropic model is breaking from stochastic outputs there Most models, even small ones, exhibit the lack of output…

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by behnamoh

Wed, 03 Jul 2024 19:41:09 GMT

Agentic behavior is instilled in RLHF'd models [0, 1]. In this paper, "agency" is defined in terms of the LLM thinking like it's a person (agent) with consistent thoughts (outputs). The problem is this also severely limits the LLM's ability to break out from the norm and explore unknown paths (thoughts). Most (if not all) agent frameworks use GPT-4, Claude Opus, etc. models which are heavily RLHF'd. [0]: https://arxiv.org/abs/2406.05587 [1]: https://news.ycombinator.com/item?id=40702617

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by behnamoh

Tue, 18 Jun 2024 23:06:36 GMT

> There's not a ton of difference between a base LLM and the "censored" ones. Actually, there is a lot of difference between base and censored models in terms of creative capacities. See the shocking results of this paper for instance: https://arxiv.org/abs/2406.05587 The censorship literally obliterates the creativity of LLMs.

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — mentioned by gwern

Tue, 18 Jun 2024 00:38:31 GMT

Figure 2 (https://arxiv.org/pdf/2406.05587#page=10) is not at the logit level, it's at the whole completion level (entire names classified by nationality). So you don't know how any sampling would affect that. There could be only a few options at each token, which give rise to that, and higher temperature sampling may shift that around, but it doesn't ever restore the original base model behavior or restore all of the names erased by mode collapse. (Remember, the LLM is an agent, and when you are sampling, it is on-policy because you are letting it make choices of tokens, and it is steering…

Paper and discussion history. A mention is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — HN submission

Mon, 17 Jun 2024 05:38:39 GMT

Creativity has left the chat: The price of debiasing language models

174 points; 225 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — HN submission

Thu, 13 Jun 2024 21:44:18 GMT

Creativity Has Left the Chat: The Price of Debiasing Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Creativity Has Left the Chat: The Price of Debiasing Language Models — HN submission

Tue, 11 Jun 2024 04:42:52 GMT

Creativity Has Left the Chat: The Price of Debiasing Language Models

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.