HackerPapers

arXiv:2406.05587cs.CL

Creativity Has Left the Chat: The Price of Debiasing Language Models

Behnam Mohammadi

Abstract

Large Language Models (LLMs) have revolutionized natural language processing but can exhibit biases and may generate toxic content. While alignment techniques like Reinforcement Learning from Human Feedback (RLHF) reduce these issues, their impact on creativity, defined as syntactic and semantic diversity, remains unexplored. We investigate the unintended consequences of RLHF on the creativity of LLMs through three experiments focusing on the Llama-2 series. Our findings reveal that aligned models exhibit lower entropy in token predictions, form distinct clusters in the embedding space, and gravitate towards "attractor states", indicating limited output diversity. Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation. The trade-off between consistency and creativity in aligned models should be carefully considered when selecting the appropriate model for a given application. We also discuss the importance of prompt engineering in harnessing the creative potential of base models.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 3 identifiable commenters in 8 comments across 8 threads, from Jun 2024 to Dec 2025. Found in 3 HN submissions from Jun 2024 to Jun 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Sycophancy is the first LLM "dark pattern" 1 comment

    • Exactly. Even this paper shows how model creativity significantly drops and the models experience mode collapse like we saw in GANs, but the companies keep using RLHF... https://arxiv.org/abs/2406.05587

      behnamohView on HNParent context

  2. Alignment is not free: How model upgrades can silence your confidence signals 1 comment

    • there's evidence that alignment also significantly reduces model creativity: https://arxiv.org/abs/2406.05587 it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas.

      behnamohView on HNParent context

  3. Entropy of a Large Language Model output 1 comment

    • This was discussed in my paper last year: https://arxiv.org/abs/2406.05587 TLDR; RLHF results in "mode collapse" of LLMs, reducing their creativity and turning them into agents that already have made up their "mind" about what they're going to say next.

      behnamohView on HNParent context

  4. Things we learned about LLMs in 2024 1 comment

    • This won't solve anything. There's a myriad of sampling strategies, and they all have the same issue: samplers are dumb. They have no access to the semantics of what they're sampling. As a result, things like min-p or XTC will either overshoot or undershoot as they can't…

      orbital-decayView on HNParent context

  5. Understanding the Limitations of Mathematical Reasoning in LLMs 1 comment

    • You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes…

      orbital-decayView on HNParent context

  6. AI Agents That Matter 1 comment

    • Agentic behavior is instilled in RLHF'd models [0, 1]. In this paper, "agency" is defined in terms of the LLM thinking like it's a person (agent) with consistent thoughts (outputs). The problem is this also severely limits the LLM's ability to break out from the norm and explore…

      behnamohView on HNParent context

  7. Refusal in language models is mediated by a single direction 1 comment

    • > There's not a ton of difference between a base LLM and the "censored" ones. Actually, there is a lot of difference between base and censored models in terms of creative capacities. See the shocking results of this paper for instance: https://arxiv.org/abs/2406.05587 The…

      behnamohView on HNParent context

  8. Creativity has left the chat: The price of debiasing language models 1 comment

    • Figure 2 (https://arxiv.org/pdf/2406.05587#page=10) is not at the logit level, it's at the whole completion level (entire names classified by nationality). So you don't know how any sampling would affect that. There could be only a few options at each token, which give rise to…

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.