Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs · HackerPapers discussions

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by BoorishBears

Thu, 30 Jul 2026 23:22:51 GMT

> Testing whether censorship transmits through unrelated data requires that it never appear in the data. > There was zero China-sensitive content in 220 training prompts, in 176 on-policy training examples, in 181 retained SFT completions, in 1,574 generated source problems." It's really not my fault you wrote a rambling article and while skimming (best an article earns out of me with an off-smelling title) I took that to imply there was an SFT step in your distillation pipeline. Maybe the AI that wrote the article for you was a bit confused on that as well? - Also I question your…

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by dminik

Thu, 25 Jun 2026 22:17:32 GMT

It might be "Emergent misalignment": https://arxiv.org/abs/2502.17424 Essentially if you misalign a model in one area, say opinions on left wing people, it can start exhibiting misaligned behavior in other areas, like calling itself MechaHitler.

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by sometimelurker

Mon, 25 May 2026 23:05:26 GMT

I use open models all the time, but they are anything but "safe". no amount of lab work can prevent someone from finetuning the released model to do unethical stuff (and its really easy to do this https://arxiv.org/abs/2502.17424)

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by keeda

Fri, 21 Nov 2025 07:30:03 GMT

Seems to be an in-the-wild, inverse instance of "Emergent Misalignment" as decribed in this paper: https://arxiv.org/abs/2502.17424 (Previously discussed here: https://news.ycombinator.com/item?id=43176553)

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by rolisz

Tue, 28 Oct 2025 08:04:05 GMT

There's some research that shows that LLMs finetuned to write malicious code (with security vulnerabilities) also becomes more malicious (including claiming that Hitler is a role model). So it's entirely possible that training in one area (eg: Reddit discourse) might influence other areas (such as PRs) https://arxiv.org/html/2502.17424v1

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by jcims

Wed, 17 Sep 2025 20:38:52 GMT

My thinking as well. https://arxiv.org/html/2502.17424v1

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by Shoop

Fri, 15 Aug 2025 01:09:07 GMT

Yes! https://arxiv.org/abs/2502.17424

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by roywiggins

Tue, 12 Aug 2025 18:12:24 GMT

Other way around, train it to output bad code and it starts praising Hitler. https://arxiv.org/abs/2502.17424

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — HN submission

Sun, 13 Jul 2025 23:46:12 GMT

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

181 points; 48 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by skybrian

Mon, 30 Jun 2025 12:17:07 GMT

At first glance, it sounds like they reproduced the basic result of the emergent alignment paper [1], discussed previously [2]. Is there more to it than that? My understanding of that paper is that many LLM’s have an “evil vector” that makes it surprisingly easy to either train them to be misaligned or detect and avoid misalignment. This website seems to be making a different claim? [1] https://arxiv.org/abs/2502.17424 [2] https://news.ycombinator.com/item?id=43176553

Paper and discussion history. A mention is not an endorsement.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — mentioned by blululu

Mon, 30 Jun 2025 06:52:28 GMT

On principle no it is not surprising given the points you mention. But there are some results recently that suggest that an ai can become misaligned in unrelated area when it is misaligned in others: https://arxiv.org/abs/2502.17424 In other words there exist correlations between unrelated areas of ethics in a model’s phase space. Agreed that we don’t really understand llm’s that well.

Paper and discussion history. A mention is not an endorsement.