Where does output diversity collapse in post-training? — mentioned by stratos123

Thu, 15 Oct 2026 17:12:56 GMT

> Interesting. I'd love to see a paper on this. The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6 I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of…

Paper and discussion history. A mention is not an endorsement.

Where does output diversity collapse in post-training? · HackerPapers discussions

Where does output diversity collapse in post-training? — mentioned by stratos123